The Strange Case of the Chinese Nuclear Tunnels is, for the time being, closed. Professor Philip Karber and his students at Georgetown (where I earned my doctorate, I am proud to add) have given their 363-page report to the Defense Department and the arms control community. It has landed with a thud.
Karber, you may remember from earlier posts here, was the origin of a weeks-long story on tunnels under China. The issue, actually, wasn’t the tunnels, which have been there since the 1950s and which were not a secret. The real question revolved around what was in them, and Karber and his students claimed they knew the answer: nuclear weapons.
Now, no one’s disputing that the Chinese might be building tunnels to protect their nuclear forces. The nuclear-owning states have a history of doing weird things, and the Chinese and the former Soviets have an even longer history of out-weirding everyone.
But Karber and his student team said that China could be using those tunnels not to protect existing nuclear forces, but to hide extra weapons that no one knows exist. The current Chinese arsenal is estimated to be somewhere around 300-400 weapons; Karber claims there could be three thousand more hiding down there.
Where did that number come from? Unfortunately, it seems to have come from huge assumptions, sloppy scholarship — including a basic fact-checking error that would not have escaped even a rookie book editor — and highly elastic interpretations of open source material like Google Earth.
The answer from the professional arms control and defense analysis community, apparently, is unanimous:
Actually, the answer is a little more pointed than that.
“China has not produced enough fissile material to produce 3,000 nuclear weapons,” said Hans Kristensen, director of the nuclear information project at the Federation of American Scientists.”Nor do they have delivery systems for so many weapons. It’s just inaccurate on all fronts, that estimate.”
Meanwhile, over at the Union of Concerned Scientists, China expert Gregory Kulacki was brutal, calling the scholarship “incompent,” “lazy,” and below the standards a high school would set for abusing Internet evidence:
This is not competent scholarship. There are thousands of claims on Internet blog posts, supposedly based on US government documents, that aliens from outer space were recovered in Roswell, New Mexico in 1947. Just because a lot of people repeat them in their personal blogs does not give them credibility.
Kulacki also suggested that the tunnels might encourage the Chinese to have fewer rather than more missiles in the belief that they were better protected, which is an explanation that makes far more sense to me. (What’s the gain or purpose of having an extra 3000 weapons if they’re essentially invisible? As Dr. Strangelove said, if you have a Doomsday Device, you’ve got to tell someone.)
The DoD reaction has been frosty, to say the least, probably because their analysts were led on a wild goose chase:
The report has been reviewed by the Defense Department but has not led to a revision of estimates on the size of China’s nuclear force, said a senior defense official, who spoke on condition of anonymity. “The findings of the report have been noted,” said the official. But “there is no change in our assessment.”
For those of you not conversant with government-speak, “noted” is what your superiors say when you raise something and are overruled, usually with irritation.
This was compounded by overselling the report before it was released. The advance story in The Washington Post was breathless:
The Chinese have called it their “Underground Great Wall” — a vast network of tunnels designed to hide their country’s increasingly sophisticated missile and nuclear arsenal. For the past three years, a small band of obsessively dedicated students at Georgetown University has called it something else: homework.
Led by their hard-charging professor, a former top Pentagon official, they have translated hundreds of documents, combed through satellite imagery, obtained restricted Chinese military documents and waded through hundreds of gigabytes of online data.
Man, I wish I could get press like that. (Hey, I’m “hard-charging,” too, just ask around.) But no matter how anyone might spin it, it’s a bad sign when the Federation of American Scientists, the Union of Concerned Scientists, and the Defense Department all conclude that your research was, um, sub-par, to say the least.
The Chinese Tunnel Mystery is, in my view, a classic example of what happens when people get wedded to a theory and then cherry-pick the evidence until it all fits.
Full disclosure: I don’t know Professor Karber personally and I don’t have a copy of the report yet, so I can only speculate from a distance and based on the reviews by Kristensen and Kulacki, two of the top experts in the world on nuclear weapons.
From here, it looks like Karber and his students developed the really cool “extra nukes” theory while poking around into the much more substantive question of the tunnels themselves, and then got so excited about it that they went on to find plenty of evidence by taking into account anything that could possibly confirm their theory, like Chinese blogs — and we all know how reliable blogs are — and even a fictional Chinese TV series.
All of this created a serious buzz earlier on, because no one had read the actual report. And that’s the other problem with controversial research: it all sounds exciting as hell if you just skip to the executive summary.
This is a constant hazard of research in any field. But it’s especially a problem when studying closed societies like China (or during the Cold War, the USSR), for obvious reasons. Karber is a veteran of those Cold War era efforts and that may have predisposed him to worst-casing the Chinese scenario.
But this is why people submit things to editors, journals, and peer reviewers: so that someone who isn’t in love with the theory can look at it more dispassionately. You’re supposed to work just as hard — harder, even — to disprove your own theory as you did to create it. (Trust me, your peers will if you won’t.) That’s how scholarship works.
So it’s no surprise the study got hammered by the kinds of objections that would normally have been raised in the review process; those objections probably would have prevented the study from being published at all. A solid project review early on could have saved what was clearly thousands of student hours, lots of computer time, and some serious money invested in this whole fixation. That’s too bad, because I’d still like to know why the Chinese are acting like giant earthworms.
I have a particular burr in my saddle here about the involvement of the students. Working with students on research is a practice I really enjoy and have actively supported in my career, including taking the first initiative ever to establish a successful (and still operating) civilian graduate student internship program at the Naval War College many years ago. Working with your students is crucial to their training, and it sure helps to keep the professor on his toes.
But if the scholar in charge isn’t careful, it can turn into a hothouse, because the students have a vested interest in finding what the professor wants to find. Controversial or difficult research requires a hard-assed skeptic on the team; your own students simply cannot fulfill that role, which is normally one reserved for editors and peer-reviewers.
Karber, for his part, argues he succeeded by sparking debate on the subject. I disagree: it’s one thing to shed light, it’s another thing to sow analytical chaos with sloppy research. (Look at the climate change debate. At this point, I think the earth is now heating up from sheer bloviation.)
You also get some hyperbolic claims as the news spreads at the speed of blogging, with triumphant messages like “Georgetown students help uncover Chinese nuclear weapons.” (No, they didn’t.)
Of course, you also get some
bat-shit crazy odd blog posts, like this one about how Georgetown’s government department is “shadowy.” Oy.
Worse, after all the hullaballoo, Karber admits that he really won’t stand by those colossal numbers:
“I don’t have the slightest idea how many nuclear weapons China really has, but neither does anyone else in the arms control community,” Karber told the [Washington] Post. “That’s the problem with China — no one really knows except them.”
But okay: let’s grant some uncertainty in the nuclear count. But seriously: three thousand hidden nuclear weapons? That would qualify as one of the greatest strategic deceptions in history. Karber and his students would have been taken a lot more seriously, I think, had they been more circumspect with their numbers — but let’s face it, circumspection does not get the attention of the media.
If the Chinese really had 3000 extra weapons, it would indict the entire U.S. national security establishment, with its billions of dollars of equipment and thousands of analysts, as being functionally stupider than just one guy and a bunch of kids at Georgetown with some Macs and a Google Earth account — to say nothing of the scads of Russian, British, French, Israeli and other nuclear watchdogs who all would have to be dumber than dirt to have missed this.
I suppose you can never go wrong underestimating the competence of a big bureaucracy like the DoD or the CIA, and I could be part of a legion of people with egg-drop soup on my face when Chinese paratroopers come ashore in Seattle while we’re cowering under a massive Chinese nuclear threat we didn’t see coming. But it’s just too hard to believe that only Karber and his students managed to notice a hidden arsenal 3/5 the size of of the entire U.S. nuclear force that was created, apparently, in between shifts of the people who were taking turns watching the Chinese.
Hopefully, this will all end up (as Kulacki noted) as a cautionary tale about research in the internet age.
Case closed — for now.