Think-3 Investment Idea Sharing Platform

"The Numbers Guys" column in WSJ
July 7, 2005

Several news organizations ran some scary headlines recently about threats from terrorism and war. The stories cited a new survey that found, in part, a 29.2% chance of a nuclear attack in the next decade. The survey also put odds on an attack using a biological weapon, such as anthrax.

But how do you predict the likelihood of an event that has never happened before?

The past is the baseline for predicting the future. In forecasting company revenue, economic indicators and hurricane counts, experts start with prior numbers and adjust them higher or lower to reflect expected future trends. When it comes to estimating the chance of a terrorist attack using biological or nuclear weapons, it's hard to go beyond an educated guess.

Two weeks ago, Sen. Richard Lugar (R., Ind.), chairman of the Senate Foreign Relations Committee, released the results of an ambitious survey of arms experts. The study was conducted in late 2004 and early 2005. On average, the 85 respondents predicted a 29.2% chance of a nuclear attack in the next decade, with 79% saying that such an attack was more likely to be carried out by terrorists than by a government. Sen. Lugar said in the report that "the estimated combined risk of a WMD attack over five years is as high as 50%. Over 10 years this risk expands to as much as 70%."

The survey received wide press coverage. CNN reported that "arms experts say there is a high chance of an attack with a weapons of mass destruction. A new survey out says there's a 70% chance of a WMD attack somewhere in the world within the next 10 years." In a teaser at the beginning of Tucker Carlson's MSNBC show, he said, "Just how vulnerable are we to a nuclear attack? The answer may terrify you" -- though during the show he questioned the report for potentially raising hysteria. Agence France Presse headlined its report, "Survey of weapons experts finds sharp rise in chance of nuclear attack," though there's no clear reference to what the chance of a nuclear attack used to be. Reuters also ran the story, quoting a Lugar aide who said 70% was "a very conservative estimate." (Numbers Guy reader John Pinto saw some of the reports and suggested I write about the Lugar study.)

Other analysts have taken similar approaches. Last August, former Secretary of Defense William Perry told New York Times columnist Nicholas Kristof there was a 50% chance of a nuclear terrorist attack by 2010. Mr. Kristof also cited a widely quoted prediction by Graham Allison, a Harvard professor who wrote in his book "Nuclear Terrorism" that there is a greater than 50% chance of a terrorist attack in the next 10 years, barring major mitigating steps. (Mr. Kristof placed a $5 bet with Mr. Allison against those odds, writing, "If I were guessing wildly, I would say a 20% risk over 10 years. In any case, if I lose the bet, then I'll probably be vaporized and won't have much use for money.")

ABOUT THIS COLUMN The Numbers Guy examines numbers and statistics in the news, business, politics and health. Some numbers are flat-out wrong, misleading or biased. Others are valid and useful, helping us to make informed decisions. As the Numbers Guy, I will try to sort through which numbers to trust, question or discard altogether. And I'd like to hear from you at numbersguy@wsj.com. I'll post and respond to your letters. WSJ.com subscribers can sign up to receive email when new columns are published (nonsubscribers click here to sign up), and you can read more columns at WSJ.com/NumbersGuy.

Bias in the Results

It's understandable why politicians and the media would turn to arms experts for their predictions, because they're about as well-equipped as anyone to weigh the available data. Yet there are also drawbacks. As well-informed as arms experts are, and as well-intentioned, I'd argue they have a natural bias toward overstating risk -- greater risk increases the value of their expertise, and, therefore, their prominence and even funding. Politicians who commission such predictions likely do so because they want to raise awareness, a goal best served by alarming results.

"If you survey the boys who cry wolf, they cry wolf," said Anthony Cordesman, former director of intelligence assessment in the Office of the Secretary of Defense and current Arleigh A. Burke Chair in Strategy at the Center for Strategic and International Studies in Washington, D.C. Professor Cordesman was surveyed by Sen. Lugar. "That doesn't mean there aren't wolves and they won't show up tomorrow, but it certainly doesn't help you know when they'll show up." Mr. Cordesman said in an email that he didn't answer many of the questions about percentages because "the questions were too imprecise to have meaningful results, and were semantically loaded in ways that would produce misleading and exaggerated probabilities of attack." He declined to make available his survey responses.

"It was not our intention" to elicit overstated terrorism risks, says Dan Diller, who oversaw the survey. Mr. Diller, deputy staff director of the Senate Foreign Relations Committee, pointed out some of the surveyed arms experts predicted probabilities of attack that were low, or even 0%.

The report also stated, "This study is not meant to be a scientific poll of the entire national security community." The survey excluded administration officials, "given the provocative nature of some questions and other factors," and many of the respondents were former officials and think-tank scholars. As a result, its results don't reflect much information outside the public domain. "We feel like the folks who served in past administrations and scholars who spent a good deal of time looking at proliferation issues are equipped to answer these types of questions," Mr. Diller said.

Predicting the Unpredictable

But how did the experts attempt to predict the chance of unprecedented catastrophe? To find out, I emailed about two-thirds of the 85 respondents last week, and heard back from more than a half-dozen. None said they used any sort of computation. "My answers were not based on statistics or models," said Michael Krepon, co-founder of the Henry L. Stimson Center, a nonpartisan Washington think tank focused on international security. "They were based on judgments derived from working in the field and assessing how well or badly efforts to control bad things are going."

James Dobbins called his responses "guesstimates." Mr. Dobbins, director of RAND's international security and defense policy center, added, "I expect it was not a scientifically selected sampling, and that a different group of equally informed people, perhaps from the technical rather than policy community, might come to different conclusions." John Wolf, president of the Eisenhower Fellowships, said his response was an "educated guess" with "lots" of uncertainty. (Respondents declined to reveal their survey responses to me.)

I also asked respondents what value the survey's results had, and some of their responses made plain the dual role of the survey as prediction and as public-awareness tool. Mr. Wolf called it a "wake up call to policy makers all around the world."

"When you measure something, you also change it, and just as speeches, news articles, polls, etc. impact on politics, this survey has/will do the same," Gerald Steinberg, a professor of political studies at Bar Ilan University in Israel, and one of the Lugar survey's respondents, said in an email. He added that the value of the survey is "in drawing public attention to the continued threats."

Richard Falkenrath, senior fellow in foreign policy studies at the Brookings Institution, said in a telephone interview that the study has some value in comparing probabilities of different attacks -- for instance, experts' median predicted probability for a radiological attack in the next decade was 40%, twice that for a biological attack, a result that could direct government priorities. "You have some leg to stand on when you're making comparative probability estimates, but you have no leg to stand on when you're predicting an absolute number," says Dr. Falkenrath, who participated in the study.

A 'Considered Judgment'

None of this means the experts haven't extensively studied terrorism nor that they don't have good reasons for their predictions. Prof. Steinberg sent me a lengthy explanation of the reasoning behind his survey responses, comparing the process to an effort he was once involved in to "estimate the probability of catastrophic malfunctions of complex technical systems" like nuclear reactors. "The process involves attempting to build relevant scenarios, breaking down the various dimensions of each scenario, and estimating the probability for each stage and factor -- such as the odds of a given terror group obtaining the necessary materials or completed weapons, their ability to deliver them, the role of deterrence and retaliation on their decision making, etc., and, on the other side, intelligence capabilities to detect diversions, active and passive defense, etc.," he says.

Professor Allison, the Harvard professor who was referenced in the New York Times column, said that there is "no established methodology; no statistical sample from which to derive," so he looked at the "who? what? where? when? how?" and arrived at a "considered judgment," as explained in his book.

In the book, Mr. Allison buttresses his argument that a nuclear attack is likely in the next decade with frightful imagery and anecdotal support. The introduction notes that much is uncertain about future terrorist attacks, but what is certain is the devastation such an attack would bring to a U.S. city -- he describes a firestorm engulfing Rockefeller Center and Carnegie Hall, or seeing San Francisco landmarks vaporized. Mr. Allison even quotes billionaire investor Warren Buffett, who controls the holding company Berkshire Hathaway Inc. and whom he calls "a legendary oddsmaker in pricing insurance policies for unlikely but catastrophic events like earthquakes," as having said that a nuclear terrorist attack is "the ultimate depressing thing. It will happen. It's inevitable. I don't see any way that it won't happen."

As it turns out, when Mr. Buffett said that in a 2002 interview with Fortune, he was talking about any nuclear explosion, not just a terrorist attack. He also told me in a phone interview, "When I say it is certain to happen, I am talking about the next century, not any five-year or 10-year period."

The reported probability of WMD attacks "can't be meaningful," Mr. Buffett said. "I would not regard any specific number as being meaningful. I would regard the importance of reducing the probability as terribly meaningful."