Assessing uncertainty in intelligence

Intelligence and National Security

Volumn 27, Issue 6, 2012, Pages 824-847

  Friedman, Jeffrey A ^a   Zeckhauser, Richard ^a  

^a HARVARD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84871324227     PISSN: 02684527     EISSN: 17439019     Source Type: Journal    
DOI: 10.1080/02684527.2012.708275     Document Type: Article

References (77)

1. A related example was the intelligence community's ten-year effort to locate Osama bin Laden. There was a correct answer to the question of where bin Laden was hiding, but uncertainty about bin Laden's location remained even up to the point where President Obama authorized a raid against his compound.
2. Rob Johnston, Analytic Culture in the US Intelligence Community (Washington, DC: Center for the Study of Intelligence 2005) p.xviii.
3. Cf. Michael Heazle, 'Policy Lessons from Iraq on Managing Uncertainty in Intelligence Assessment: Why the Strategic/Tactical Distinction Matters', Intelligence and National Security 25/3 (2010) p.297, which refers to a similar aspect of analytic culture that the author calls 'positivist perceptions of knowledge' expressed as 'the truth is out there and it can be known'. This aspect of analytic culture is as old as the estimative process itself.
4. Roger Hilsman, Jr., 'Intelligence and Policy-Making in Foreign Affairs', World Politics 5/1 (1952) pp.11 and 13,
5. criticizes 'the implied assumption. that truth is obvious once all the facts are known', a point of view 'accepted with so little question'.
6. Paul R. Pillar levels this critique in his book Intelligence and US Foreign Policy: Iraq, 9/11, and Misguided Reform (New York: Columbia 2011) p.4.
7. For examples, see Allen Dulles, The Craft of Intelligence (New York: Harper and Row 1963) p.155
8. Harold P. Ford, Estimative Intelligence: The Purposes and Problems of National Intelligence Estimating (Lanham, MD: Defense Intelligence College 1993) p.86
9. Mark M. Lowenthal, Intelligence: From Secrets to Policy, 4th ed. (Washington, DC: CQ Press 2009) p.148
10. Kristan J. Wheaton, 'Evaluating Intelligence: Answering Questions Asked and Not', International Journal of Intelligence and CounterIntelligence 22/4 (2009) pp.614-31
11. John A. Gentry, 'Assessing Intelligence Performance' in Loch K. Johnson (ed.) Oxford Handbook of National Security Intelligence (New York: Oxford 2010) pp.93-97
12. Thomas J. Fingar, Reducing Uncertainty: Intelligence Analysis and National Security (Stanford, CA: Stanford University Press 2011) p.3.
13. This is not to say that the analyst stakes her entire reputation on each estimate, but rather to point out that the strategy which maximizes the analyst's payoff across many predictions is to make single-outcome judgments on each one.
14. For a related example regarding the fall of the Shah of Iran, see Robert Jervis, Why Intelligence Fails: Lessons from the Iranian Revolution and the Iraq War (Ithaca, NY: Cornell 2010) ch.2. On consequence neglect more generally
15. see Alan Berger et al., 'The Five Neglects: Risks Gone Amiss' in Howard Kunreuther and Michael Useem (eds.) Learning from Catastrophes (Philadelphia, PA: Wharton 2010) pp.83-99.
16. Similarly, Mark Lowenthal's book, Intelligence, contains thoughtful analyses of many issues raised in this article. But he also writes (p.148) that the 'accuracy' of a prediction should be judged according to whether its likelihood is 'something more than 50 per cent and something less than 100 per cent' - even though it can, of course, be perfectly accurate to say that something has a probability lower than 50 per cent. Ford's Estimative Intelligence reinforces many of this article's themes, while at the same time stating that 'one of the main purposes of national intelligence estimating is to lessen policymakers' uncertainties about the world' (p.179).
17. Richard K. Betts's article, 'Analysis, War, and Decision: Why Intelligence Failures Are Inevitable', World Politics 31/1 (1978) pp.61-89
18. is one of the most well-cited treatments of the dangers of making estimates with too much certainty, yet he does imply that certainty should be the goal: 'It is the role of intelligence to extract certainty from uncertainty' (p.69).
19. Kristan Wheaton's article, 'The Revolution Begins on Page Five: The Changing Nature of NIEs', International Journal of Intelligence and CounterIntelligence 25/2 (2012), pp.330-49
20. discusses the challenges of conveying uncertainty in an accurate fashion, but states (p.331) that the purpose of National Intelligence Estimates is ultimately 'to reduce national security policy-makers' level of uncertainty'.
21. As one intelligence scholar recently stated: 'The NIE is arguably the highest form of the intelligence art'.
22. Wheaton, 'The Revolution Begins on Page Five', p.340.
23. A recent discussion of NIEs can be found in Loch K. Johnson, The Threat on the Horizon: An Inside Account of America's Search for National Security after the ColdWar (NewYork:Oxford 2011) pp.164-85.
24. There are actually 426 entries in the database, but 47 were dropped for various reasons: some entries are not estimates but update memoranda, some remain heavily classified, and some appear in the database twice. The database was accessed through 5foia.cia.gov4 between October 2010 and May 2011.
25. See, for example, Willis C. Armstrong et al., 'The Hazards of Single-Outcome Forecasting', Studies in Intelligence 28/3 (1984) pp.57-70.
26. Richards J. Heuer and Randolph H. Pherson, Structured Analytic Techniques for Intelligence Analysis (Washington, DC: CQ Press 2011) ch.7.1.
27. US Government, A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis (Washington, DC: US Government Printing Office 2009) p.14.
28. Richards J. Heuer, The Psychology of Intelligence Analysis (Washington, DC: Center for the Study of Intelligence 1999) p.95.
29. Heuer and Pherson, Structured Analytic Techniques, p.160.
30. US Government, Tradecraft Primer, p.15.
31. By way of analogy, if the best economic models predict a 10 per cent chance of a recession in a given year, then this does not constitute a 'weak possibility' or a scenario that should be disproved. The important thing for economic forecasters (both in the private and the public sectors) is to assess the chances of recession accurately. The same is true for doctors assessing potential complications during surgery, meteorologists predicting the chances of inclement weather, and experts in many other fields.
32. More broadly, an increasing amount of recent intelligence work has been devoted to techniques of 'competitive analysis', which are designed to pit different viewpoints against one another. See Richard L. Russell, 'Competitive Analysis' in Johnson (ed.) Oxford Handbook of National Security Intelligence.
33. Richard K. Betts, Enemies of Intelligence: Knowledge and Power in American National Security (New York: Columbia 2007) p.101.
34. A fundamental introductory text on this subject is Howard Raiffa's Decision Analysis: Introductory Lectures on Choices under Uncertainty (Reading, MA: Addison-Wesley 1968).
35. A more recent text is John W. Pratt, Raiffa, and Robert Schlaifer, Introduction to Statistical Decision Theory (Cambridge, MA: MIT 1995).
36. See also Daniel Kahneman, Paul Slovic, and Amos Tversky (eds.) Judgment under Uncertainty: Heuristics and Biases (New York: Cambridge University Press 1982)
37. Daniel Kahneman, Thinking Fast and Slow (New York: Farrar, Straus, and Giroux 2011) which examine behavioral aspects of decision-making under uncertainty and discuss ways to mitigate relevant problems.
38. The decision theory literature often deals with subjects such as doctors prescribing medical treatments, business executives making economic forecasts, and gamblers assessing their prospects. In these cases and many others, decision-makers face uncertainties about the current and future states of the world that will influence their choice of action. Sherman Kent explains how intelligence analysis and foreign policy relate to other kinds of decision-making under uncertainty in Strategic Intelligence for American World Policy (Hamden, CT: Archon 1965) pp.58-61
39. See also Walter Laqueur, 'The Question of Judgment: Intelligence and Medicine', Journal of Contemporary History 18/4 (1983) pp.533-48
40. Charles Weiss, 'Communicating Uncertainty in Intelligence and Other Professions', International Journal of Intelligence and CounterIntelligence 2/11 (2008) pp.57-85
41. David T. Moore, Sensemaking: A Structure for an Intelligence Revolution (Washington, DC: National Defense Intelligence College 2011) pp.95-9.
42. Heuer, Psychology of Intelligence Analysis, ch.10.
43. Ford, in Estimative Intelligence (p.21 cf. pp.78, 101) writes that NIEs are often criticized for being 'wishy-washy', because the coordination process tends to produce 'coordinated mush'. His rough dichotomy between mush and split decisions, however, is a dichotomy that is not necessary when thinking in terms of probability distributions, rather than of discrete, single-outcome judgments.
44. Cf. Gregory Treverton, Reshaping National Intelligence for an Age of Information (Santa Monica, CA: RAND 2001) p.204
45. Roger Z. George, 'Beyond Analytic Tradecraft', International Journal of Intelligence and CounterIntelligence 23/2 (2010) pp.300-1
46. Betts, Enemies of Intelligence, pp.32, 101.
47. Heuer and Pherson, Structured Analytic Techniques, p.163.
48. For example, 'We assess with high confidence that until fall 2003, Iranian military entities were working under government direction to develop nuclear weapons'; 'We judge with high confidence that the halt lasted at least several years'; 'We continue to assess with moderate-tohigh confidence that Iran does not currently have a nuclear weapon'.
49. It seems as though the real operative distinction between words of likelihood and words of confidence in the Iran NIE is that the latter are used to make predictions about events that had already happened (such as whether or not Iran had stopped and/or restarted its nuclear research) while the former is used to make predictions about the future (e.g., 'Iran probably would use covert facilities' for a given purpose; 'Iran probably would be technically capable' of producing a weapon in a given time frame).
50. None of this discussion implies that intelligence collectors should not seek to eliminate uncertainty. But when analysts confront uncertainty, they must assess their information in light of its limitations.
51. High confidence implies the estimate will change little in response to new information, such as close inspection of the coin. This is an important point to which the article returns below.
52. See Wheaton, "The Revolution Begins on Page Five' for a recent and thorough discussion of this debate, which also covers the 2007 Iran NIE.
53. For example, Prospects for the South African Transition (1994) estimates a 70 per cent chance that an election will occur on schedule; Russia Over the Next Four Years (1992) traces several contingencies and identifies the chances of two of them as 'slightly better than even' and 'one in three'; and Soviet Ballistic Missile Defense (1982) assesses the odds that the Soviets will abrogate the Anti-Ballistic Missile Treaty at 10-20 per cent.
54. For instance, several estimates discuss different 'illustrative force models' for how Soviet strategic forces might evolve without giving a sense of the likelihood that each model is correct. Examples of NIEs that lay out more than three possible scenarios while providing the reader with almost no sense of their likelihoods include Soviet Policy toward theWest (1989); The Changing Sino-Soviet Relationship (1985); and Soviet Military Options in the Middle East (1975).
55. This is not to say that other estimates do not share some of its characteristics. For instance the NIE Implications of Alternative Soviet Futures (1991) presents a range of potential scenarios in a concise table, though without explicit probabilities. The NIE Russia over the Next Four Years (1992) gives numeric probabilities over three or more possible outcomes but does not present the information in a concise table.
56. Sherman Kent wrote one of the earliest and best-known articles on this subject: 'Words of Estimative Probability', Studies in Intelligence 8/4 (1964) pp.49-65
57. more recent examples include Weiss, "Communicating Uncertainty'
58. Joab Rosenberg, "The Interpretation of Probability in Intelligence Estimation and Strategic Assessment', Intelligence and National Security 23/2 (2008) pp.139-52.
59. A reasonable compromise between clearly expressing likelihood and not giving an undue sense of scientific precision might be for analysts to assess most probabilities in intervals of five percentage points (e.g., 20 per cent or 85 per cent), while expressing small probabilities in tighter intervals (e.g., 1 per cent or 2 per cent).
60. For literature reviews on this subject, see Paul H. Garthwaite, Joseph B. Kadane, and Anthony O'Hagan, "Statistical Methods for Eliciting Probability Distributions', Journal of the American Statistical Association 100/470 (2005) pp.680-700
61. Frederick Mosteller and Cleo Youtz, "Quantifying Probabilistic Expressions', Statistical Science 5/1 (1990) pp.2-34.
62. Another set of cases that is relevant to this argument is a series of NIEs written about potential security concerns accompanying presidential visits to various countries: e.g., Security Conditions in China (1972), Security Conditions in Mexico (1970); Security Conditions in Mexico City (1968); and The President's Trip to Central America (1968). Each of these estimates the threat to the president's safety to be low - but this is exactly the kind of issue for which consumers of intelligence rely on having a more fine-grained sense of what the phrase 'low probability' actually means.
63. This has been a long-standing critique: see, for example, Senate Committee on Armed Services, Preparedness Investigating Subcommittee (Stennis Report), Investigation of the Preparedness Program, 88th Cong., 1st Sess. (1963), pp.5, 10.
64. For a more recent example, see MG Michael Flynn, Fixing Intel: A Blueprint for Making Intelligence Relevant in Afghanistan (Washington, DC: Center for a New American Security 2010) p.7.
65. Jennifer E. Sims, "A Theory of Intelligence and International Politics' in Gregory F. Treverton and Wilhelm Agrell (eds.) National Intelligence Systems: Current Research and Future Prospects (New York: Cambridge 2009) p.81.
66. US Army Field Manual 2-22.3, Human Intelligence Collector Operations (Washington, DC: Department of the Army 2006), para. 12-12 and Appendix B.
67. Ibid, para. 12-3.
68. Ibid., para. B-2. As a further indication that 'information content' is a way of judging a report's accuracy, it is worth noting that the categories along this scale are defined as 'confirmed', 'probably true', 'possibly true', 'doubtfully true', 'improbable', and 'cannot be judged'.
69. Ibid., Appendix B and para. 12-3.
70. This problem is compounded by the way case officers and analysts are trained to determine what information to pass along. As intelligence professionals make this decision, they are instructed to think in terms of 'thresholds' for what makes intelligence significant. It certainly makes sense to say that NIEs should represent the most significant information, but the threshold concept blurs the line between significance and reliability.
71. Ariel Levite, Intelligence and Strategic Surprises (New York: Columbia 1987) p.72
72. cf. Roberta Wohlstetter, Pearl Harbor: Warning and Decision (Stanford, CA: Stanford University Press 1964) pp.368, 386.
73. See Stennis Report, pp.5, 10
74. Klaus Knorr, 'Failures in National Intelligence Estimates: The Case of the Cuban Missiles', World Politics 16/3 (1964) pp.455-67
75. Shlomo Nakdimon, Low Probability (Tel Aviv: Revivim 1982)
76. Amy B. Zegart, Spying Blind: The CIA, the FBI, and the Origins of 9/11 (Stanford, CA: Stanford University Press 2007) ch.5
77. Betts, Enemies of Intelligence, p.22.