Accuracy of forecasts in strategic intelligence

Proceedings of the National Academy of Sciences of the United States of America

Volumn 111, Issue 30, 2014, Pages 10984-10989

  Mandel, David R ^a   Barnes, Alan ^b  

^a DEFENCE RESEARCH AND DEVELOPMENT CANADA   (Canada)

^b Privy Council Office   (Canada)

Author keywords

Forecasting; Intelligence analysis; Prediction; Quality control; Recalibration

Indexed keywords

ACCURACY; ARTICLE; DECISION MAKING; EXPERIENCE; FORECASTING; HUMAN; INTELLIGENCE; OPTIMISM; PRIORITY JOURNAL;

EID: 84905011515     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1406138111     Document Type: Article

References (48)

1. Treverton GF (2008) Analyzing Intelligence: Origins, Obstacles, and Innovations, eds George RZ, Bruce JB (Georgetown Univ Press, Washington, DC), pp 91-106.
2. Fingar T (2011) Intelligence Analysis: Behavioral and Social Scientific Foundations, eds Fischhoff B, Chauvin C (Natl Acad Press, Washington, DC), pp 3-27.
3. Kent S (1964) Words of estimative probability. Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/ csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national- estimates-collected-essays/6words.html.
4. Tetlock PE, Mellers BA (2011) Intelligent management of intelligence agencies: beyond accountability ping-pong. Am Psychol 66(6):542-554.
5. Rieber S (2004) Intelligence analysis and judgmental calibration. Int J Intell Counter-Intell 17(1):97-112.
6. Reuters (October 28, 2011) Fiscal 2011 U.S. intelligence budget was 54.6 bln. Available at http://www.reuters.com/article/2011/10/28/usa-intelligence- budgetidUSN1E79R1CN20111028.
7. Betts RK (2007) Enemies of Intelligence: Knowledge and Power in American National Security (Columbia Univ Press, New York), 241 pp.
8. Horowitz MC, Tetlock PE (2012) Trending upward: How the intelligence community can better see into the future. Foreign Policy (Sept. 6). Available at http://www.foreignpolicy.com/articles/2012/09/06/trending-upward.
9. Weiss C (2008) Communicating uncertainty in intelligence and other professions. Int J Intell CounterIntell 21(1):57-85.
10. Fischhoff B, Chauvin C, eds (2011) Intelligence Analysis: Behavioral and Social Scientific Foundations (Natl Acad Press, Washington, DC), 338 pp.
11. Derbentseva N, McLellan L, Mandel DR (2011) Issues in Intelligence Production: Summary of Interviews with Canadian Intelligence Managers (Def Res and Dev Can, Toronto), Tech Rep 2010-144.
12. National Research Council (2010) Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary (Natl Acad Press, Washington, DC), 114 pp.
13. National Research Council (2011) Intelligence Analysis for Tomorrow: Advances from the Behavioral and Social Sciences (Natl Acad Press, Washington, DC), 102 pp.
14. National Commission on Terrorist Attacks upon the United States (2004) The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks upon the United States (US Gov Print Off, Washington, DC), 585 pp.
15. Lichtenstein S, Fischhoff B, Phillips LD (1982) Judgment Under Uncertainty: Heuristics and Biases, eds Kahneman D, Slovic P, Tversky A (Cambridge Univ Press, New York), pp 306-334.
16. Fischhoff B, Slovic P, Lichtenstein S (1977) Knowing with certainty: The appropriateness of extreme confidence. J Exp Psychol Hum Percept Perform 3(4):552-564.
17. Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton Univ Press, Princeton), 321 pp.
18. Dawes RM (1979) The robust beauty of improper linear models in decision making. Am Psychol 34(7):571-582.
19. Meehl PE (1954) Clinical versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence (Univ Minn Press, Minneapolis), 149 pp.
20. Murphy AH, Winkler RL (1984) Probability forecasting in meteorology. J Am Stat Assoc 79(387):489-500.
21. Charba JP, Klein WH (1980) Skill in precipitation forecasting in the National Weather Service. Bull Am Meteorol Soc 61(12):1546-1555.
22. Keren G (1987) Facing uncertainty in the game of bridge: A calibration study. Organ Behav Hum Decis Process 39(1):98-114.
23. Berner ES, Graber ML (2008) Overconfidence as a cause of diagnostic error in medicine. Am J Med 121(5, Suppl):S2-S23. (Pubitemid 351608076)
24. Goodman-Delahunty J, Granhag PA, Hartwig M, Loftus EF (2010) Insightful or wishful: Lawyers' ability to predict case outcomes. Psychol Public Policy Law 16(2):133-157.
25. Murphy AH (1973) A new vector partition of the probability score. J Appl Meteorol 12(4):595-600.
26. Swets JA (1986) Form of empirical ROCs in discrimination and diagnostic tasks: Implications for theory and measurement of performance. Psychol Bull 99(2):181-198.
27. Yaniv I, Yates JF, Smith JEK (1991) Measures of discrimination skill in probabilistic judgment. Psychol Bull 110(3):611-617.
28. Yates JF (1990) Judgment and Decision Making (Prentice Hall, Englewood Cliffs, NJ), 430 pp.
29. Cox DR (1958) Two further applications of a model for binary regression. Biometrika 45(3-4):562-565.
30. Budescu DV, Johnson TR (2011) A model-based approach for the analysis of the calibration of probability judgments. Judgm Decis Mak 6(8):857-869.
31. Lichtenstein S, Fischhoff B (1977) Do those who know more also know more about how much they know? Organ Behav Hum Perform 20(2):159-183.
32. Karvetski CW, Olson KC, Mandel DR, Twardy CR (2013) Probabilistic coherence weighting for optimizing expert forecasts. Decis Anal 10(4):305-326.
33. Shlomi Y, Wallsten TS (2010) Subjective recalibration of advisors' probability estimates. Psychon Bull Rev 17(4):492-498.
34. Turner BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2014) Forecast aggregation via recalibration. Mach Learn 95(3):261-289.
35. Baron J, Mellers BA, Tetlock PE, Stone E, Ungar LH (2014) Two reasons to make aggregated probability forecasts more extreme. Decis Anal 11(2):133-145.
36. Pearce J, Ferrier S (2000) Evaluating the predictive performance of habitat models developed using logistic regression. Ecol Modell 133(3):225-245.
37. Akaike H (1974) A new look at the statistical model identification. IEEE Trans Automat Contr 19(6):716-723.
38. Koehler DJ, Brenner L, Griffin D (2002) Heuristics and Biases: The Psychology of Intuitive Judgment, eds Gilovich T, Griffin D, Kahneman D (Cambridge Univ Press, Cambridge, UK), pp 686-715.
39. Wallsten TS, Budescu DV, Zwick R (1993) Comparing the calibration and coherence of numerical and verbal probability judgments. Manage Sci 39(2):176-190.
40. Fischhoff B, Bruine de Bruin W (1999) Fifty-fifty=50%? J Behav Decis Making 12(2):149-163.
41. Karmarkar US (1978) Subjectively weighted utility: A descriptive extension of the expected utility model. Organ Behav Hum Perform 21(1):61-72.
42. Tetlock PE, Kim JI (1987) Accountability and judgment processes in a personality prediction task. J Pers Soc Psychol 52(4):700-709.
43. Tetlock PE (1985) Accountability: A social check on the fundamental attribution error. Soc Psychol Q 48(3):227-236.
44. Chaiken S (1980) Heuristic versus systematic information processing and the use of source versus message and cues in persuasion. J Pers Soc Psychol 39(5):752-766.
45. Hagafors R, Brehmer B (1983) Does having to justify one's decisions change the nature of the judgment process? Organ Behav Hum Perform 31(2):223-232.
46. Arkes HR, Kajdasz J (2011) Intelligence Analysis: Behavioral and Social Scientific Foundations, eds Fischhoff B, Chauvin C (Natl Acad Press, Washington, DC), pp 143-168.
47. Mellers B, et al. (2014) Psychological strategies for winning geopolitical forecasting tournaments. Psychol Sci 25(5):1106-1115.
48. Kadane J, Fischhoff B (2013) A cautionary note on global recalibration. Judgm Decis Mak 8(1):25-28. Canada