Intelligent Management of Intelligence Agencies: Beyond Accountability Ping-Pong

American Psychologist

Volumn 66, Issue 6, 2011, Pages 542-554

  Tetlock, Philip E ^a   Mellers, Barbara A ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Accountability; Expert judgment; Intelligence analysis

Indexed keywords

ARTICLE; DECISION MAKING; GOVERNMENT; HUMAN; SOCIAL BEHAVIOR; UNITED STATES;

GOVERNMENT AGENCIES; HUMANS; JUDGMENT; SOCIAL RESPONSIBILITY; UNITED STATES;

EID: 80052614050     PISSN: 0003066X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0024285     Document Type: Article

References (89)

1. Ariely D., Kahneman D., Loewenstein G. Joint comment on " When does duration matter in judgment and decision making?" (Ariely & Loewenstein, 2000). Journal of Experimental Psychology: General 2000, 129:524-529. 10.1037/0096-3445.129.4.524.
2. Arkes H.R. Overconfidence in judgmental forecasting. Principles of forecasting: A handbook for researchers and practitioners 2001, 495-516. Kluwer Academic, Norwell, MA. J.S. Armstrong (Ed.).
3. Arkes H.R., Faust D., Guilmette T.J., Hart K. Eliminating the hindsight bias. Journal of Applied Psychology 1988, 73:305-307. 10.1037/0021-9010.73.2.305.
4. Armstrong J.S. Principles of forecasting: A handbook for researchers and practitioners 2005, Kluwer Academic, Boston, MA.
5. Barber B.M., Odean T. Boys will be boys: Gender, overconfidence, and common stock investment. Quarterly Journal of Economics 2001, 116:261-292. 10.1162/003355301556400.
6. Baron J., Hershey J.C. Outcome bias in decision evaluation. Journal of Personality and Social Psychology 1988, 54:569-579. 10.1037/0022-3514.54.4.569.
7. Bayarri M., Berger J. The interplay of Bayesian and frequentist analysis. Statistical Science 2004, 19:58-80. 10.1214/088342304000000116.
8. Betts R.K. Enemies of intelligence: Knowledge and power in American national security 2009, Columbia University Press, New York, NY.
9. Bruine De Bruin W., Fischhoff B., Brilliant L., Caruso D. Expert judgments of pandemic influenza. Global Public Health 2006, 1:178-193. 10.1080/1744169060067394.
10. Budescu D., Wallsten T. Processing linguistic probabilities: General principles and empirical evidence. Psychology of Learning and Motivation 1995, 32:275-318. 10.1016/S0079-7421(08)60313-8.
11. Bueno de Mesquita B. The predictioneer's game: Using the logic of brazen self-interest to see and shape the future 2009, Random House, New York, NY.
12. Ceci S., Liker J. A day at the races: A study of IQ, expertise, and cognitive complexity. Journal of Experimental Psychology: General 1986, 115:255-266. 10.1037/0096-3445.115.3.255.
13. Christensen-Szalanski J.J., Bushyhead J.B. Physicians' use of probabilistic information in a real clinical setting. Journal of Experimental Psychology: Human Perception and Performance 1981, 7:928-935. 10.1037/0096-1523.7.4.928.
14. Christoffersen P.F., Diebold F.X. Optimal prediction under asymmetric loss. Economic Theory 1997, 13:808-817.
15. Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction Report to the President of the United States http://www.gpoaccess.gov/wmd/pdf/full_wmd_report.pdf.
16. Daubert, W., et al., Petitioners v. Merrell Dow Pharmaceuticals, Inc. (1993). Supreme Court of the United States. 509 U.S. 579; 113 S. Ct 2786; 125 L. Ed. 2d 469. March 30. 1993, ArguEd. June 28, 1993, DecidEd.
17. Erev I., Wallsten T., Budescu D. Simultaneous over and underconfidence: The role of error in judgment processes. Psychological Review 1994, 101:519-527. 10.1037/0033-295X.101.3.519.
18. Fischhoff B. Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance 1975, 1:288-299. 10.1037/0096-1523.1.3.288.
19. Fischhoff B., Slovic P., Lichtenstein S. Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance 1977, 3:552-564. 10.1037/0096-1523.3.4.552.
20. Fuller S. Thinking the unthinkable as a radical scientific project. Critical Review: A Journal of Politics and Society 2010, 22:397-413. 10.1080/08913811.2010.541691.
21. Gardner D. Future babble: Why expert predictions fail-and why we believe them anyway 2010, Penguin Books, New York, NY.
22. Gilbert D.T. How mental systems believe. American Psychologist 1991, 46:107-119. 10.1037/0003-066X.46.2.107.
23. Gilovich T. How we know what isn't so: The fallibility of human reason in everyday life 1991, Free Press, New York, NY.
24. Gilovich T., Medvec V.H., Kahneman D. Varieties of regret: A debate and partial resolution. Psychological Review 1998, 105:602-605. 10.1037/0033-295X.105.3.602.
25. Goodman-Delahunty J., Granhag P.A., Hartwig M., Loftus E.F. Insightful or wishful: Lawyers' ability to predict case outcomes. Psychology, Public Policy, and Law 2010, 16:133-157. 10.1037/a0019060.
26. Granger C.W.J., Machina M.J. Forecasting and decision theory. Handbook of economic forecasting 2006, Vol. 1:81-98. Elsevier, Oxford, England. G. Elliott, C.W.J. Granger, A. Timmermann (Eds.).
27. Griffin D.W., Ross L. Subjective construal, social inference, and human misunderstanding. Advances in experimental social psychology 1991, Vol. 24:319-359. Academic Press, San Diego, CA. M.P. Zanna (Ed.).
28. Hawkins S., Hastie R. Hindsight: Biased judgments of past events after the outcomes are known. Psychological Bulletin 1990, 107:311-327. 10.1037/0033-2909.107.3.311.
29. Henrion M., Fischhoff B. Assessing uncertainty in physical constants. American Journal of Physics 1986, 54:791-798. 10.1119/1.14447.
30. Heuer R.J. Psychology of intelligence analysis 1999, Central Intelligence Agency, Center for the Study of Intelligence, Washington, DC.
31. Heuer R.J., Pherson R.H. Structured analytic techniques for intelligence analysis 2010, CQ Press, Washington, DC.
32. Jervis R. Perception and misperception in international politics 1976, Princeton University Press, Princeton, NJ.
33. Jervis R. Why intelligence fails: Lessons from the Iranian revolution and the Iraq war 2010, Cornell University Press, Ithaca, NY.
34. Kahneman D., Klein G. Conditions for intuitive expertise: A failure to disagree. American Psychologist 2009, 64:515-526. 10.1037/a0016755.
35. Kelman H.C., Bloom A. Assumptive frameworks in international politics. Handbook of political psychology 1973, 261-295. Jossey-Bass, San Francisco, CA. J.N. Knutson (Ed.).
36. Kent S. Strategic intelligence for American world policy 1951, Princeton University Press, Princeton, NJ, (2nd printing).
37. Klayman J. Varieties of confirmation bias. Psychology of learning and motivation: Vol. 32. Decision making from a cognitive perspective 1995, 365-418. Academic Press, New York, NY. J. Busemeyer, R. Hastie, D.L. Medin (Eds.).
38. Kluger A.N., DeNisi A. Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin 1996, 119:254-284. 10.1037/0033-2909.119.2.254.
39. Kunda Z. The case for motivated reasoning. Psychological Bulletin 1990, 108:480-498. 10.1037/0033-2909.108.3.480.
40. Lerner J.S., Tetlock P. Accounting for the effects of accountability. Psychological Bulletin 1999, 125:255-275. 10.1037/0033-2909.125.2.255.
41. Lichtenstein S., Fischhoff B., Phillips L. Calibration of probabilities: The state of the art to 1980. Judgment under uncertainty: Heuristics and biases 1982, 306-334. Cambridge University Press, New York, NY. D. Kahneman, P. Slovic, A. Tversky (Eds.).
42. Lind A., Tyler T., Huo Y. Procedural context and culture: Variation in the antecedents of procedural justice judgments. Journal of Personality and Social Psychology 1997, 73:767-780. 10.1037/0022-3514.73.4.767.
43. March J., Olsen J.P. Rediscovering institutions 1989, Free Press, New York, NY.
44. McClelland G. Use of signal detection theory as a tool for enhancing performance and evaluating tradecraft in intelligence analysis. Intelligence analysis: Behavioral and social scientific foundations 2011, 83-100. National Academies Press, Washington, DC. B. Fischhoff, C. Chauvin (Eds.).
45. McGraw A.P., Mellers B.A., Ritov I. The affective costs of overconfidence. Journal of Behavioral Decision Making 2004, 17:281-295. 10.1002/bdm.472.
46. McGraw A.P., Todorov A., Kunreuther H. A policy maker's dilemma: Preventing terrorism or preventing blame. Organizational Behavior and Human Decision Processes 2011, 115:25-34. 10.1016/j.obhdp.2011.01.004.
47. Mellers B., Hertwig R., Kahneman D. Do frequency representations eliminate conjunction effects: An exercise in adversarial collaboration. Psychological Science 2001, 12:269-275. 10.1111/1467-9280.00350.
48. Merton R.K. The sociology of science 1973, University of Chicago Press, Chicago, IL.
49. Meyer J., Rowan B. Institutionalized organizations: Formal structure as myth and ceremony. The American Journal of Sociology 1977, 83:340-363. 10.1086/226550.
50. Moore D., Cain D. Over and underconfidence: When and why people underestimate (and overestimate) the competition. Organizational Behavior and Human Decision Processes 2007, 103:197-213. 10.1016/j.obhdp.2006.09.002.
51. Murphy A.H., Winkler R.L. Probability forecasting in meteorology. Journal of the American Statistical Association 1984, 79:489-500. 10.2307/2288395.
52. National Commission on Terrorist Attacks Upon the United States The 9/11 Commission report 2004, Norton, New York, NY. http://www.9-11commission.gov/report/911Report.pdf.
53. National Research Council, Board on Behavioral, Cognitive, and Sensory Sciences, Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security Scientific intelligence analysis 2011, National Academies Press, Washington, DC.
54. National Research Council, Committee on Identifying the Needs of the Forensic Sciences Community Strengthening forensic science in the United States: A path forward 2009, National Academies Press, Washington, DC.
55. Neale M.A., Bazerman M.H. The effects of framing and negotiator overconfidence on bargaining behaviors and outcomes. Academy of Management Journal 1985, 28:34-49. 10.2307/256060.
56. Nickerson R. Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology 1998, 2:175-220. 10.1037/1089-2680.2.2.175.
57. Office of the Director of National Intelligence United States intelligence community information sharing strategy http://www.dni.gov/reports/IC_Information_Sharing_Strategy.pdf.
58. Office of the Director of National Intelligence An overview of the United States intelligence community for the 111th Congress http://www.dni.gov/overview.pdf.
59. Oskamp S. Overconfidence in case-study judgments. Journal of Consulting Psychology 1965, 29:261-265. 10.1037/h0022125.
60. Posner R.A. Remaking domestic intelligence 2005, Hoover Institution Press, Stanford, CA.
61. Robinson R., Keltner D., Ward A., Ross L. Actual versus assumed differences in construal: " Naive realism" in intergroup perception and conflict. Journal of Personality and Social Psychology 1995, 68:404-417. 10.1037/0022-3514.68.3.404.
62. Sanna L.J., Schwarz N. Debiasing the hindsight bias: The role of accessibility experiences and (mis)attributions. Journal of Experimental Social Psychology 2003, 39:287-295.
63. Sporer S., Penrod S., Read D., Cutler B. Choosing, confidence, and accuracy: A meta-analysis of the confidence-accuracy relation in eyewitness identification studies. Psychological Bulletin 1995, 118:315-327. 10.1037/0033-2909.118.3.315.
64. Stael von Holstein C. Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance 1972, 8:139-158. 10.1016/0030-5073(72)90041-4.
65. Staw B.M., Sandelands L.E., Dutton J.E. Threat rigidity effects in organizational behavior: A multilevel analysis. Administrative Science Quarterly 1981, 26:501-524. 10.2307/2392337.
66. Suedfeld P., Tetlock P.E. Integrative complexity of communications in international crises. Journal of Conflict Resolution 1977, 21:169-184.
67. Susskind R. One percent doctrine 2006, Simon and Schuster, New York, NY.
68. Swets J.A., Dawes R.M., Monahan J. Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest 2000, 1:1-26. 10.1111/1529-1006.001.
69. Taleb N.N. The black swan: The impact of the highly improbable 2007, Random House, New York, NY.
70. Taleb N.N. Anti-fragility: How to live in a world we don't understand 2011, Random House, New York, NY.
71. Tetlock P.E. Policy-makers' images of international conflict. Journal of Social Issues 1983, 39:67-86.
72. Tetlock P.E. The impact of accountability on judgment and choice: Toward a social contingency model. Advances in experimental social psychology 1992, Vol. 25:331-376. Academic Press, New York, NY. M. Zanna (Ed.).
73. Tetlock P.E. Close-call counterfactuals and belief system defenses: I was not almost wrong but I was almost right. Journal of Personality and Social Psychology 1998, 75:639-652. 10.1037/0022-3514.75.3.639.
74. Tetlock P.E. Cognitive biases and organizational correctives: Do both disease and cure depend on the ideological beholder?. Administrative Science Quarterly 2000, 45:293-326. 10.2307/2667073.
75. Tetlock P.E. Expert local judgment: How good is it? How can we know? 2005, Princeton University Press, Princeton, NJ.
76. Tetlock, P. E. (2009, September-October). Reading tarot on K Street. The National Interest, 57-67.
77. Tetlock P.E. Second thoughts about expert political judgment: Reply to the symposium. Critical Review 2010, 22.
78. Tetlock P.E., Kristel O., Elson B., Green M., Lerner J. The psychology of the unthinkable: Taboo trade-offs, forbidden base rates, and heretical counterfactuals. Journal of Personality and Social Psychology 2000, 78:853-870. 10.1037/0022-3514.78.5.853.
79. Tetlock P.E., Mellers B.A. Structuring accountability systems in organizations: Key trade-offs and critical unknowns. Intelligence analysis: Behavioral and social scientific foundations 2011, 249-270. National Academies Press, Washington, DC. Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security (Ed.).
80. Tetlock P.E., Mitchell G. Adversarial collaboration aborted, but our offer still stands. Research in organizational behavior 2009, Vol. 29:77-79. Elsevier, New York, NY. B.M. Staw, A. Brief (Eds.).
81. Tetlock P.E., Mitchell G. Implicit bias and accountability systems: What must organizations do to prevent discrimination?. Research in organizational behavior 2009, Vol. 29:3-38. Elsevier, New York, NY. B.M. Staw, A. Brief (Eds.).
82. Tetlock, P. E., Mitchell, G., & Anastasopoulos, J. (2011). Embracing or rejecting emerging mind-reading technologies: Intuitive prosecutors play ideological favorites but most stop when caught. Unpublished manuscript, University of Pennsylvania.
83. Theriault S.M. Party polarization in the US Congress. Party Politics 2006, 12:483-503. 10.1177/1354068806064730.
84. Theriault S. Party polarization and the rise of partisan voting in U.S. House elections. American Politics Research 2008, 36:62-84.
85. Thibaut J., Walker L. Procedural justice: A psychological analysis 1975, Erlbaum, Hillsdale, NJ.
86. Winkler R. Evaluating probabilities: Asymmetric scoring rules. Management Science 1994, 40:1395-1405. 10.1287/mnsc.40.11.1395.
87. Wohlstetter R. Pearl Harbor: Warning and decision 1962, Stanford University Press, Stanford, CA.
88. Wolfers J., Zitzewitz E. Prediction markets. Journal of Economic Perspectives 2004, 18:107-126. 10.1257/0895330041371321.
89. Zegart A. Spying blind: The CIA, the FBI, and the origins of 9/11 2007, Princeton University Press, Princeton, NJ.