The psychology of intelligence analysis: Drivers of prediction accuracy in world politics

Journal of Experimental Psychology: Applied

Volumn 21, Issue 1, 2015, Pages 1-14

  Mellers, Barbara ^a   Stone, Eric ^a   Atanasov, Pavel ^a   Rohrbaugh, Nick ^a   Emlen Metz, S ^a   Ungar, Lyle ^b   Bishop, Michael M ^a   Horowitz, Michael ^a   Merkle, Ed ^c   Tetlock, Philip ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

^c UNIVERSITY OF MISSOURI   (United States)

Author keywords

Accuracy; Forecasting; Predictions; Probability judgment; Skill

Indexed keywords

ADULT; DECISION MAKING; FEMALE; FORECASTING; HUMAN; INTELLIGENCE; MALE; POLITICS; PROBABILITY; PSYCHOLOGICAL AND PSYCHIATRIC PROCEDURES; STATISTICAL MODEL;

ADULT; FEMALE; FORECASTING; HUMANS; INTELLIGENCE; JUDGMENT; MALE; MODELS, STATISTICAL; POLITICS; PROBABILITY; PSYCHOLOGICAL TECHNIQUES;

EID: 84925704214     PISSN: 1076898X     EISSN: None     Source Type: Journal    
DOI: 10.1037/xap0000040     Document Type: Article

References (73)

1. Almond, G. A., & Genco, S. J. (1977). Clouds, clocks, and the study of politics. World Politics, 29, 489-522. http://dx.doi.org/10.2307/2010037
2. Arkes, H. R. (2001). Overconfidence in judgmental forecasting. In S. Armstrong (Ed.), Principles of forecasting (pp. 495-515). New York, NY: Springer.
3. Atanasov, P., Rescober, P., Stone, E., Servan-Schreiber, E., Tetlock, P., Ungar, L., & Mellers, B. (2014). Crowd forecasting with prediction polls and prediction markets. Manuscript submitted for publication.
4. Baron, J. (2000). Thinking and deciding (3rd ed.). New York, NY: Cambridge University Press.
5. Baron, J., Scott, S. E., Fincher, K., & Metz, S. E. (2014). Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory & Cognition. Advance online publication. http://dx.doi.org/10.1016/j.jarmar.2014.09.003
6. Bors, D. A., & Stokes, T. L. (1998). Raven's Advanced Progressive Matrices: Norms for first-year university students and the development of a short form. Educational and Psychological Measurement, 58, 382-398. http://dx.doi.org/10.1177/0013164498058003002
7. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3. http://dx.doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
8. Bueno de Mesquita, B. (2009). The predictioneer's game: Using the logic of brazen self-interest to see and shape the future. New York, NY: Random House.
9. Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York, NY: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511571312
10. Cattell, R. B. (1963). Theory of fluid and crystallized intelligence: A critical experiment. Journal of Educational Psychology, 54, 1-22. http://dx.doi.org/10.1037/h0046743
11. Cattell, R. B., & Horn, J. L. (1978). A check on the theory of fluid and crystallized intelligence with description of new subtest designs. Journal of Educational Measurement, 15, 139-164. http://dx.doi.org/10.1111/j.1745-3984.1978.tb00065.x
12. Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674. http://dx.doi.org/10.1126/science.2648573
13. Del Missier, F., Mäntylä, T., & Bruine de Bruin, W. (2012). Decision-making competence, executive functioning, and general cognitive abilities. Journal of Behavioral Decision Making, 25, 331-351. http://dx.doi.org/10.1002/bdm.731
14. Dweck, C. (2006). Mindset: The new psychology of success. New York, NY: Random House.
15. Einhorn, H. J. (1982). Learning from experience and suboptimal rules in decision making. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 268-286). Cambridge, UK: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511809477.020
16. Ericsson, K. A. (2006). The influence of experience and deliberative practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, R. Hoffman, & P. Feltovich (Eds.), Cambridge handbook of expertise and expert performance (pp. 683-704). New York, NY: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511816796.038
17. Ericsson, K. A. (2014). Why expert performance is special and cannot be extrapolated from studies of performance in the general population: A response to criticisms. Intelligence, 45, 81-103. http://dx.doi.org/10.1016/j.intell.2013.12.001
18. Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406. http://dx.doi.org/10.1037/0033-295X.100.3.363
19. Fischhoff, B., & Bruine De Bruin, W. (1999). Fifty-Fifty=50%? Journal of Behavioral Decision Making, 12, 149-163. http://dx.doi.org/10.1002/(SICI)1099-0771(199906)12:2<149::AID-BDM314>3.0.CO;2-J
20. Fischhoff, B., & Chauvin, C. (Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations. Washington, DC: National Academies Press.
21. Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3, 552-564. http://dx.doi.org/10.1037/0096-1523.3.4.552
22. Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19, 25-42. http://dx.doi.org/10.1257/089533005775196732
23. Funder, D., & Ozer, D. (1983). Behavior as a function of the situation. Journal of Personality and Social Psychology, 44, 107-112. http://dx.doi.org/10.1037/0022-3514.44.1.107
24. Furnham, A., & Monsen, J. (2009). Personality traits and intelligence predict academic school grades. Learning and Individual Differences, 19, 28-33. http://dx.doi.org/10.1016/j.lindif.2008.02.001
25. Green, K. C., & Armstrong, J. S. (2007). Global warming: Forecasts by scientists versus scientific forecasts. Energy & Environment, 18, 997-1021. http://dx.doi.org/10.1260/095830507782616887
26. Guilford, J. P., & Hoepfner, R. (1971). The analysis of intelligence. New York, NY: McGraw-Hill.
27. Haran, U., Ritov, I., & Mellers, B. A. (2013). The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Judgment and Decision Making, 8, 188-201.
28. Itoh, S., Ikeda, M., Mori, Y., Suzuki, K., Sawaki, A., Iwano, S., Ishigaki, T. (2002). Lung: Feasibility of a method for changing tube current during low-dose helical CT. Radiology, 224, 905-912. http://dx.doi.org/10.1148/radiol.2243010874
29. Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64, 515-526. http://dx.doi.org/10.1037/a0016755
30. Kahneman, D., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. New York, NY: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511809477
31. Kruglanski, A. W., & Webster, D. M. (1996). Motivated closing of the mind: "seizing" and "freezing." Psychological Review, 103, 263-283. http://dx.doi.org/10.1037/0033-295X.103.2.263
32. Laughlin, P. R. (2011). Group problem solving. Princeton, NJ: Princeton University Press.
33. Laughlin, P. R., Hatch, E. C., Silver, J. S., & Boh, L. (2006). Groups perform better than the best individuals on letters-to-numbers problems: Effects of group size. Journal of Personality and Social Psychology, 90, 644-651. http://dx.doi.org/10.1037/0022-3514.90.4.644
34. Lichtenstein, S., & Fischhoff, B. (1980). Training for calibration. Organizational Behavior & Human Performance, 26, 149-171. http://dx.doi.org/10.1016/0030-5073(80)90052-5
35. Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37-44. http://dx.doi.org/10.1177/0272989X0102100105
36. Loftus, E. (1996). Eyewitness testimony. Cambridge, MA: Harvard University Press.
37. MacCoun, R. J. (2012). The burden of social proof: Shared thresholds and social influence. Psychological Review, 119, 345-372. http://dx.doi.org/10.1037/a0027121
38. McGraw, P., Todorov, A., & Kunreuther, H. (2011). A policy maker's dilemma: Preventing terrorism or preventing blame? Organizational Behavior and Human Decision Processes, 115, 25-34. http://dx.doi.org/10.1016/j.obhdp.2011.01.004
39. Mellers, B. A., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Tetlock, P. (2014). Improving probabilistic predictions by identifying and cultivating "superforecasters". Manuscript submitted for publication.
40. Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Tetlock, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25, 1106-1115. http://dx.doi.org/10.1177/0956797614524255
41. Murphy, A. H., & Winkler, R. L. (1984). Probability forecasting in meteorology. Journal of the American Statistical Association, 79, 489-500.
42. Nickerson, R. S. (1987). Understanding understanding. New York, NY: Bolt Beranek and Newman.
43. Parker, A., & Fischhoff, B. (2005). Decision making competence: External validation through an individual-differences approach. Journal of Behavioral Decision Making, 18, 1-27. http://dx.doi.org/10.1002/bdm.481
44. Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006). Numeracy and decision making. Psychological Science, 17, 407-413. http://dx.doi.org/10.1111/j.1467-9280.2006.01720.x
45. Plomin, R., Shakeshaft, N. G., McMillan, A., & Trzaskowski, M. (2014). Nature, nurture, and expertise. Intelligence, 45, 46-59. http://dx.doi.org/10.1016/j.intell.2013.06.008
46. Ree, M. J., & Earles, J. A. (1992). Intelligence is the best predictor of job performance. Current Directions in Psychological Science, 1, 86-89. http://dx.doi.org/10.1111/1467-8721.ep10768746
47. Reyna, V. F., Chick, C. F., Corbin, J. C., & Hsia, A. N. (2014). Developmental reversals in risky decision making: Intelligence agents show larger decision biases than college students. Psychological Science, 25, 76-84. http://dx.doi.org/10.1177/0956797613497022
48. Riding, R., & Cheema, I. (1991). Cognitive styles-An overview and integration. Educational Psychology, 11, 193-215. http://dx.doi.org/10.1080/0144341910110301
49. Ruthsatz, J., Detterman, D. K., Griscom, W. S., & Cirullo, B. A. (2008). Becoming an expert in the musical domain: It takes more than just practice. Intelligence, 36, 330-338. http://dx.doi.org/10.1016/j.intell.2007.08.003
50. Schkade, D. A., & Kahneman, D. (1998). Does living in California make people happy? A focusing illusion in judgments of life satisfaction. Psychological Science, 9, 340-346. http://dx.doi.org/10.1111/1467-9280.00066
51. Schmidt, F. L., & Hunter, J. (2004). General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality and Social Psychology, 86, 162-173. http://dx.doi.org/10.1037/0022-3514.86.1.162
52. Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252-266. http://dx.doi.org/10.1016/0749-5978(92)90064-E
53. Soll, J. B., Milkman, K. L., & Payne, J. W. (in press). A user's guide to debiasing. In G. Wu & G. Keren (Eds.), Handbook of judgment and decision making. New York: Wiley.
54. Spearman, C. (1927). The abilities of man: Their nature and measurement. New York, NY: Macmillan.
55. Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89, 342-357. http://dx.doi.org/10.1037/0022-0663.89.2.342
56. Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought. Journal of Experimental Psychology: General, 127, 161-188. http://dx.doi.org/10.1037/0096-3445.127.2.161
57. Stanovich, K. E., & West, R. F. (2007). Natural myside bias is independent of cognitive ability. Thinking & Reasoning, 13, 225-247. http://dx.doi.org/10.1080/13546780600780796
58. Steiner, I. D. (1972). Group processes and group productivity. New York, NY: Academic Press.
59. Steyerberg, E. W. (2009). Clinical prediction models. New York, NY: Springer. http://dx.doi.org/10.1007/978-0-387-77244-8
60. Strenze, T. (2007). Intelligence and socioeconomic success: A metaanalytic review of longitudinal research. Intelligence, 35, 401-426. http://dx.doi.org/10.1016/j.intell.2006.09.004
61. Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1-26. http://dx.doi.org/10.1111/1529-1006.001
62. Taleb, N. N. (2007). Black swans and the domains of statistics. The American Statistician, 61, 198-200. http://dx.doi.org/10.1198/000313007X219996
63. Tetlock, P. E. (1998). Close-call counterfactuals and belief-system defenses: I was not almost wrong but I was almost right. Journal of Personality and Social Psychology, 75, 639-652. http://dx.doi.org/10.1037/0022-3514.75.3.639
64. Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know? Princeton, NJ: Princeton University Press.
65. Tetlock, P. E., & Mellers, B. A. (2011). Intelligent management of intelligence agencies: Beyond accountability ping-pong. American Psychologist, 66, 542-554. http://dx.doi.org/10.1037/a0024285
66. Thurstone, L. L., & Thurstone, T. G. (1941). Factorial studies of intelligence. Chicago, IL: Psychometric Monographs.
67. Vannoy, J. S. (1965). Generality of cognitive complexity-simplicity as a personality construct. Journal of Personality and Social Psychology, 2, 385-396. http://dx.doi.org/10.1037/h0022270
68. Webster, D. M., & Kruglanski, A. W. (1994). Individual differences in need for cognitive closure. Journal of Personality and Social Psychology, 67, 1049-1062. http://dx.doi.org/10.1037/0022-3514.67.6.1049
69. Wells, G. L. (2014). Eyewitness identification: Probative value, criterion shifts, and policy regarding the sequential lineup. Current Directions in Psychological Science, 23, 11-16. http://dx.doi.org/10.1177/0963721413504781
70. Wells, G. L., & Olson, E. A. (2003). Eyewitness testimony. Annual Review of Psychology, 54, 277-295. http://dx.doi.org/10.1146/annurev.psych.54.101601.145028
71. Wilson, T. D., & Brekke, N. (1994). Mental contamination and mental correction: Unwanted influences on judgments and evaluations. Psychological Bulletin, 116, 117-142. http://dx.doi.org/10.1037/0033-2909.116.1.117
72. Wilson, T. D., & Gilbert, D. (2005). Affective forecasting: Knowing what to want. Current Directions in Psychological Science, 14, 131-134. http://dx.doi.org/10.1111/j.0963-7214.2005.00355.x
73. Witkin, H. A., Oltman, P. K., Raskin, E., & Karp, S. A. (1971). A manual for the group embedded figures test. Palo Alto, CA: Consulting Psychologists Press.