Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions

Perspectives on Psychological Science

Volumn 10, Issue 3, 2015, Pages 267-281

  Mellers, Barbara ^a   Stone, Eric ^a   Murray, Terry ^b   Minster, Angela ^c   Rohrbaugh, Nick ^a   Bishop, Michael ^a   Chen, Eva ^a   Baker, Joshua ^a   Hou, Yuan ^a   Horowitz, Michael ^a   Ungar, Lyle ^a   Tetlock, Philip ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c TEMPLE UNIVERSITY   (United States)

Author keywords

expertise; forecasts; predictions; probability training; teams

Indexed keywords

AREA UNDER THE CURVE; COGNITION; DECISION MAKING; ENVIRONMENT; FORECASTING; HUMAN; LEARNING; MOTIVATION; PROBABILITY; PROCEDURES; PSYCHOLOGICAL MODEL; RECEIVER OPERATING CHARACTERISTIC; TIME;

AREA UNDER CURVE; COGNITION; ENVIRONMENT; FORECASTING; HUMANS; JUDGMENT; LEARNING; MODELS, PSYCHOLOGICAL; MOTIVATION; PROBABILITY; ROC CURVE; TIME;

EID: 84930337393     PISSN: 17456916     EISSN: 17456924     Source Type: Journal    
DOI: 10.1177/1745691615577794     Document Type: Article

References (55)

1. Arthur W.,Tubre T. C.,Paul D. S.,Sanchez-Ku M. L.College-sample psychometric and normative data on a short form of the Raven Advanced Progressive Matrices Test.Journal of Psychoeducational Assessment. 1999;17:354-361
2. AtanasovP.RescoberP.StoneE.Servan-SchreiberE.TetlockP.UngarL.MellersB. (2014). Distilling the wisdom of crowds: Prediction markets versus prediction polls. Manuscript under review.
3. Balboni G.,Naglieri J. A.,Cubelli R.Concurrent and predictive validity of the Raven Progressive Matrices and the Naglieri Nonverbal Ability Test.Journal of Psychoeducational Assessment. 2010;28:222-235
4. Baron J.Thinking and deciding. (2000). Thinking and deciding (3rd ed.). New York, NY: Cambridge University Press.New York, NY: Cambridge University Press; 2000:
5. Baron J.,Scott S. E.,Fincher K.,Metz S. E.Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)?.Journal of Applied Research in Memory and Cognition, Special Issue on Modeling and Aiding Intuitions in Organizational Decision Making. 2014;:
6. Baron J.,Spranca M.Protected values.Organizational Behavior and Human Decision Processes. 1997;70:1-16
7. Betts J. R.,Shkolnik J. L.The effects of ability grouping on student achievement and resource allocation in secondary schools.Economics of Education Review. 2000;19:1-15
8. Birnbaum M. H.,Mellers B. A.Bayesian inference: Combining base rates with opinions of sources who vary in credibility.Journal of Personality and Social Psychology. 1983;45:792-804
9. Brier G. W.Verification of forecasts expressed in terms of probability.Monthly Weather Review. 1950;78:1-3
10. Cacioppo J. T.,Petty R. E.The need for cognition.Journal of Personality and Social Psychology. 1982;42:116-131
11. Cacioppo J. T.,Petty R. E.,Kao C.-F.The efficient assessment of need for cognition.Journal of Personality Assessment. 1984;48:306-307
12. Cattell R. B.,Horn J. L.A check on the theory of fluid and crystallized intelligence with description of new subtest designs.Journal of Educational Measurement. 1978;15:139-164
13. Christensen-Szalanski J.,Wilhelm C.The hindsight bias: A meta-analysis.Organizational Behavior and Human Decision Processes. 1991;48:147-168
14. Cooper A.,Woo C.,Dunkelberg W.Entrepreneurs’ perceived chances for success.Journal of Business Venturing. 1988;3:97-108
15. Dawes R. M.,Faust D.,Meehl P. E.Clinical versus actuarial judgment.Science. 1989;243:1668-1673
16. Desvouges W. H.,Johnson F.,Dunford R.,Hudson S.,Wilson K.,Boyle K.Contingent valuation, A critical assessment. Hausman J. A., ed. Amsterdam, The Netherlands: North Holland; 1993:91-164.
17. Duckworth A. L.,Peterson C.,Matthew M. D.,Kelly D. R.Grit: Perseverance and passion for long-term goals.Journal of Personality and Social Psychology. 2007;92:1087-1101
18. Dweck C. S.Mindset: The new psychology of success. New York, NY: Ballantine Books; 2006:
19. Epple D.,Romano R.Peer effects in education: A survey of the theory and evidence.Handbook of Social Economics. 2011;1:1053-1163
20. Ericsson K. A.,Krampe R. T.,Romer C. T.The role of deliberate practice in the acquisition of expert performance.Psychological Review. 1993;100:363-406
21. Fiske A.,Tetlock P. E.Taboo tradeoffs: Reactions to transgressions that transgress spheres of influence.Political Psychology. 1997;18:255-297
22. Frederick S.Cognitive reflection and decision making.Journal of Economic Perspective. 2005;19 (4): 25-42
23. Gilovich T.,Griffin D.,Kahneman D.Heuristics and biases: The psychology of intuitive judgment. New York, NY: Cambridge University Press; 2002:
24. Gottfredson L.Why g matters: The complexity of everyday life.Intelligence. 1997;24:79-132
25. Haran U.,Ritov I.,Mellers B.The role of actively open-minded thinking in information acquisition, accuracy, and calibration.Journal of Decision Making. 2013;8:188-201
26. Hartzmark M. L.Luck versus forecast ability: Determinants of trader performance in futures markets.The Journal of Business. 1991;64:49-74
27. Heisenberg W.Physics and philosophy: The revolution in modern science. Amherst, NY: Prometheus Books; 1999:
28. Hunter J. E.,Schmidt F. L.Intelligence and job performance: Economic and social implications.Psychology, Public Policy, and the Law. 1996;2:447-472
29. Jagacinski R. J.,Isaac P. D.,Burke M. W.Application of signal detection theory to perceptual-motor skills: Decision processes in basketball shooting.Journal of Motor Behavior. 1977;9:225-234
30. Jervis R.Why intelligence fails. Ithaca, NY: Cornell University Press; 2010:
31. Kahneman D.Thinking, fast and slow. New York, NY: Cambridge University Press; 2011:
32. Kahneman D.,Knetsch J.Valuing public goods: The purchase of moral satisfaction.Journal of Environmental Economics and Management. 1992;22:57-70
33. Kahneman D.,Tversky A.Prospect theory: An analysis of decision under risk.Econometrica. 1979;47:263-292
34. Kahneman D.,Tversky A.Judgment under uncertainty: Heuristics and biases. Kahneman D.Slovic P.Tversky A., ed. New York, NY: Cambridge University Press; 1985:153-160.
35. Kray L. J.,George L. G.,Galinsky A. D.,Roese N. J.,Liljenquist K. A.,Tetlock P. E.From what might have been to what must have been: Counterfactual thinking creates meaning.Journal of Personality and Social Psychology. 2010;98:106-118
36. Levitt B.,March J. G.Organizational learning.Annual Review of Sociology. 1988;14:319-340
37. Lichtenstein S.,Fischhoff B.,Phillips L. D.Judgment under uncertainty: Heuristics and biases. Kahneman D.Slovic P.Tversky A., ed. Cambridge, England: Cambridge University Press; 1982:306-334.
38. Lipkus I. M.,Samsa G.,Rimer B. K.General performance on a numeracy scale among highly educated samples.Medical Decision Making. 2001;21:37-44
39. MellersB. A.StoneE.AtanasovP.RohrbaughN.MetzE. S.UngarL.. TetlockP. (2014). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript under review.
40. Mellers B. A.,Ungar L.,Baron J.,Ramos J.,Gurcay B.,Fincher K.,.. Tetlock P.Psychological strategies for winning a geopolitical tournament.Psychological Science. 2014;25:1106-1115
41. Milgram S.Behavioral study of obedience.Journal of Abnormal and Social Psychology. 1963;67:371-378
42. Murphy A. H.,Winkler R. L.A general framework for forecast verification.Monthly Weather Review. 1987;115:1330-1338
43. Ofir C.Pseudodiagnosticity in judgment under uncertainty.Organizational Behavior and Human Decision Making. 1988;42:343-363
44. Peters E.,Västfjäll D.,Slovic P.,Mertz C. K.,Mazzocco K.,Dickert S.Numeracy and decision making.Psychological Science. 2006;17:407-413
45. Rosenthal R.Experimenter effects in behavioral research. East Norwalk, CT: Appleton-Century-Crofts; 1966:
46. Satopää V. A.,Baron J.,Foster D.,Mellers B. A.,Tetlock P. E.,Ungar L. H.Combining multiple probability predictions using a simple logit model.International Journal of Forecasting. 2014;30:344-356
47. Satopää V. A.,Jensen S.,Mellers B. A.,Tetlock P. E.,Ungar L. H.Probability aggregation in time series: Dynamic hierarchical modeling of sparse expert beliefs.Annuals of Applied Statistics. 2014;8:1256-1280
48. Schkade D.,Payne J.How people respond to contingent valuation questions: A verbal protocol analysis of willingness to pay for an environmental regulation.Journal of Environmental Economics and Management. 1994;26:88-109
49. Shipley W. C.Shipley–2 manual. Torrance, CA: Western Psychological Services; 2009:
50. Svenson O.Are we all less risky and more skillful than our fellow drivers?.Acta Psychologica. 1981;47:143-148
51. Swets J. A.,Dawes R.,Monahan J.Psychological science can improve diagnostic decisions.Psychological Science in the Public Interest. 2000;1:1-26
52. Tetlock P. E.,Mellers B. A.The great rationality debate.Psychological Science. 2002;13:94-99
53. Wason P. C.Reasoning about a rule.Quarterly Journal of Experimental Psychology. 1968;20:273-281
54. Yaniv I.,Foster D.Graininess of judgment under uncertainty: An accuracy-informativeness tradeoff.Journal of Experimental Psychology: General. 1995;4:424-432
55. Zimbardo P.On the ethics of intervention in human psychological research: With special reference to the Stanford Prison Experiment.Cognition. 1973;2:243-256