Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate

Current Directions in Psychological Science

Volumn 23, Issue 4, 2014, Pages 290-295

  Tetlock, Philip E ^a   Mellers, Barbara A ^a   Rohrbaugh, Nick ^a   Chen, Eva ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

accuracy; forecasting; probability; tournaments

Indexed keywords

EID: 84905995856     PISSN: 09637214     EISSN: 14678721     Source Type: Journal    
DOI: 10.1177/0963721414534257     Document Type: Article

References (50)

1. Gardner D.Offers compelling contemporary and historical examples illustrating how it is possible for charlatans to flourish when we fail to systematically monitor judgmental accuracy. 2011;:
2. Kahneman D.Provides an elegant overview of the heuristics-and-biases research program, written by arguably the most influential psychologist of the last century. 2011;:
3. Tetlock P. E.Reports the first large-scale effort to use psychological theories and methods to study judgmental accuracy in world politics. 2005;:
4. Tetlock P. E.Reading tarot on K Street.The National Interest. 2009;:57-67
5. Tetlock P. E.,Mellers B. A.Explores how difficult it is for people and organizations to commit to improving judgmental accuracy when they are caught in the middle of accountability blame games. 2011;:
6. Principles of forecasting: A handbook for researchers and practitioners. Armstrong J. S., ed. New York, NY: Springer; 2001:
7. Atanasov P.,Rescober P.,Stone E.,Swift S.,Servan-Schreiber E.,Tetlock P.,Ungar L.,Mellers B.Crowd forecasting with prediction polls versus prediction markets. 2014;:
8. Axelrod R. M.The evolution of cooperation. New York, NY: Basic Books; 2006:
9. Baron J.Thinking and deciding. New York, NY: Cambridge University Press; 2000:
10. Baron J.,Ungar L. H.,Mellers B. A.,Tetlock P. E.Two reasons to make aggregated probability forecasts more extreme.Decision Analysis. 2014;:
11. Betts J. R.,Shkolnik J. L.Key difficulties in identifying the effects of ability grouping on student achievement.Economics of Education Review. 2000;19:21-26
12. Bielby W. T.Minimizing workplace gender and racial bias.Contemporary Sociology. 2000;29:120-129
13. Bielby W. T.Diversity at work. Brief A. P., ed. Cambridge, England: Cambridge University Press; 2008:53-86.
14. Brier G. W.Verification of forecasts expressed in terms of probability.Monthly Weather Review. 1950;78:1-3
15. BudescuD.ChenE. (in press). Identifying expertise to extract the wisdom of crowds. Management Science. Advanced online publication. doi: http://dx.doi.org/10.1287/mnsc.2014.1909
16. Cronbach L. J.Metatheory in social science: Pluralisms and subjectivities. Fiske D. W.Shweder R. A., ed. Chicago, IL: University of Chicago Press; 1986:83-107.
17. Dawes R. M.The handbook of social psychology. (1998). Behavioral decision making and judgment. In GilbertD. T.FiskeS. T.LindzeyG. (Eds), The handbook of social psychology (Vols. 1–2, 4th ed., pp. 497–548), New York, NY: McGraw-Hill.Gilbert D. T.Fiske S. T.Lindzey G., ed. New York, NY: McGraw-Hill; 1998:497-548.
18. Erev I.,Ert E.,Roth A. E.,Haruvy E.,Herzog S. M.,Hau R.,Lebiere C.A choice prediction competition: Choices from experience and from description.Journal of Behavioral Decision Making. 2010;23:15-47
19. Gardner D.Future babble: Why expert predictions are next to worthless, and you can do better. New York, NY: Dutton; 2011:
20. Greenwald A. G.,Poehlman T. A.,Uhlmann E. L.,Banaji M. R.Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.Journal of Personality and Social Psychology. 2009;97:17-41
21. Hanson R.Logarithmic market scoring rules for modular combinatorial information aggregation.Journal of Prediction Markets. 2007;1:3-15
22. Hartzmark M. L.Luck versus forecast ability: Determinants of trader performance in futures markets.The Journal of Business. 1991;64:49-74
23. Hunter J. E.,Hunter R. F.Validity and utility of alternative predictors of job performance.Psychological Bulletin. 1984;96:72-98
24. Jervis R.Perception and misperception in international politics. Princeton, NJ: Princeton University Press; 1976:
25. Jervis R.Why intelligence fails: Lessons from the Iranian Revolution and the Iraq War. Ithaca, NY: Cornell University Press; 2010:
26. Kahneman D.Thinking, fast and slow. London, England: Macmillan; 2011:
27. Kahneman D.,Klein G.Conditions for intuitive expertise: A failure to disagree.American Psychologist. 2009;64:515-526
28. Karvetski C. W.,Olson K. C.,Mandel D. R.,Twardy C. R.Probabilistic coherence weighting for optimizing expert forecasts.Decision Analysis. 2013;10:305-326
29. Lerner J. S.,Tetlock P. E.Accounting for the effects of accountability.Psychological Bulletin. 1999;125:255-275
30. Levitt B.,March J. G.Organizational learning.Annual Review of Sociology. 1988;14:319-340
31. MacCoun R. J.Biases in the interpretation and use of research results.Annual Review of Psychology. 1998;49:259-287
32. Makridakis S.,Andersen A.,Carbone R.,Fildes R.,Hibon M.,Lewandowski R.,Winkler R.The accuracy of extrapolation (time series) methods: Results of a forecasting competition.Journal of Forecasting. 1982;1:111-153
33. Mellers B.,Hertwig R.,Kahneman D.Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration.Psychological Science. 2001;12:269-275
34. MellersB.StoneE.AtanasovP.RohrbaughN.MetzS. E.UngarL.. TetlockP. (2014). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript submitted for publication.
35. Mellers B. A.,Ungar L.,Baron J.,Ramos J.,Gurcay B.,Fincher K.,.. Tetlock P.Psychological strategies for winning geopolitical forecasting tournaments.Psychological Science. 2014;:
36. MooreD. A.SwiftS. A.MinsterA.MellersB. A.UngarL.TetlockP. E.. TenneyE. R. (2014). Confidence and accuracy over time. Manuscript in preparation.
37. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science. 2012;7:657-660
38. Oswald F. L.,Mitchell G.,Blanton H.,Jaccard J.,Tetlock P. E.Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies.Journal of Personality and Social Psychology. 2013;105:171-192
39. Rosenthal R.Experimenter effects in behavioral research. East Norwalk, CT: Appleton-Century-Crofts; 1966:
40. Satopää V. A.,Baron J.,Foster D. P.,Mellers B. A.,Tetlock P. E.,Ungar L. H.Combining multiple probability predictions using a simple logit model.International Journal of Forecasting. 2014;30:344-356
41. Satopää V. A.,Jensen S. T.,Mellers B. A.,Tetlock P. E.,Ungar L. H.Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs.Annals of Applied Statistics. in press;:
42. Simmons J. P.,Nelson L. D.,Simonsohn U.False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.Psychological Science. 2011;22:1359-1366
43. Steyvers M.,Wallsten T. S.,Merkle E. C.,Turner B. M.Evaluating probabilistic forecasts with Bayesian signal detection models.Risk Analysis. 2014;34:435-452
44. Stone E.,Luu J..Poster session presented at the 33rd Annual Conference of the Society for Judgment and Decision Making; 2012Minneapolis, MN; 2012.
45. Tetlock P. E.Expert political judgment: How good is it? How can we know?. Princeton, NJ: Princeton University Press; 2005:
46. Tetlock P. E.,Manstead A. S.Impression management versus intrapsychic explanations in social psychology: A useful dichotomy?.Psychological Review. 1985;92:59-77
47. Tetlock P. E.,Mellers B. A.Intelligent management of intelligence agencies: Beyond accountability ping-pong.American Psychologist. 2011;66:542-554
48. Tetlock P. E.,Mitchell G.Implicit bias and accountability systems: What must organizations do to prevent discrimination?.Research in Organizational Behavior. 2009;29:3-38
49. Turner B. M.,Steyvers M.,Merkle E. C.,Budescu D. V.,Wallsten T. S.Forecast aggregation via recalibration.Machine Learning. 2013;:
50. Wallsten T. S.,Budescu D. V.,Zwick R.Comparing the calibration and coherence of numerical and verbal probability judgments.Management Science. 1993;39:176-190