Distilling the wisdom of crowds: Prediction markets vs. prediction polls

Management Science

Volumn 63, Issue 3, 2017, Pages 691-706

  Atanasov, Pavel ^a,b   Rescober, Phillip ^c   Stone, Eric ^d   Swift, Samuel A ^e   Servan Schreiber, Emile ^f   Tetlock, Philip ^a   Ungar, Lyle ^g   Mellers, Barbara ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b Polly Portfolio   (United States)

^c Good Judgment Inc   (United States)

^d Pluralsight Inc   (United States)

^e Bettermentcom   (United States)

^f Lumenogic Inc   (France)

^g UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Belief elicitation; Crowdsourcing; Forecasting; Prediction markets

Indexed keywords

COMMERCE; CROWDSOURCING; SURVEYS;

BELIEF ELICITATION; CONTINUOUS DOUBLE AUCTION; EXPERIMENTAL TEST; LONG DURATION; PREDICTION MARKETS; RECALIBRATIONS; TEMPORAL DECAY; WISDOM OF CROWDS;

FORECASTING;

EID: 85015207276     PISSN: 00251909     EISSN: 15265501     Source Type: Journal    
DOI: 10.1287/mnsc.2015.2374     Document Type: Article

References (52)

1. Armstrong S (2001) Combining forecasts. Armstrong JS, ed. Principles of Forecasting (Springer, New York), 417-439.
2. Atanasov P, Rescober P, Servan-Schreiber E, Mellers B, Tetlock P, Ungar L (2013) The marketcast method for aggregating prediction market forecasts. Proc. 2013 Internat. Conf. Social Comput., Behavioral-Cultural Modeling, and Prediction (Springer, New York).
3. Baron J, Mellers BA, Tetlock PE, Stone E, Ungar LH (2014) Two reasons to make aggregated probability forecasts more extreme. Decision Anal. 11(2):133-145.
4. Berg J, Forsythe R, Nelson F, Rietz T (2001) Results from a dozen years of election futures markets research. Plott CR, Smith VL, eds. Handbook of Experimental Economics Results, Vol. 1(North-Holland, Amsterdam), 742-751.
5. Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1-3.
6. Camerer CF, Hogarth RM (1999) The effects of financial incentives in experiments: A review and capital-labor-production framework. J. Risk Uncertainty 19(1-3):7-42.
7. Chen Y, Chu C, Mullen T, Pennock D (2005) Information markets vs. opinion pools: An empirical comparison. Proc. 6th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 58-67.
8. Cowgill B, Zitzewitz E (2015) Corporate prediction markets: Evidence from Google, Ford, and Firm X. Rev. Econom. Stud. 82(4):1309-1341.
9. Fama EF (1970) Efficient capital markets: A review of theory and empirical work. J. Finance 25(2):383-417.
10. Forsythe R, Nelson F, Neumann GR, Wright J (1992) Anatomy of an experimental political stock market. Amer. Econom. Rev. 82(5):1142-1161.
11. Goel S, Reeves D, Watts D, Pennock D (2010) Prediction without markets. Proc. 11th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 357-366.
12. Graefe A, Armstrong J (2011) Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. Internat. J. Forecasting 27(1):183-195.
13. Gürkaynak RS, Wolfers J (2006) Macroeconomic derivatives: An initial analysis of market-based macro forecasts, uncertainty, and risk. NBER Working Paper 11929, National Bureau of Economic Research, Cambridge, MA.
14. Hanson R (2003) Combinatorial information market design. Inform. Systems Frontiers 5(1):107-119.
15. Hayek F (1945) The use of knowledge in society. Amer. Econom. Rev. 35(4):519-530.
16. Hertwig R, Ortmann A (2001) Experimental practices in economics: A methodological challenge for psychologists? Behav. Brain Sci. 24(3):383-403.
17. Lichtendahl KC Jr, Winkler RL (2007) Probability elicitation, scoring rules, and competition among forecasters. Management Sci. 53(11):1745-1755.
18. Laster D, Bennett P, Geoum IS (1999) Rational bias in macroeconomic forecasts. Quart. J. Econom. 114(1):293-318.
19. Maciejovsky B, Sutter M, Budescu D, Bernau P (2013) Teams make you smarter: How exposure to teams improves individual decisions in probability and reasoning tasks. Management Sci. 59(6):1255-1270.
20. Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J. Personality Soc. Psych. 107(2):276-299.
21. Manski C (2006) Interpreting the predictions of prediction markets. Econom. Lett. 91(3):425-429.
22. Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Tetlock P (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci. 25(5):1106-1115.
23. Mellers B, Stone E, Atanasov P, Rohrbaugh N, Metz SE, Ungar L, Bishop MM, Horowitz M, Merkle E, Tetlock P (2015a) The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. J. Experiment. Psychol.: Appl. 21(1):1-14.
24. Mellers B, Stone E, Murray T, Minster A, Rohrbaugh N, Bishop M, Chen E, et al. (2015b) Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psychol. Sci. 10(3):267-281.
25. Moore DA, Swift SA, Minster A, Mellers B, Ungar L, Tetlock P, Yang H, Tenney R (2015) Confidence calibration in a multiyear geopolitical forecasting competition. Working paper, University of California, Berkeley, Berkeley.
26. Murphy AH (1973) A new vector partition of the probability score. J. Appl. Meteorology 12(4):595-600.
27. Ostrovsky M (2012) Information aggregation in dynamic markets with strategic traders. Econometrica 80(6):2595-2647.
28. Page L, Clemen RT (2012) Do prediction markets produce wellcalibrated probability forecasts? Econom. J. 123(568):491-513.
29. Pennock DM, Lawrence S, Nielsen FA, Giles CL (2001) Extracting collective probabilistic forecasts from Web games. Proc. 7th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 174-183.
30. Plott C, Chen K (2002) Information aggregation mechanisms: Concept, design and implementation for a sales forecasting problem. Working paper, California Institute of Technology, Pasadena.
31. Ranjan R, Gneiting T (2008) Combining probability forecasts. Technical Report 543, University of Washington, Seattle.
32. Rieg R, Schoder R (2010) Forecasting accuracy: Comparing prediction markets and surveys-An experimental study. J. Prediction Markets 4(3):1-19.
33. Rosenbloom ES, Notz W (2006) Statistical tests of real-money versus play-money prediction markets. Electronic Markets 16(1):63-69.
34. Rothschild D (2009) Forecasting elections: Comparing prediction markets, polls, and their biases. Public Opinion Quart. 73(5):895-916.
35. Rothschild D, Wolfers J (2010) Forecasting elections: Voter intentions versus expectations. Working paper, National Institute of Economic Research, Cambridge, MA.
36. Satopää VA, Jensen ST, Mellers BA, Tetlock PE, Ungar LH (2014a) Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs. Ann. Appl. Statist. 8(2):1256-1280.
37. Satopää VA, Baron J, Foster DP, Mellers BA, Tetlock PE, Ungar LH (2014b) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344-356.
38. Servan-Schreiber E (2007) About competitive forecasting. White paper, NewsFutures, Paris.
39. Servan-Schreiber E (2012) Prediction markets: Trading uncertainty for collective wisdom. Landemore H, Elster J, eds. Collective Wisdom: Principles and Mechanisms (Cambridge University Press, New York).
40. Servan-Schreiber E, Wolfers J, Pennock DM, Galebach B (2004) Prediction markets: Does money matter? Electronic Markets 14(3):243-251.
41. Sherden WA (1998) The Fortune Sellers: The Big Business of Buying and Selling Predictions (John Wiley & Sons, Hoboken, NJ).
42. Silver N (2012) The Signal and the Noise: Why So Many Predictions Fail-But Some Don't (Penguin, New York).
43. Snowberg E, Wolfers J (2010) Explaining the favorite-long shot bias: Is it risk-love or misperceptions? J. Political Econom. 118(4):723-746.
44. Sonnemann U, Camerer C, Fox C, Langer T (2013) How psychological framing affects economic market prices in the lab and field. Proc. Natl. Acad. Sci. USA 110(29):11779-11784.
45. Spann M, Skiera B (2003) Internet-based virtual stock markets for business forecasting. Management Sci. 49(10):1310-1326.
46. Surowiecki J (2005) The Wisdom of Crowds (Anchor Books, New York).
47. Swets J (1996) Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers (Lawrence Erlbaum Associates, Mahwah, NJ).
48. Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).
49. Tetlock PE, Mellers BA, Rohrbaugh N, Chen E (2014) Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions Psych. Sci. 23(4):290-295.
50. Wolfers J (2009) Prediction markets: The collective knowledge of market participants. CFA Institute Conf. Proc. Quart. 26(2):37-44.
51. Wolfers J, Zitzewitz E (2006) Interpreting prediction market prices as probabilities. NBER Working Paper 12200, National Bureau of Economic Research, Cambridge, MA.
52. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J. Royal Statist. Soc. 67(2):301-320.