Using prediction markets to estimate the reproducibility of scientific research

Proceedings of the National Academy of Sciences of the United States of America

Volumn 112, Issue 50, 2015, Pages 15343-15347

  Dreber, Anna ^a   Pfeiffer, Thomas ^b,c   Almenberg, Johan ^d   Isaksson, Siri ^a   Wilson, Brad ^e   Chen, Yiling ^f   Nosek, Brian A ^g,h   Johannesson, Magnus ^a   Wachter, Kenneth W ⁱ  

^a STOCKHOLM SCHOOL OF ECONOMICS   (Sweden)

^b MASSEY UNIVERSITY   (New Zealand)

^c Wissenschaftskolleg zu Berlin Institute for Advanced Study   (Germany)

^d SVERIGES RIKSBANK   (Sweden)

^e Consensus Point ^*  (United States)

^f HARVARD UNIVERSITY   (United States)

^g UNIVERSITY OF VIRGINIA   (United States)

^h CENTER FOR OPEN SCIENCE   (United States)

ⁱ UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Prediction markets; Replications; Reproducibility

Indexed keywords

ARTICLE; CLINICAL RESEARCH; COST; HUMAN; MARKET; PREDICTION; PRIORITY JOURNAL; PSYCHOLOGY; PUBLICATION; PUBLISHING; REPLICATION STUDY; REPRODUCIBILITY; SCIENTIFIC LITERATURE; COMMERCIAL PHENOMENA; FORECASTING; PROBABILITY; QUESTIONNAIRE; RESEARCH; SCIENCE;

COMMERCE; FORECASTING; PROBABILITY; REPRODUCIBILITY OF RESULTS; RESEARCH; SCIENCE; SURVEYS AND QUESTIONNAIRES;

EID: 84950139698     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1516179112     Document Type: Article

References (76)

1. Prinz F, Schlange T, Asadullah K. (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10(9):712.
2. Begley CG, Ellis L.M. (2012) Drug development: Raise standards for preclinical cancer research. Nature 483 (7391):531-533.
3. Freedman LP, Cockburn IM, Simcoe T.S. (2015) The economics of reproducibility in preclinical research. PLoS Biol 13(6):e1002165.
4. Button KS, et al (2013) Power failure: Why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14(5):365-376.
5. Hewitt JK (2012) Editorial policy on candidate gene association and candidate geneby-environment interaction studies of complex traits. Behav Genet 42(1):1-2.
6. Lawrence MS, et al (2013) Mutational heterogeneity in cancer and the search for new cancer-associated genes. Nature 499 (7457):214-218.
7. Simmons JP, Nelson LD, Simonsohn U. (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22(11):1359-1366.
8. Carpenter S (2012) Psychology research. Psychology's bold initiative. Science 335 (6076): 1558-1561.
9. Open Science Collaboration (2012) An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect Psychol Sci 7(6):657-660.
10. Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349 (6251):aac4716.
11. Bohannon J (2014) Psychology. Replication effort provokes praise-and "bullying" charges. Science 344 (6186):788-789.
12. Ioannidis J, Doucouliagos C.J. (2013) What's to know about the credibility of empirical economics? J Econ Surv 27(5):997-1004.
13. Maniadis Z, Tufano F, List J.A. (2014) One swallow doesn't make a summer: How economists (mis-) use experimental methods and their results. Am Econ Rev 104(1): 277-290.
14. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8): e124.
15. Nuzzo R (2014) Scientific method: Statistical errors. Nature 506 (7487):150-152.
16. Easterbrook PJ, Berlin JA, Gopalan R, Matthews D.R. (1991) Publication bias in clinical research. Lancet 337 (8746):867-872.
17. Stern JM, Simes R.J. (1997) Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. BMJ 315 (7109):640-645.
18. Ioannidis JPA, Munafò MR, Fusar-Poli P, Nosek BA, David SP (2014) Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends Cogn Sci 18(5):235-241.
19. Miguel E, et al (2014) Social science. Promoting transparency in social science research. Science 343 (6166):30-31.
20. Hanson RD (1995) Could gambling save science? Encouraging an honest consensus. Soc Epistemology 9(1):3-33.
21. Almenberg J, Kittlitz K, Pfeiffer T. (2009) An experiment on prediction markets in science. PLoS One 4(12):e8500.
22. Park I-U, Peacey M.W., Munafò MR (2014) Modelling the effects of subjective and objective decision making in scientific peer review. Nature 506 (7486):93-96.
23. Wolfers J, Zitzewitz E. (2004) Prediction markets. J Econ Perspect 18(2):107-126.
24. Tziralis G, Tatsiopoulos I. (2007) Prediction markets: An extended literature review. J Pred Markets 1(1):75-91.
25. Arrow KJ, et al (2008) Economics. The promise of prediction markets. Science 320 (5878):877-878.
26. Berg J, Forsythe R, Nelson F., Rietz T. (2008) Results from a dozen years of election futures markets research. Handbook of Experimental Economics Results, eds Plott CR, Smith VL (North-Holland, Amsterdam), pp 742-751.
27. Manski CF (2006) Interpreting the predictions of prediction markets. Econ Lett 91(3): 425-429.
28. Wolfers J, Zitzewitz E. (2006) Interpreting Prediction Market Prices as Probabilities (National Bureau of Economic Research, Cambridge, MA), NBER Working Paper No. 12200.
29. Manning WG, et al (1987) Health insurance and the demand for medical care: Evidence from a randomized experiment. Am Econ Rev 77(3):251-277.
30. Hanson R (2006) Decision markets for policy advice. Promoting the General Welfare: American Democracy and the Political Economy of Government Performance, eds Patashnik EM, Gerber AS (Brookings Institution Press, Washington, DC), pp 151-173.
31. Chen Y, Kash IA, Ruberry M, Shnayder V. (2014) Eliciting predictions and recommendations for decision making. ACM Trans Econ Comput 2(2):6:1-6:27.
32. Hanson R (2007) Logarithmic market scoring rules for modular combinatorial information aggregation. J Pred Markets 1(1):3-15.
33. Richeson JA, Trawalter S. (2008) The threat of appearing prejudiced and race-based attentional biases. Psychol Sci 19(2):98-102.
34. Reynolds M, Besner D. (2008) Contextual effects on reading aloud: Evidence for pathway control. J Exp Psychol Learn Mem Cogn 34(1):50-64.
35. Rule NO, Ambady N. (2008) The face of success: Inferences from chief executive officers' appearance predict company profits. Psychol Sci 19(2):109-111.
36. Morris AL, Still M.L. (2008) Now you see it, now you don't: Repetition blindness for nonwords. J Exp Psychol Learn Mem Cogn 34(1):146-166.
37. Shnabel N, Nadler A. (2008) A needs-based model of reconciliation: Satisfying the differential emotional needs of victim and perpetrator as a key to promoting reconciliation. J Pers Soc Psychol 94(1):116-132.
38. Correll J (2008) 1/f noise and effort on implicit measures of bias. J Pers Soc Psychol 94(1):48-59.
39. Fischer P, Greitemeyer T, Frey D. (2008) Self-regulation and selective exposure: The impact of depleted self-regulation resources on confirmatory information processing. J Pers Soc Psychol 94(3):382-395.
40. Alter AL, Oppenheimer D.M. (2008) Effects of fluency on psychological distance and mental construal (or why New York is a large city, but New York is a civilized jungle). Psychol Sci 19(2):161-167.
41. Mirman D, Magnuson J.S. (2008) Attractor dynamics and semantic neighborhood density: Processing is slowed by near neighbors and speeded by distant neighbors. J Exp Psychol Learn Mem Cogn 34(1):65-79.
42. Estes Z, Verges M, Barsalou L.W. (2008) Head up, foot down: Object words orient attention to the objects' typical location. Psychol Sci 19(2):93-97.
43. Nairne JS, Pandeirada JNS, Thompson S.R. (2008) Adaptive memory: The comparative value of survival processing. Psychol Sci 19(2):176-180.
44. White PA (2008) Accounting for occurrences: A new view of the use of contingency information in causal judgment. J Exp Psychol Learn Mem Cogn 34(1):204-218.
45. Pacton S, Perruchet P. (2008) An attention-based associative account of adjacent and nonadjacent dependency learning. J Exp Psychol Learn Mem Cogn 34(1):80-96.
46. Pleskac TJ (2008) Decision making and learning while taking sequential risks. J Exp Psychol Learn Mem Cogn 34(1):167-185.
47. Masicampo EJ, Baumeister R.F. (2008) Toward a physiology of dual-process reasoning and judgment: Lemonade, willpower, and expensive rule-based analysis. Psychol Sci 19(3):255-260.
48. Janssen N, Schirm W, Mahon B.Z., Caramazza A. (2008) Semantic interference in a delayed naming task: Evidence for the response exclusion hypothesis. J Exp Psychol Learn Mem Cogn 34(1):249-256.
49. Henderson MD, de Liver Y, Gollwitzer P.M. (2008) The effects of an implemental mindset on attitude strength. J Pers Soc Psychol 94(3):396-411.
50. Turk-Browne N.B., Isola PJ, Scholl B.J., Treat T.A. (2008) Multidimensional visual statistical learning. J Exp Psychol Learn Mem Cogn 34(2):399-407.
51. Vul E, Nieuwenstein M, Kanwisher N. (2008) Temporal selection is suppressed, delayed, and diffused during the attentional blink. Psychol Sci 19(1):55-61.
52. Vohs KD, Schooler J.W. (2008) The value of believing in free will: Encouraging a belief in determinism increases cheating. Psychol Sci 19(1):49-54.
53. Fischer P, Schulz-Hardt S, Frey D. (2008) Selective exposure and information quantity: How different information quantities moderate decision makers' preference for consistent and inconsistent information. J Pers Soc Psychol 94(2):231-244.
54. Bressan P, Stranieri D. (2008) The best men are (not always) already taken: Female preference for single versus attached males depends on conception risk. Psychol Sci 19(2):145-151.
55. Sahakyan L, Delaney PF, Waldum E.R. (2008) Intentional forgetting is easier after two "shots" than one. J Exp Psychol Learn Mem Cogn 34(2):408-414.
56. Lobue V, DeLoache J.S. (2008) Detecting the snake in the grass: Attention to fearrelevant stimuli by adults and young children. Psychol Sci 19(3):284-289.
57. Nurmsoo E, Bloom P. (2008) Preschoolers' perspective taking in word learning: Do they blindly follow eye gaze? Psychol Sci 19(3):211-215.
58. Wolf ST, Insko CA, Kirchner J.L., Wildschut T. (2008) Interindividual-intergroup discontinuity in the domain of correspondent outcomes: The roles of relativistic concern, perceived categorization, and the doctrine of mutual assured destruction. J Pers Soc Psychol 94(3):479-494.
59. Förster J., Liberman N, Kuschel S. (2008) The effect of global versus local processing styles on assimilation versus contrast in social judgment. J Pers Soc Psychol 94(4): 579-599.
60. Lau GP, Kay AC, Spencer S.J. (2008) Loving those who justify inequality: The effects of system threat on attraction to women who embody benevolent sexist ideals. Psychol Sci 19(1):20-21.
61. Stanovich KE, West R.F. (2008) On the relative independence of thinking biases and cognitive ability. J Pers Soc Psychol 94(4):672-695.
62. Lemay EP, Jr, Clark MS (2008) "Walking on eggshells": How expressing relationship insecurities perpetuates them. J Pers Soc Psychol 95(2):420-441.
63. Stinson DA, et al (2008) The cost of lower self-esteem: Testing a self- and social-bonds model of health. J Pers Soc Psychol 94(3):412-428.
64. McCrea SM (2008) Self-handicapping, excuse making, and counterfactual thinking: Consequences for self-esteem and future motivation. J Pers Soc Psychol 95(2):274-292.
65. Halevy N, Bornstein G, Sagiv L. (2008) "In-group love" and "out-group hate" as motives for individual participation in intergroup conflict: A new game paradigm. Psychol Sci 19(4):405-411.
66. Yap MJ, Balota DA, Tse C.S., Besner D. (2008) On the additive effects of stimulus quality and word frequency in lexical decision: Evidence for opposing interactive influences revealed by RT distributional analyses. J Exp Psychol Learn Mem Cogn 34(3):495-513.
67. Koo M, Fishbach A. (2008) Dynamics of self-regulation: How (un)accomplished goal actions affect motivation. J Pers Soc Psychol 94(2):183-195.
68. Schmidt JR, Besner D. (2008) The Stroop effect: Why proportion congruent has nothing to do with congruency and everything to do with contingency. J Exp Psychol Learn Mem Cogn 34(3):514-523.
69. Dodson CS, Darragh J, Williams A. (2008) Stereotypes and retrieval-provoked illusory source recollections. J Exp Psychol Learn Mem Cogn 34(3):460-477.
70. Fiedler K (2008) The ultimate sampling dilemma in experience-based decision making. J Exp Psychol Learn Mem Cogn 34(1):186-203.
71. Berry CJ, Shanks DR, Henson RNA (2008) A single-system account of the relationship between priming, recognition, and fluency. J Exp Psychol Learn Mem Cogn 34(1): 97-111.
72. Tamir M, Mitchell C, Gross J.J. (2008) Hedonic and instrumental motives in anger regulation. Psychol Sci 19(4):324-328.
73. Liefooghe B, Barrouillet P, Vandierendonck A., Camos V (2008) Working memory costs of task switching. J Exp Psychol Learn Mem Cogn 34(3):478-494.
74. Farris C, Treat TA, Viken R.J., McFall R.M. (2008) Perceptual mechanisms that characterize gender differences in decoding women's sexual intent. Psychol Sci 19(4): 348-354.
75. Purdie-Vaughns V., Steele CM, Davies P.G., Ditlmann R., Crosby J.R. (2008) Social identity contingencies: How diversity cues signal threat or safety for African Americans in mainstream institutions. J Pers Soc Psychol 94(4):615-630.
76. Bassok M, Pedigo SF, Oskarsson A.T. (2008) Priming addition facts with semantic relations. J Exp Psychol Learn Mem Cogn 34(2):343-352.