To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study

Economic Journal

Volumn 127, Issue 605, 2017, Pages F209-F235

  Maniadis, Zacharias ^a   Tufano, Fabio ^b   List, John A ^c  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^c UNIVERSITY OF CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ECONOMICS; EMPIRICAL ANALYSIS; MODELING; REPRODUCTION; RESEARCH; THEORETICAL STUDY;

EID: 85032365799     PISSN: 00130133     EISSN: 14680297     Source Type: Journal    
DOI: 10.1111/ecoj.12527     Document Type: Article

References (59)

1. Alempaki, D., Canic, E., Matthews, W., Mullett, T., Stewart, N., Starmer, C. and Tufano, F. (2016). ‘Examining how utility and weighting functions get their shapes: a multi-level, quasi-adversarial, replication’, Working Paper, University of Nottingham.
2. Andreoni, J. and Harbaugh, W. (2005). ‘Power indices for revealed preference tests’, Working Paper, Social Systems Research Institute, University of Wisconsin.
3. Angrist, J. and Pischke, J.S. (2010). ‘The credibility revolution in empirical economics: How better research design is taking the con out of econometrics’, Journal of Economic Perspectives, vol. 24(2), pp. 3–30.
4. Babcock, L. and Loewenstein, G. (1997). ‘Explaining bargaining impasse: the role of self-serving biases’, Journal of Economic Perspectives, vol. 11(1), pp. 109–26.
5. Bakker, M., van Dijk, A. and Wicherts, J.M. (2012). ‘The rules of the game called psychological science’, Perspectives on Psychological Science, vol. 7(6), pp. 543–54.
6. Bettis, R.A. (2012). ‘The search for asterisks: compromised statistical tests and flawed theories’, Strategic Management Journal, vol. 33(1), pp. 108–13.
7. Brodeur, A., Lé, M., Sangnier, M. and Zylberberg, Y. (2016). ‘Star wars: the empirics strike back’, American Economic Journal: Applied Economics, vol. 8(1), pp. 1–32.
8. Camerer, C.F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M. and Wu, H. (2016). ‘Evaluating replicability of laboratory experiments in economics’, Science, vol. 351(6280), pp. 1433–6.
9. Card, D., Della Vigna, S. and Malmendier, U. (2011). ‘The role of theory in field experiments’, Journal of Economic Perspectives, vol. 25(3), pp. 39–62.
10. Cohen, J. (1962). ‘The statistical power of abnormal-social psychological research: a review’, Journal of Abnormal and Social Psychology, vol. 65(3), pp. 145–53.
11. Dawson, E., Gilovich, T. and Regan, D.T. (2002). ‘Motivated reasoning and performance on the Wason selection task’, Personality and Social Psychology Bulletin, vol. 28(10), pp. 1379–87.
12. Della Vigna, S. and Pope, D. (2016). ‘Predicting experimental results: who knows what?’ Working Paper, National Bureau of Economic Research No. w22566.
13. Della Vigna, S., List, J.A. and Malmendier, U. (2012). ‘Testing for altruism and social pressure in charitable giving’, Quarterly Journal of Economics, vol. 127(1), pp. 1–56.
14. DeLong, J.B. and Lang, K. (1992). ‘Are all economic hypotheses false?’, Journal of Political Economy, vol. 100(6), pp. 1257–72.
15. Di Tillio, A., Ottaviani, M. and Sorensen, P.N. (2017). ‘Persuasion bias in science: can economics help?’ Economic Journal, vol. 127(605), pp. F266–F304.
16. Doucouliagos, C. and Stanley, T.D. (2013). ‘Are all economic facts greatly exaggerated? Theory competition and selectivity’, Journal of Economic Surveys, vol. 27(2), pp. 316–39.
17. Duflo, E. (2006). ‘Field experiments in development economics’, in (R. Blundell, W.K. Newey and T. Persson, eds.), Advances in Economics and Econometrics Theory and Applications, Ninth World Congress, vol. 2, pp. 322–48, Cambridge: Cambridge University Press.
18. Duvendack, M., Palmer-Jones, R.W. and Reed, W.R. (2015). ‘Replications in economics: a progress report’, Econ Journal Watch, vol. 12(2), pp. 164–91.
19. Engel, C. (2011). ‘Dictator games: a meta study’, Experimental Economics, vol. 14(4), pp. 583–610.
20. Evanschitzky, H., Baumgarth, C., Hubbard, R. and Armstrong, J.S. (2007). ‘Replication research's disturbing trend’, Journal of Business Research, vol. 60(4), pp. 411–15.
21. Fanelli, D. (2010a). ‘Do pressures to publish increase scientists’ bias? An empirical support from US states data’, PLoS ONE, vol. 5(4), pp. 1–7.
22. Fanelli, D. (2010b). ‘‘Positive’ results increase down the hierarchy of the sciences’, PLoS ONE, vol. 5(3), p. e10068.
23. Fanelli, D. (2013). ‘Redefine misconduct as distorted reporting’, Nature, vol. 494(7436), p. 149.
24. Feld, L.P., Necker, S. and Frey, B.S. (2012). ‘Scientific misbehaviour in economics-evidence from Europe’, Working Paper, Walter-Euken Institute and University of Freiburg.
25. Franco, A., Malhotra, N. and Simonovits, G. (2014). ‘Publication bias in the social sciences: unlocking the file drawer’, Science, vol. 345(6203), pp. 1502–5.
26. Gilbert, D.T., King, G., Pettigrew, S. and Wilson, T.D. (2016). ‘Comment on ‘Estimating the reproducibility of psychological science’’, Science, vol. 351(6277), pp. 1037a–b.
27. Goodman, S.N., Fanelli, D. and Ioannidis, J.P.A. (2016). ‘What does research reproducibility mean?’ Science Translational Medicine, vol. 8(341), 341ps12, pp. 1–6.
28. Harrison, G.W. and List, J.A. (2004). ‘Field experiments’, Journal of Economic Literature, vol. 42(4), pp. 1009–55.
29. Hubbard, R. and Armstrong, S.J. (1994). ‘Replications and extensions in marketing: rarely published but quite contrary’, International Journal of Research in Marketing, vol. 11(3), pp. 233–48.
30. Ioannidis, J.P.A. (2005). ‘Why most published research findings are false’, PLoS Medicine, vol. 2(8), pp. 1418–22.
31. Ioannidis, J.P.A. (2008). ‘Why most discovered true associations are inflated’, Epidemiology, vol. 19(5), pp. 640–8.
32. Ioannidis, J.P.A. (2012). ‘Why science is not necessarily self-correcting’, Perspectives on Psychological Science, vol. 7(6), pp. 645–54.
33. Ioannidis, J. and Doucouliagos, C. (2013). ‘What's to know about the credibility of empirical economics?’, Journal of Economic Surveys, vol. 27(5), pp. 997–1004.
34. Ioannidis, J.P.A., Fanelli, D., Dunne, D. and Goodman, S.N. (2015). ‘Meta-research: evaluation and improvement of research methods and practices’, PLoS Biology, vol. 13(10), p. e1002264.
35. Ioannidis, J., Doucoulagos, C. and Stanley, T.D. (2017). ‘The power of bias in economics research’, Economic Journal, vol. 127(605), pp. F236–65.
36. Jennions, M.D. and Moller, A.P. (2002). ‘Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution’, Proceedings of the Royal Society of London, vol. 269(1486), pp. 43–8.
37. Landis, S.C., Amara, S.G., Asadulah, K., Austin, C.P., Blumenstein, R., Bradley, E.W., Crystal, R.G., Darnell, R.B., Ferrante, R.J., Fillit, H., Finkelstein, R., Fisher, M., Gendelman, H.E., Golub, R.M., Goudreau, J.L., Gross, R.A., Gubitz, A.K., Hesterlee, D.W., Howells, D.W., Huguenard, J., Kelner, K., Koroshetz, W., Krainc, D., Lazic, S.E. and Levine, M.S. (2012). ‘A call for transparent reporting to optimise the predictive value of preclinical research’, Nature, vol. 490(7419), pp. 187–91.
38. Levitt, S.D. and List, J.A. (2009). ‘Field experiments in economics: the past, the present, and the future’, European Economic Review, vol. 53(1), pp. 1–18.
39. List, J.A., Bailey, C., Euzent, P. and Martin, T. (2001). ‘Academic economists behaving badly? A survey on three areas of unethical behaviour’, Economic Inquiry, vol. 39(1), pp. 162–70.
40. List, J.A., Sadoff, S. and Wagner, M. (2011). ‘So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design’, Experimental Economics, vol. 14(4), pp. 439–57.
41. Makel, M.C., Plucker, J.A. and Hegarty, B. (2012). ‘Replications in psychology research: how often do they really occur?’, Perspectives on Psychological Science, vol. 7(6), pp. 537–42.
42. Maniadis, Z., Tufano, F. and List, J.A. (2014). ‘One swallow doesn't make a summer: new evidence on anchoring effects’, American Economic Review, vol. 104(1), pp. 277–90.
43. Maniadis, Z., Tufano, F. and List, J.A. (2015). ‘How to make experimental economics research more reproducible: lessons from other disciplines and a new proposal’, in (C. Deck, E. Fatas and T. Rosenblat, eds.), Research in Experimental Economics, Volume 18: Replication in Economic Experiments, pp. 215–30, Bingley: Emerald Group Publishing.
44. Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K.M., Gerber, A., Glennerster, R., Green, D.P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B.A., Petersen, M., Sedlmayr, R., Simmons, J.P., Simonsohn, U. and Van der Laan, M. (2014). ‘Promoting transparency in social science research’, Science, vol. 343(6166), pp. 30–1.
45. Moonesinghe, R., Khoury, M.J. and Janssens, C.J.W. (2007). ‘Most published research findings are false-but a little replication goes a long way’, PLoS Medicine, vol. 4(2), pp. 0218–21.
46. Nikiforakis, N. and Slonim, R. (2015). ‘Editors’ preface: introducing JESA’, Journal of Economic Science Association, vol. 1(1), pp. 1–7.
47. Nosek, B.A., Spies, J.R. and Motyl, M. (2012). ‘Scientific utopia II. Restructuring incentives and practices to promote truth over publishability’, Perspectives on Psychological Science, vol. 7(6), pp. 615–31.
48. Open Science Collaboration. (2015). ‘Estimating the reproducibility of psychological science’, Science, 349(6251), p. aac4716.
49. Oswald, A. (2007). ‘An examination of the reliability of prestigious scholarly journals: evidence and implications for decision-makers’, Economica, vol. 74(293), pp. 21–31.
50. Pashler, H. and Wagenmakers, E. (2012). ‘Editors’ introduction to the special section on replicability in psychological science a crisis of confidence?’, Perspectives on Psychological Science, vol. 7(6), pp. 528–30.
51. Pfeiffer, T., Bertram, L. and Ioannidis, J.P.A. (2011). ‘Quantifying selective reporting and the proteus phenomenon for multiple datasets with similar bias’, PLoS ONE, vol. 6(3), p. e18362.
52. Schmidt, S. (2009). ‘Shall we really do it again? The powerful concept of replication is neglected in the social sciences’, Review of General Psychology, vol. 13(2), pp. 90–100.
53. Sedlmeier, P. and Gigerenzer, G. (1989). ‘Do studies of statistical power have an effect on the power of studies?’, Psychological Bulletin, vol. 105(2), pp. 309–16.
54. Simmons, J.P., Nelson, L.D. and Simonsohn, U. (2011). ‘False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant’, Psychological Science, vol. 22(11), pp. 1359–66.
55. Simonsohn, U. (2015). ‘Small telescopes detectability and the evaluation of replication results’, Psychological Science, vol. 26(5), pp. 559–69.
56. Wacholder, S., Chanock, S., Garcia-Closas, M., El-ghormli, L. and Rothman, N. (2004). ‘Assessing the probability that a positive report is false: an approach for molecular epidemiology studies’, Journal of the National Cancer Institute, vol. 96(6), pp. 434–42.
57. Young, N., Ioannidis, J. and Al-Ubaydli, O. (2008). ‘Why current publication practices may distort science’, PLoS Medicine, vol. 5(10), p. e201.
58. Zhang, L. and Ortmann, A. (2013). ‘Exploring the meaning of significance in experimental economics’, Australian School of Business Research Paper No. 2013 ECON 32.
59. Zingales, L. (2013). ‘Preventing cconomists’ capture’, Chicago Booth Research Paper 13–81.