How to make experimental economics research more reproducible: Lessons from other disciplines and a new proposal

Research in Experimental Economics

Volumn 18, Issue , 2015, Pages 215-230

  Maniadis, Zacharias ^a   Tufano, Fabio ^b   List, John A ^c  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^c UNIVERSITY OF CHICAGO   (United States)

Author keywords

False positives; Replication; Reproducibility

Indexed keywords

EID: 84945187445     PISSN: 01932306     EISSN: None     Source Type: Book Series    
DOI: 10.1108/S0193-230620150000018008     Document Type: Article

References (44)

1. Andreoli Versbach, P., & Mueller-Langer, F. (2013). Open access to data: An ideal professed but not practised. RatSWD Working Papers No. 215. Retrieved March 2015.
2. Bateman, I., Kahneman, D., Munro, A., Starmer, C., & Sugden, R. (2005). Testing competing models of loss aversion: An adversarial collaboration. Journal of Public Economics, 89(8), 1561-1580.
3. Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature 483(7391), 531-533.
4. Bettis, R. A. (2012). The search for asterisks: Compromised statistical tests and flawed theories. Strategic Management Journal, 33(1), 108-113.
5. Bicchieri, C. (2006). The grammar of society: The nature and dynamics of social norms. Cambridge: Cambridge University Press.
6. Camfield, L., & Palmer-Jones, R. (2013). Three 'Rs' of econometrics: Repetition, reproduction and replication. Journal of Development Studies, 49(12), 1607-1614.
7. Cavalli-Sforza, L. L., & Feldman, M. W. (1981). Cultural transmission and evolution: A quantitative approach. Princeton, NJ: Princeton University Press.
8. Coffman, L., & Niederle, M. (2014). Pre-analysis plans are not the solution, replications might be. Retrieved from http://web.stanford.edu/~niederle/Coffman.Niederle.PAP. JEP.October2014.pdf.
9. Dewald, W. G., Thursby, J. G., & Anderson, R. G. (1986). Replication in empirical economics: The journal of money, credit and banking project. American Economic Review, 76(4), 587-603.
10. Doshi, P., Goodman, S. N., & Ioannidis, J. P. A. (2013). Raw data from clinical trials: Within reach? Trends in Pharmacological Sciences, 34(12), 645-647.
11. Dufwenberg, M., & Martinsson, P. (2014). Keeping researchers honest: The case for sealedenvelope- submissions. Mimeo.
12. Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One, 4(5), e5738.
13. Fanelli, D. (2013). Redefine misconduct as distorted reporting. Nature, 494(7436), 149.
14. Gall, T., & Maniadis, Z. (2015). Evaluating solutions to the problem of false positives. University of Southampton Discussion Paper in Economics and Econometrics 1504.
15. Goldacre, B. (2011). Researchers don't mean to exaggerate, but lots of things can distort findings. The Guardian, August 2011.
16. Henry, E. (2009). Strategic disclosure of research results: The cost of proving your honesty. The Economic Journal, 119, 1036-1064.
17. Hertwig, R., & Ortmann, A. (2001). Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24(3), 383-403.
18. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
19. Ioannidis, J. P. A. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7(6), 645-654.
20. Ioannidis, J. P. A. (2014). How to make more published research true. PLoS Medicine, 11(10), e1001747.
21. Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., & Van Noort, V. (2009). Repeatability of published microarray gene expression analyses. Nature Genetics, 41, 149-155.
22. Jennions, M. D., & Møller, A. P. (2002). Relationships fade with time: A meta-analysis of temporal trends in publication in ecology and evolution. Proceedings of the Royal Society of London. Series B: Biological Sciences, 269(1486), 43-48.
23. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.
24. Kahneman, D. (2014). A new etiquette for replication. Social Psychology, 45(4), 299-311.
25. Kiri, B., Lacetera, N., & Zirulia, L. (2014). Above a swamp: A theory of high & quality scientific production. Mimeo.
26. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, ? S., Bernstein, M. J., & Woodzicka, J. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45(3), 142.
27. Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., & Silberberg, S. D. (2012). A call for transparent reporting to optimize the predictive value of preclinical research. Nature, 490(7419), 187-191.
28. Lehrer, J. (2010). The truth wears off. The New Yorker, 13, 52.
29. List, J. A., Bailey, C. D., Euzent, P. J., & Martin, T. L. (2001). Academic economists behaving badly? A survey on three areas of unethical behaviour. Economic Inquiry, 39(1), 162-170.
30. Maniadis, Z., Tufano, F., & List, J. A. (2014). One swallow doesn't make a summer: New evidence on anchoring effects. American Economic Review, 104(1), 277-290.
31. Maniadis, Z., Tufano, F., & List, J. A. (2015). One swallow doesn't make a summer: Reply to Simonsohn, Nelson and Simmons (2014). Mimeo.
32. Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., & Van der Laan, M. (2014). Promoting transparency in social science research. Science, 343(6166), 30.
33. Moonesinghe, R., Khoury, M. J., & Janssens, C. J. W. (2007). Most published research findings are false-but a little replication goes a long way. PLoS Medicine, 4(2), 0218-0221.
34. Nosek, B. A., & Lakens, D. (2014). Registered reports. Social Psychology, 45(3), 137-141.
35. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.
36. Poldrack, R. A., & Poline, J-B. (2014). The publication and reproducibility challenges of shared data. Trends in Cognitive Sciences, 19(2), 59-61.
37. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366.
38. Simonsohn, U., Simmons, J., & Nelson, L. (2014). Anchoring is not a false-positive: Maniadis, Tufano, and List's (2014) 'failure-to-replicate' is actually entirely consistent with the original. doi:10.2139/ssrn.2351926
39. The Economist. (2013). Trouble at the lab. 19 October 2013, pp. 23-27.
40. The Economist. (2014). Metaphysicians: Sloppy researchers beware. A new institute has you in its sights. March 2014.
41. Wacholder, S., Chanock, S., Garcia-Closas, M., El-ghormli, L., & Rothman, N. (2004). Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. Journal of the National Cancer Institute, 96(6), 434-442.
42. Wagenmakers, E.-J., & Forstmann, B. U. (2014). Rewarding high-power replication research. Cortex, 51, 10.
43. Young, E. (2012). Nobel laureate challenges psychologists to clean up their act. Nature News. doi:10.1038/nature.2012.11535
44. Zimmer, C. (2011). It's science, but not necessarily right. New York Times opinion. New York Times, New York, NY.