What does research reproducibility mean?

Science Translational Medicine

Volumn 8, Issue 341, 2016, Pages

  Goodman, Steven N ^a   Fanelli, Daniele ^a   Ioannidis, John P A ^a  

^a STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYTIC METHOD; ARTICLE; BAYES THEOREM; CONCEPTUAL FRAMEWORK; CONSENSUS DEVELOPMENT; DATA ANALYSIS; HUMAN; MEASUREMENT ERROR; MEASUREMENT REPEATABILITY; METHODOLOGY; NOMENCLATURE; NONHUMAN; PRIORITY JOURNAL; REPRODUCIBILITY; STATISTICAL ANALYSIS; MEDICAL RESEARCH; PROCEDURES; STANDARDS;

BIOMEDICAL RESEARCH; REPRODUCIBILITY OF RESULTS;

EID: 84973100833     PISSN: 19466234     EISSN: 19466242     Source Type: Journal    
DOI: 10.1126/scitranslmed.aaf5027     Document Type: Article

References (39)

1. C. G. Begley, An unappreciated challenge to oncology drug discovery: Pitfalls in preclinical research. Am. Soc. Clin. Oncol. Educ. Book 466-468 (2013).
2. Open Science Collaboration, Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
3. C. G. Begley, J. P. A. Ioannidis, Reproducibility in science: Improving the standard for basic and preclinical research. Circ. Res. 116, 116-126 (2015).
4. J. Claerbout, M. Karrenbach, Electronic documents give reproducible research a new meaning, in Proceedings of the 62nd Annual International Meeting of the Society of Exploration Geophysics, New Orleans, USA, 25 to 29 October 1992.
5. R. D. Peng, F. Dominici, S. L. Zeger, Reproducible epidemiologic research. Am. J. Epidemiol. 163, 783-789 (2006).
6. E. Afgan, D. Baker, M. van den Beek, D. Blankenberg, D. Bouvier, M. Čech, J. Chilton, D. Clements, N. Coraor, C. Eberhard, B. Grüning, A. Guerler, J. Hillman-Jackson, G. Von Kuster, E. Rasche, N. Soranzo, N. Turaga, J. Taylor, A. Nekrutenko, J. Goecks, The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update. Nucleic Acids Res. 10.1093/nar/gkw343 (2016).
7. J. Ioannidis, C. Doucouliagos, What's to know about the credibility of empirical economics? J. Econ. Surv. 27, 997-1004 (2013).
8. B. Lo, Sharing clinical trial data: Maximizing benefits, minimizing risk. JAMA 313, 793-794 (2015).
9. K. Bollen, J. T. Cacioppo, R. Kaplan, J. Krosnick, J. L. Olds, Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science (National Science Foundation, Arlington, VA, 2015).
10. F. S. Collins, L. A. Tabak, Policy: NIH plans to enhance reproducibility. Nature 505, 612-613 (2014).
11. D. Fanelli, W. Glänzel, Bibliometric evidence for a hierarchy of the sciences. PLOS One 8, e66938 (2013).
12. M. A. Clemens, The meaning of failed replications: A review and proposal. J. Econ. Surv. 10.1111/joes.12139 (2015).
13. B. V. Lewenstein, W. Baur, A cold fusion chronology. J. Radioanal. Nucl. Ch. 152, 273-297 (1991).
14. A. De Los Angeles, F. Ferrari, Y. Fujiwara, R. Mathieu, S. Lee, S. Lee, H.-C. Tu, S. Ross, S. Chou, M. Nguyen, Z. Wu, T. W. Theunissen, B. E. Powell, S. Imsoonthornruksa, J. Chen, M. Borkent, V. Krupalnik, E. Lujan, M. Wernig, J. H. Hanna, K. Hochedlinger, D. Pei, R. Jaenisch, H. Deng, S. H. Orkin, P. J. Park, G. Q. Daley, Failure to replicate the STAP cell phenomenon. Nature 525, E6-E9 (2015).
15. Early Breast Cancer Trialists' Collaborative Group, Effects of adjuvant tamoxifen and of cytotoxic therapy on mortality in early breast cancer. An overview of 61 randomized trials among 28, 896 women. N Engl. J. Med. 319, 1681-1692 (1988).
16. J. D. Schoenfeld, J. P. A. Ioannidis, Is everything we eat associated with cancer? A systematic cookbook review. Am. J. Clin. Nutr. 97, 127-134 (2013).
17. R. J. Hardy, S. G. Thompson, Detecting and describing heterogeneity in meta-analysis. Stat. Med. 17, 841-856 (1998).
18. E. E. Irons, J. W. Flatt, K. Doronin, T. L. Fox, M. Acchione, P. L. Stewart, D. M. Shayakhmetov, Coagulation factor binding orientation and dimerization may influence infectivity of adenovirus-coagulation factor complexes. J. Virol. 87, 9610-9619 (2013).
19. D. T. Gilbert, G. King, S. Pettigrew, T. D. Wilson, Comment on "Estimating the reproducibility of psychological science". Science 351, 1037 (2016).
20. S. N. Goodman, Toward evidence-based medical statistics. 2: The Bayes factor. Ann. Intern. Med. 130, 1005-1013 (1999).
21. S. N. Goodman, Toward evidence-based medical statistics. 1: The P value fallacy. Ann. Intern. Med. 130, 995-1004 (1999).
22. J. N. Rouder, R. D. Morey, A Bayes factor meta-analysis of Bem's ESP claim. Psychon. Bull. Rev. 18, 682-689 (2011).
23. S. N. Goodman, J. Gerson, Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (Agency for Healthcare Research and Quality, Rockville, MD, 2013); http://www.ncbi.nlm.nih.gov/books/NBK154584/.
24. P. Keating, A. Cambrosio, Cancer on Trial: Oncology as a New Style of Practice (University of Chicago Press, Chicago, 2011).
25. S. N. Goodman, A comment on replication, P-values and evidence. Stat. Med. 11, 875-879 (1992).
26. D. D. Boos, L. A. Stefanski, P-value precision and reproducibility. Am. Stat. 65, 213-221 (2011).
27. A. Gelman, H. Stern, The difference between "significant" and "not significant" is not itself statistically significant. Am. Stat. 60, 328-331 (2006).
28. L. C. Lazzeroni, Y. Lu, I. Belitskaya-Lévy, Solutions for quantifying P-value uncertainty and replication power. Nat. Methods 13, 107-108 (2016).
29. R. Rosenthal, The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638-641 (1979).
30. N. L. Kerr, HARKing: Hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196-217 (1998).
31. U. Simonsohn, L. D. Nelson, J. P. Simmons, P-curve: A key to the file-drawer. J. Exp. Psychol. Gen. 143, 534-547 (2014).
32. S. H. Hurlbert, Pseudoreplication and the design of ecological field experiments. Ecol. Monogr. 54, 187-211 (1984).
33. K. J. Rothman, Significance questing. Ann. Intern. Med. 105, 445-447 (1986).
34. J.L. Mills, Data torturing. N. Engl. J. Med. 329, 1196-1199 (1993).
35. H. White, A reality check for data snooping. Econometrica 68, 1097-1126 (2000).
36. A.-W. Chan, A. Hróbjartsson, M. T. Haahr, P. C. Gøtzsche, D. G. Altman, Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291, 2457-2465 (2004).
37. D. Berry, Multiplicities in cancer research: Ubiquitous and necessary evils. J. Natl. Cancer Inst. 104, 1125-1133 (2012).
38. E. E. Leamer, Specification Searches: Ad Hoc Inference with Nonexperimental Data (Wiley, Hoboken, NJ, 1978). Funding: This work was funded by a grant by from the John and Laura Arnold Foundation. Competing interests: The authors declare that they have no competing interests.
39. 1126/scitranslmed.aaf5027 Citation: S. N. Goodman, D. Fanelli, J. P. A. Ioannidis, What does research reproducibility mean? Sci. Transl. Med. 8, 341ps12 (2016).