A manifesto for reproducible science

Nature Human Behaviour

Volumn 1, Issue 1, 2017, Pages

  Munafò, Marcus R ^a   Nosek, Brian A ^b,c   Bishop, Dorothy V M ^d   Button, Katherine S ^e   Chambers, Christopher D ^f   Percie Du Sert, Nathalie ^g   Simonsohn, Uri ^h   Wagenmakers, Eric Jan ⁱ   Ware, Jennifer J ^j   Ioannidis, John P A ^k,l  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b UNIVERSITY OF VIRGINIA   (United States)

^c CENTER FOR OPEN SCIENCE   (United States)

^d UNIVERSITY OF OXFORD   (United Kingdom)

^e University of Bath   (United Kingdom)

^f CARDIFF UNIVERSITY   (United Kingdom)

^g National Centre for the Replacement   (United Kingdom)

^h UNIVERSITY OF PENNSYLVANIA   (United States)

ⁱ UNIVERSITY OF AMSTERDAM   (Netherlands)

^j CHDI MANAGEMENT CHDI FOUNDATION   (United States)

^k STANFORD UNIVERSITY   (United States)

^l STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

more..

Author keywords

[No Author keywords available]

Indexed keywords

ADOPTION; ARTICLE; EMPIRICISM; INCENTIVE; REPRODUCIBILITY; SIMULATION;

EID: 85029593950     PISSN: None     EISSN: 23973374     Source Type: Journal    
DOI: 10.1038/s41562-016-0021     Document Type: Review

References (87)

1. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).
2. Button, K. S., et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365-376 (2013).
3. Fanelli, D. "Positive" results increase down the Hierarchy of the Sciences. PloS ONE 5, e10068 (2010).
4. John, L. K., Loewenstein, G., Prelec, D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524-532 (2012).
5. Makel, M. C., Plucker, J. A., Hegarty, B. Replications in psychology research: How often do they really occur Perspect. Psychol. Sci. 7, 537-542 (2012).
6. Wicherts, J. M., Borsboom, D., Kats, J., Molenaar, D. Tpoor availability of psychological research data for reanalysis. Am. Psychol. 61, 726-728 (2006).
7. Kerr, N. L. HARKing: Hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196-217 (1998).
8. Al-Shahi Salman, R., et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet 383, 176-185 (2014).
9. Begley, C. G., Ioannidis, J. P. Reproducibility in science: Improving the standard for basic and preclinical research. Circ. Res. 116, 116-126 (2015).
10. Chalmers, I., et al. How to increase value and reduce waste when research priorities are set. Lancet 383, 156-165 (2014).
11. Chan, A. W., et al. Increasing value and reducing waste: Addressing inaccessible research. Lancet 383, 257-266 (2014).
12. Glasziou, P., et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383, 267-276 (2014).
13. Ioannidis, J. P., et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383, 166-175 (2014).
14. Macleod, M. R., et al. Biomedical research: Increasing value, reducing waste. Lancet 383, 101-104 (2014).
15. Baker, M. 1, 500 scientists lift the lid on reproducibility. Nature 533, 452-454 (2016).
16. Ioannidis, J. P., Fanelli, D., Dunne, D. D., Goodman, S. N. Meta-research: Evaluation and improvement of research methods and practices. PLoS Biol. 13, e1002264 (2015).
17. Paneth, N. Assessing the contributions of John Snow to epidemiology: 150 years after removal of the broad street pump handle. Epidemiology 15, 514-516 (2004).
18. Berker, E. A., Berker, A. H., Smith, A. Translation of Broca's 1865 report. Localization of speech in the third left frontal convolution. Arch. Neurol. 43, 1065-1072 (1986).
19. Wade, N. Discovery of pulsars: A graduate student's story. Science 189, 358-364 (1975).
20. Nickerson, R. S. Confirmation bias: A ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175-220 (1998).
21. Levenson, T. The Hunt for Vulcan.and How Albert Einstein Destroyed a Planet, Discovered Relativity, and Deciphered the University (Random House, 2015).
22. Rosenthal, R. Experimenter Effects in Behavioral Research (Appleton-Century-Crofts, 1966).
23. de Groot, A. D. The meaning of "significance" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychol. 148, 188-194 (2014).
24. Heininga, V. E., Oldehinkel, A. J., Veenstra, R., Nederhof, E. I just ran a thousand analyses: Benefits of multiple testing in understanding equivocal evidence on gene-environment interactions. PloS ONE 10, e0125383 (2015).
25. Patel, C. J., Burford, B., Ioannidis, J. P. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J. Clin. Epidemiol. 68, 1046-1058 (2015).
26. Carp, J. The secret lives of experiments: Methods reporting in the fMRI literature. Neuroimage 63, 289-300 (2012).
27. Carp, J. On the plurality of (methodological) worlds: Estimating the analytic flexibility of FMRI experiments. Front. Neurosci. 6, 149 (2012).
28. Simonsohn, U., Nelson, L. D., Simmons, J. P. P-curve: A key to the file-drawer. J. Exp. Psychol. Gen. 143, 534-547 (2014).
29. Nuzzo, R. Fooling ourselves. Nature 526, 182-185 (2015).
30. MacCoun, R., Perlmutter, S. Blind analysis: Hide results to seek the truth. Nature 526, 187-189 (2015).
31. Greenland, S., et al. Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. Eur. J. Epidemiol. 31, 337-350 (2016).
32. Sterne, J. A., Davey Smith, G. Sifting the evidence-what's wrong with significance tests BMJ 322, 226-231 (2001).
33. Brand, A., Bradley, M. T., Best, L. A., Stoica, G. Accuracy of effect size estimates from published psychological research. Percept. Motor Skill. 106, 645-649 (2008).
34. Vankov, I., Bowers, J., Munafò, M. R. On the persistence of low power in psychological science. Q. J. Exp. Psychol. 67, 1037-1040 (2014).
35. Sedlmeier, P., Gigerenzer, G. Do studies of statistical power have an effect on the power of studies Psychol. Bull. 105, 309-316 (1989).
36. Cohen, J. The statistical power of abnormal-social psychological research: A review. J. Abnorm. Soc. Psychol. 65, 145-153 (1962).
37. Etter, J. F., Burri, M., Stapleton, J. The impact of pharmaceutical company funding on results of randomized trials of nicotine replacement therapy for smoking cessation: A meta-analysis. Addiction 102, 815-822 (2007).
38. Etter, J. F., Stapleton, J. Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals. J. Clin. Epidemiol. 62, 831-837 (2009).
39. Panagiotou, O. A., Ioannidis, J. P. Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J. Clin. Epidemiol. 65, 740-747 (2012).
40. Nosek, B. A., Spies, J. R., Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631 (2012).
41. Bath, P. M. W., Macleod, M. R., Green, A. R. Emulating multicentre clinical stroke trials: A new paradigm for studying novel interventions in experimental models of stroke. Int. J. Stroke 4, 471-479 (2009).
42. Dirnagl, U., et al. A concerted appeal for international cooperation in preclinical stroke research. Stroke 44, 1754-1760 (2013).
43. Milidonis, X., Marshall, I., Macleod, M. R., Sena, E. S. Magnetic resonance imaging in experimental stroke and comparison with histology systematic review and meta-analysis. Stroke 46, 843-851 (2015).
44. Klein, R. A., et al. Investigating variation in replicability: A "many labs" replication project. Soc. Psychol. 45, 142-152 (2014).
45. Ebersole, C. R., et al. Many Labs 3: Evaluating participant pool quality across the academic semester via replication. J. Exp. Soc. Psychol. 67, 68-82 (2016).
46. Lenzer, J., Hoffman, J. R., Furberg, C. D., Ioannidis, J. P. A. Ensuring the integrity of clinical practice guidelines: A tool for protecting patients. BMJ 347, f5535 (2013).
47. Sterling, T. D. Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. J. Am. Stat. Assoc. 54, 30-34 (1959).
48. Rosenthal, R. File drawer problem and tolerance for null results. Psychol. Bull. 86, 638-641 (1979).
49. Sterling, T. D. Consequence of prejudice against the null hypothesis. Psychol. Bull. 82, 1-20 (1975).
50. Franco, A., Malhotra, N., Simonovits, G. Publication bias in the social sciences: Unlocking the file drawer. Science 345, 1502-1505 (2014).
51. Simmons, J. P., Nelson, L. D., Simonsohn, U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366 (2011).
52. Chambers, C. D. Registered Reports: A new publishing initiative at Cortex. Cortex 49, 609-610 (2013).
53. Nosek, B. A., Lakens, D. Registered Reports: A method to increase the credibility of published results. Soc. Psychol. 45, 137-141 (2014).
54. Nosek, B. A., et al. Promoting an open research culture. Science 348, 1422-1425 (2015).
55. Begg, C., et al. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 276, 637-639 (1996).
56. Moher, D., Dulberg, C. S., Wells, G. A. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 272, 122-124 (1994).
57. Schulz, K. F., Altman, D. G., Moher, D., Group, C. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ 340, c332 (2010).
58. Grant, S., et al. Developing a reporting guideline for social and psychological intervention trials. Res. Social Work Prac. 23, 595-602 (2013).
59. Liberati, A., et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Med. 6, e1000100 (2009).
60. Shamseer, L., et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ 349, g7647 (2015); erratum 354, i4086 (2016).
61. van 't Veer, A., Giner-Sorolla, R. Pre-registration in social psychology: A discussion and suggested template. J. Exp. Soc. Psychol. 67, 2-12 (2016).
62. Franco, A., Malhotra, N., Simonovits, G. Underreporting in psychology experiments: Evidence from a study registry. Soc. Psychol. Per. Sci. 7, 8-12 (2016).
63. Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., Ioannidis, J. P. Public availability of published research data in high-impact journals. PloS ONE 6, e24357 (2011).
64. Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., Ioannidis, J. P. Reproducible research practices and transparency across the biomedical literature. PLoS Biol. 14, e1002333 (2016).
65. McNutt, M. Taking up TOP. Science 352, 1147 (2016).
66. Park, I. U., Peacey, M. W., Munafò, M. R. Modelling the effects of subjective and objective decision making in scientific peer review. Nature 506, 93-96 (2014).
67. Button, K. S., Bal, L., Clark, A. G., Shipley, T. Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review. BMC Psychol. 4, 59 (2016).
68. Berg, J. M., et al. Preprints for the life sciences. Science 352, 899-901 (2016).
69. Nosek, B. A., Bar-Anan, T. Scientific utopia: I. Opening scientific communication. Psychol. Inq. 23, 217-243 (2012).
70. Walsh, E., Rooney, M., Appleby, L., Wilkinson, G. Open peer review: A randomised trial. Brit. J. Psychiat. 176, 47-51 (2000).
71. Smaldino, P. E., McElreath, R. The natural selection of bad science. R. Soc. Open Sci. 3, 160384 (2016).
72. Higginson, A. D., Munafò, M. Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLoS Biol. 14, e2000995 (2016).
73. Ioannidis, J. P. How to make more published research true. PLoS Med. 11, e1001747 (2014).
74. Eklund, A., Nichols, T. E., Knutsson, H. Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proc. Natl Acad. Sci. USA 113, 7900-7905 (2016).
75. Kidwell, M. C., et al. Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biol. 14, e1002456 (2016).
76. Munafò, M., et al. Scientific rigor and the art of motorcycle maintenance. Nat. Biotechnol. 32, 871-873 (2014).
77. Kass, R. E., et al. Ten simple rules for effective statistical practice. PLoS Comput. Biol. 12, e1004961 (2016).
78. Schweinsberg, M., et al. The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. J. Exp. Psychol. Gen. 66, 55-67 (2016).
79. Stevens, A., et al. Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: Systematic review. BMJ 348, g3804 (2014).
80. Kilkenny, C., et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PloS ONE 4, e7824 (2009).
81. Baker, D., Lidster, K., Sottomayor, A., Amor, S. Two years later: Journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol. 12, e1001756 (2014).
82. Gulin, J. E., Rocco, D. M., Garcia-Bournissen, F. Quality of reporting and adherence to ARRIVE guidelines in animal studies for Chagas disease preclinical drug research: A systematic review. PLoS Negl. Trop. Dis. 9, e0004194 (2015).
83. Liu, Y., et al. Adherence to ARRIVE guidelines in Chinese journal reports on neoplasms in animals. PloS ONE 11, e0154657 (2016).
84. Gotzsche, P. C., Ioannidis, J. P. Content area experts as authors: Helpful or harmful for systematic reviews and meta-analyses BMJ 345, e7031 (2012).
85. Morey, R. D., et al. The Peer Reviewers' Openness Initiative: Incentivizing open research practices through peer review. R. Soc. Open Sci. 3, 150547 (2016).
86. Simmons, J. P., Nelson, L. D., Simonsohn, U. A 21 word solution. Preprint at http://dx.doi.org/10.2139/ssrn.2160588(2012).
87. Eich, E. Business not as usual. Psychol. Sci. 25, 3-6 (2014).