Reproducibility in science: Improving the standard for basic and preclinical research

Circulation Research

Volumn 116, Issue 1, 2015, Pages 116-126

  Begley, C Glenn ^a,b   Ioannidis, John P A ^a,b  

^a TetraLogic Pharmaceuticals   (United States)

^b STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

funding; journals; research integrity; universities

Indexed keywords

BUDGET; CLINICAL PRACTICE; DRUG INDUSTRY; FUNDING; INTERPERSONAL COMMUNICATION; MEDICAL RESEARCH; PRACTICE GUIDELINE; PRIORITY JOURNAL; REPRODUCIBILITY; RESPONSIBILITY; REVIEW; SCIENTIFIC LITERATURE; STANDARD; ANIMAL; DRUG SCREENING; HUMAN; PROCEDURES; STANDARDS;

ANIMALS; BIOMEDICAL RESEARCH; DRUG EVALUATION, PRECLINICAL; HUMANS; REPRODUCIBILITY OF RESULTS;

EID: 84920769131     PISSN: 00097330     EISSN: 15244571     Source Type: Journal    
DOI: 10.1161/CIRCRESAHA.114.303819     Document Type: Review

References (103)

1. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10:712. doi: 10.1038/nrd3439-c1.
2. Begley CG, Ellis L. Drug development: raise standards for preclinical research. Nature. 2012;483:531-533. doi: 10.1038/483531a.
3. Peers IS, Ceuppens PR, Harbron C. In search of preclinical robustness. Nat Rev Drug Discov. 2012;11:733-734. doi: 10.1038/nrd3849.
4. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86-89. doi: 10.1016/S0140-6736(09)60329-9.
5. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383:156-165. doi: 10.1016/S0140-6736(13)62229-1.
6. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383:166-175. doi: 10.1016/S0140-6736(13)62227-8.
7. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267-276. doi: 10.1016/S0140-6736(13)62228-X.
8. Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, Macleod M, Wisely J, Chalmers I. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383:176-185. doi: 10.1016/S0140-6736(13)62297-7.
9. Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383:257-266. doi: 10.1016/S0140-6736(13)62296-5.
10. Young SS, Bang H, Oktay K. Cereal-induced gender selection? Most likely a multiple testing false positive. Proc Biol Sci. 2009;276:1211-1212; discussion 1213. doi: 10.1098/rspb.2008.1405.
11. Young SS, Karr A. Deming, data and observational studies: a process out of control and needing fixing. Significance. 2011:9:122-126.
12. Young SS, Miller HI. Are medical articles true on health, disease? Sadly, not as often as you might think. Genetic Engineering and Biotechnology News; 2014: 34.
13. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2:696-701. doi: 10.1371/journal.pmed.0020124
14. Hackam DG, Redelmeier DA. Translation of research evidence from animals to humans. JAMA. 2006;296:1731-1732. doi: 10.1001/jama.296.14.1731.
15. Perel P, Roberts I, Sena E, Wheble P, Briscoe C, Sandercock P, Macleod M, Mignini LE, Jayaram P, Khan KS. Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ. 2007;334:197-200. doi: 10.1136/bmj.39048.407928.BE.
16. Booth B, 2011 http://lifescivc.com/2011/03/academic-bias-biotech-failures/
17. Prasad V, Cifu A, Ioannidis JAP. Reversals of established medical practices evidence to abandon ship. J Am Med Assoc 2012;307:37-38. doi: 10.1001/jama.2011.1960.
18. Begley CG. Six red flags for suspect work. Nature. 2013;497:433-434. doi: 10.1038/497433a.
19. Rosenthal R, Lawson R. A longitudinal study of the effects of experimenter bias on the operant learning of laboratory rats. J Psychiatr Res. 1964;2:61-72.
20. Nickerson RS. Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol. 1998;2:175-220.
21. Mynatta CR, Doherty ME, Tweney RD. Confirmation bias in a simulated research environment: an experimental study of scientific inference. Q J Exp Psychol. 1977;29:85-95.
22. Ioannidis JP, Allison DB, Ball CA, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009;41:149-155. doi: 10.1038/ng.295.
23. Baggerly KA, Coombes KR. Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology. Ann Appl Stat. 2009;3:1309-1334.
24. Sena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 2010;8:e1000344. doi: 10.1371/journal. pbio.1000344.
25. Nekrutenko A, Taylor J. Next-generation sequencing data interpretation: enhancing reproducibility and accessibility. Nat Rev Genet. 2012;13: 667-672. doi: 10.1038/nrg3305.
26. Perrin S. Make mouse studies work. Nature. 2014;507:423-425.
27. Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, Howells DW, Al-Shahi Salman R, Macleod MR, Ioannidis JP. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol. 2013;11:e1001609. doi: 10.1371/journal.pbio.1001609.
28. Lazic SE, Essioux L. Improving basic and translational science by accounting for litter-to-litter variation in animal models. BMC Neurosci. 2013;14:37. doi: 10.1186/1471-2202-14-37.
29. Haibe-Kains B, El-Hachem N, Birkbak NJ, Jin AC, Beck AH, Aerts HJ, Quackenbush J. Inconsistency in large pharmacogenomic studies. Nature. 2013;504:389-393. doi: 10.1038/nature12831.
30. Witwer KW. Data submission and quality in microarray-based microRNA profiling. Clin Chem. 2013;59:392-400. doi: 10.1373/clinchem.2012.193813.
31. Elliott S, Busse L, Bass MB, Lu H, Sarosi I, Sinclair AM, Spahr C, Um M, Van G, Begley CG. Anti-Epo receptor antibodies do not predict Epo receptor expression. Blood. 2006;107:1892-1895. doi: 10.1182/blood-2005-10-4066.
32. Prassas I, Brinc D, Farkona S, Leung F, Dimitromanolakis A, Chrystoja CC, Brand R, Kulasingam V, Blasutig IM, Diamandis EP. False biomarker discovery due to reactivity of a commercial ELISA for CUZD1 with cancer antigen CA125. Clin Chem. 2014;60:381-388. doi: 10.1373/clinchem.2013.215236.
33. Stodden V, Guo P, Ma Z. Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS One. 2013;8:e67111. doi: 10.1371/journal.pone.0067111.
34. Baker D, Lidster K, Sottomayor A, Amor S. Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for preclinical animal studies. PLoS Biol. 2014;12:e1001756. doi: 10.1371/journal. pbio.1001756.
35. Vaux D. Know when your numbers are significant. Nature. 2012;492:180-181.
36. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365-376. doi: 10.1038/nrn3475.
37. Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med. 2013;10:e1001489. doi: 10.1371/journal.pmed.1001489.
38. Kenakin T, Bylund DB, Toews ML, Mullane K, Winquist RJ, Williams M. Replicated, replicable and relevant-target engagement and pharmacological experimentation in the 21st century. Biochem Pharmacol. 2014;87: 64-77. doi: 10.1016/j.bcp.2013.10.024.
39. McGonigle P, Ruggeri B. Animal models of human disease: challenges in enabling translation. Biochem Pharmacol. 2014;87:162-171. doi: 10.1016/j.bcp.2013.08.006.
40. Winquist RJ, Mullane K, Williams M. The fall and rise of pharmacology-(re-)defining the discipline? Biochem Pharmacol. 2014;87:4-24. doi: 10.1016/j.bcp.2013.09.011.
41. Marino MJ. The use and misuse of statistical methodologies in pharmacology research. Biochem Pharmacol. 2014;87:78-92. doi: 10.1016/j. bcp.2013.05.017.
42. Sugden LA, Tackett MR, Savva YA, Thompson WA, Lawrence CE. Assessing the validity and reproducibility of genome-scale predictions. Bioinformatics. 2013;29:2844-2851. doi: 10.1093/bioinformatics/btt508.
43. Plant AL, Parker GC. Translating stem cell research from the bench to the clinic: a need for better quality data. Stem Cells Dev. 2013;22:2457-2458. doi: 10.1089/scd.2013.0188.
44. Smith MA, Houghton P. A proposal regarding reporting of in vitro testing results. Clin Cancer Res. 2013;19:2828-2833. doi: 10.1158/1078-0432. CCR-13-0043.
45. Davis BJ, Erlanson DA. Learning from our mistakes: the 'unknown knowns' in fragment screening. Bioorg Med Chem Lett. 2013;23:2844-2852. doi: 10.1016/j.bmcl.2013.03.028.
46. Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9:e1003285. doi: 10.1371/journal.pcbi.1003285.
47. Simeon-Dubach D, Burt AD, Hall PA. Quality really matters: the need to improve specimen quality in biomedical research. J Pathol. 2012;228:431-433. doi: 10.1002/path.4117.
48. Kepes S, McDaniel MA. How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology-Perspectives on Science and Practice 2013;6:252-268.
49. Boyack KW, Klavans R, Sorensen AA, Ioannidis JP. A list of highly influential biomedical researchers, 1996-2011. Eur J Clin Invest. 2013;43:1339-1365. doi: 10.1111/eci.12171.
50. Collins FS, Tabak LA. NIH plans to enhance reproducibility. Nature. 2014;505:612-613.
51. Fanelli D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One. 2009;4:e5738. doi: 10.1371/journal.pone.0005738.
52. Bissell M. The risks of the replication drive. Nature. 2013;503:333-334.
53. Fang FC, Casadevall A. Reforming science: structural reforms. Infect Immun. 2012;80:897-901. doi: 10.1128/IAI.06184-11.
54. Casadevall A, Fang FC. Reforming science: methodological and cultural reforms. Infect Immun. 2012;80:891-896. doi: 10.1128/IAI.06183-11.
55. Alsheikh-Ali AA, Qureshi W, Al-Mallah MH, Ioannidis JP. Public availability of published research data in high-impact journals. PLoS One. 2011;6:e24357. doi: 10.1371/journal.pone.0024357.
56. Peng RD, Dominici F, Zeger SL. Reproducible epidemiologic research. Am J Epidemiol. 2006;163:783-789. doi: 10.1093/aje/kwj093.
57. Peng RD. Reproducible research and Biostatistics. Biostatistics. 2009;10:405-408. doi: 10.1093/biostatistics/kxp014.
58. Hyndman RJ. Encouraging replication and reproducible research. Int J Forecasting. 2010;26:2-3.
59. Peng RD. Reproducible research in computational science. Science. 2011;334:1226-1227. doi: 10.1126/science.1213847.
60. Roy Choudhury K, Gibson R. Reproducible research in medical imaging. Mol Imaging Biol. 2012;14:395-396. doi: 10.1007/s11307-012-0569-8.
61. LeVeque RJ, Mitchell IM, Stodden V. Reproducible research for scientific computing: tools and strategies for changing the culture. Comput Sci Eng. 2012;14:13-17.
62. Bosenberg M, Arnheiter H, Kelsh R. The test of time. Pigment Cell Melanoma Res. 2013;26:157. doi: 10.1111/pcmr.12071.
63. Ioannidis JP, Khoury MJ. Improving validation practices in "omics" research. Science. 2011;334:1230-1232. doi: 10.1126/science.1211811.
64. Hayden EC. Personalized cancer treatments suffer setback. Nature 2013; doi: 10.1038/nature.2013.14238
65. Carlin SP, Standing LG. Is intelligence enhanced by letter priming? A failure to replicate the results of Ciani and Sheldon (2010). Psychol Rep. 2013;112:533-544. doi: 10.2466/04.03.PR0.112.2.533-544.
66. Fanelli D. "Positive" results increase down the hierarchy of the sciences. PLoS One. 2010;5:e10068. doi: 10.1371/journal.pone.0010068
67. Baggerly K. More data, please! Clin Chem. 2013;59:459-461. doi: 10.1373/clinchem.2012.200501.
68. Poste G. Biospecimens, biomarkers, and burgeoning data: the imperative for more rigorous research standards. Trends Mol Med. 2012;18:717-722. doi: 10.1016/j.molmed.2012.09.003.
69. Pashler H, Wagenmakers EJ. Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspect Psychol Sci 2012;7:528-530. doi: 10.1177/1745691612465253.
70. Bakker M, van Dijk A, Wicherts JM. The rules of the game called psychological science. Perspect Psychol Sci 2012;7:543-554.
71. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14:365-376. doi: 10.1038/nrn3475.
72. Nosek BA, Spies JR, Motyl, M. Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect Psychol Sci 2012;7:615-631.
73. Alexander A, Barnett-Cowan M, Bartmess, E, et al. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect Psychol Sci 2012;7:657-660.
74. Wagenmakers EJ, Forstmann BU. Rewarding high-power replication research. Cortex. 2014;51:105-106. doi: 10.1016/j.cortex.2013.09.010.
75. Kraus WL. Do you see what I see? Quality, reliability, and reproducibility in biomedical research. Mol Endocrinol. 2014;28:277-280. doi: 10.1210/me.2014-1036.
76. Hooijmans CR, Ritskes-Hoitinga M. Progress in using systematic reviews of animal studies to improve translational research. PLoS Med. 2013;10:e1001482. doi: 10.1371/journal.pmed.1001482.
77. Macleod MR, O'Collins T, Howells DW, Donnan GA. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke. 2004;35:1203-1208. doi: 10.1161/01. STR.0000125719.25853.20.
78. Glaeser EL. Researcher Incentives and Empirical Methods [dissertation]. Harvard Institute of Economic Research. Discussion Paper Number 2122; 2006.
79. Cachat J, Bandrowski A, Grethe JS, Gupta A, Astakhov V, Imam F, Larson SD, Martone ME. A survey of the neuroscience resource landscape: perspectives from the neuroscience information framework. Int Rev Neurobiol. 2012;103:39-68. doi: 10.1016/B978-0-12-388408-4.00003-4.
80. Lemmon VP, Abeyruwan S, Visser U, Bixby JL. Facilitating transparency in spinal cord injury studies using data standards and ontologies. Neural Regen Res. 2014;9:6-7. doi: 10.4103/1673-5374.125322.
81. Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012;490:187-191. doi: 10.1038/nature11556.
82. Ross DT, Scherf U, Eisen MB, et al. Systematic variation in gene expression patterns in human cancer cell lines. Nat Genet. 2000;24:227-235. doi: 10.1038/73432.
83. Rae JM, Creighton CJ, Meck JM, Haddad BR, Johnson MD. MDA-MB-435 cells are derived from M14 melanoma cells-a loss for breast cancer, but a boon for melanoma research. Breast Cancer Res Treat. 2007;104:13-19. doi: 10.1007/s10549-006-9392-8.
84. Wadman M. NIH mulls rules for validating key results. Nature. 2013;500:14-16. doi: 10.1038/500014a.
85. Doshi P, Goodman SN, Ioannidis JP. Raw data from clinical trials: within reach? Trends Pharmacol Sci. 2013;34:645-647. doi: 10.1016/j. tips.2013.10.006.
86. Goldacre B. What the Tamiflu saga tells us about drug trials and big pharma. The Guardian 2014; http://www.theguardian.com/business/2014/apr/10/tamiflu-saga-drug-trials-big-pharma
87. Djulbegovic B, Hozo I, Ioannidis JP. Improving the drug development process: more not less randomized trials. JAMA. 2014;311:355-356. doi: 10.1001/jama.2013.283742.
88. Snow DM. Commentary on: "Facilitating transparency in spinal cord injury studies using data standards and ontologies". Neural Regen Res. 2014;9:8-9. doi: 10.4103/1673-5374.125323.
89. Shamoo A, Aannau Z. Ensuring scientific integrity. Nature 1987;327:550.
90. Van Noorden R. Meeting targets lab lapses. Nature. 2013;497:300-301. doi: 10.1038/497300a.
91. Echols HG. Operators and Promoters: The Story of Molecular Biology and Its Creators. University of California Press; 2001:341-344.
92. The Maxine Singer Papers. Profiles in Science, National Library of Medicine. http://profiles.nlm.nih.gov/ps/retrieve/Collection/CID/DJ
93. The Paul Berg Papers. Profiles in Science. National Library of Medicine. http://profiles.nlm.nih.gov/ps/retrieve/Narrative/CD/p-nid/257
94. The President. Removing barriers to responsible scientific research involving human stem cells. Federal Register 2009;74:10667-10668. http://www.gpo.gov/fdsys/pkg/FR-2009-03-11/pdf/E9-5441.pdf
95. Chosewood CL, Wilson DE. eds. Biosafety in Microbiological and Biomedical Laboratories. 5th ed. US Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Institutes of Health. HHS Publication Number (CDC) 21-1112; 2009 (http://www.cdc.gov/biosafety/publications/bmbl5/bmbl.pdf).
96. Shekman R. How journals like nature, cell and science are damaging science. The Guardian. 2013: Dec 9.
97. Announcement. Reducing our Irreproducibility. Nature 2013;496:398.
98. Begley CG. The test of time and editorial responsibility. Pigment Cell Melanoma Res. 2013;26:603-604. doi: 10.1111/pcmr.12143.
99. Ioannidis JP. Journals should publish all "null" results and should sparingly publish "positive" results. Cancer Epidemiol Biomarkers Prev. 2006;15:186. doi: 10.1158/1055-9965.EPI-05-0921.
100. Nicholson JM, Ioannidis JP. Research grants: conform and be funded. Nature. 2012;492:34-36. doi: 10.1038/492034a.
101. Ioannidis JAP. More time for research: fund people not projects Nature 2011;477:529-531.
102. Ioannidis JAP, Khoury MJ. Assessing value in biomedical research: the PQRST of appraisal and reward. JAMA. 2014;312:483-484. doi: 10.1001/jama.2014.6932.
103. Mobley A, Linder SK, Braeuer R, Ellis LM, Zwelling L. A survey on data reproducibility in cancer research provides insights into our limited ability to translate findings from the laboratory to the clinic. PLoS One. 2013;8:e63221. doi: 10.1371/journal.pone.0063221.