Reproducibility of preclinical animal research improves with heterogeneity of study samples

PLoS Biology

Volumn 16, Issue 2, 2018, Pages

  Voelkl, Bernhard ^a   Vogt, Lucile ^a   Sena, Emily S ^b   Würbel, Hanno ^a  

^a UNIVERSITY OF BERN   (Switzerland)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY; ANIMAL RESEARCH; ARTICLE; BREAST CANCER; CEREBROVASCULAR ACCIDENT; DISEASE MODEL; HEART INFARCTION; NONHUMAN; PRECLINICAL STUDY; PROBABILITY; REPRODUCIBILITY; RESEARCH; SAMPLE; SAMPLE SIZE; SIMULATION; STANDARDIZATION; STUDY DESIGN; STUDY SAMPLE HETEROGENEITY; VALIDATION PROCESS; ANIMAL; ANIMAL EXPERIMENT; LABORATORY; METHODOLOGY; ORGANIZATION AND MANAGEMENT; STANDARDS;

ANIMAL EXPERIMENTATION; ANIMALS; DISEASE MODELS, ANIMAL; LABORATORIES; PROBABILITY; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE;

EID: 85043706933     PISSN: 15449173     EISSN: 15457885     Source Type: Journal    
DOI: 10.1371/journal.pbio.2003693     Document Type: Article

References (53)

1. Loken E, Gelman A, Measurement error and the replication crisis. Science. 2017; 355: 584–585. doi: 10.1126/science.aal3618 28183939
2. Freedman L, Cockburn IM, Simcoe TS, The economics of reproducibility in preclinical research. PLoS Biol. 2015; 13: e1002165. doi: 10.1371/journal.pbio.1002165 26057340
3. Ioannidis JPA, Why most published research findings are false. PLoS Med. 2005; 2: 696–701.
4. Ioannidis JPA, Fanelli D, Dunne DD, Goodman SN, Meta-research: Evaluation and improvement of research methods and practices. PLoS Biol. 2015; 13: e1002264. doi: 10.1371/journal.pbio.1002264 26431313
5. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, du Sert NP, et al. A manifesto for reproducible science. Nat Hum Behav. 2017; 1: 0021.
6. Begley CG, Ellis LM, Drug development: Raise standards for preclinical cancer research. Nature. 2012; 483: 531–533. doi: 10.1038/483531a 22460880
7. Prinz F, Schlange T, Asadullah K, Believe it or not: How much can we rely on published data on potential drug targets?Nat Rev Drug Discov. 2011; 10: 712. doi: 10.1038/nrd3439-c1 21892149
8. Goodman SN, Fanelli D, Ioannidis JPA, What does research reproducibility mean?Sci Transl Med. 2015; 8: 341ps12–341ps12.
9. Beynen AC, Gärtner K, van Zutphen LFM, Zutphen LFM, Baumans V, Beynen AC, Standardization of animal experimentation. In: Principles of laboratory animal science. 2nd ed. Amsterdam: Elsevier Ltd; 2003. pp. 103–110.
10. Fisher RA, The design of experiments. Edinburgh: Oliver and Boyd; 1935.
11. Würbel H, Behaviour and the standardization fallacy. Nat Genet. 2000; 26: 263. doi: 10.1038/81541 11062457
12. Voelkl B, Würbel H, Reproducibility crisis: Are we ignoring reaction norms?Trends Pharmacol Sci. 2016; 37: 509–510. doi: 10.1016/j.tips.2016.05.003 27211784
13. Crabbe JC, Wahlsten D, Dudek BC, Sibilia M, Wagner EF, Genetics of mouse behavior: Interactions with laboratory environment. Science. 1999; 284: 1670–1672. 10356397
14. Richter SH, Garner JP, Auer C, Kunert J, Würbel H, Systematic variation improves reproducibility of animal experiments. Nat Methods. 2010; 7: 167–168. doi: 10.1038/nmeth0310-167 20195246
15. Richter SH, Garner JP, Würbel H, Environmental standardization: Cure or cause of poor reproducibility in animal experiments?Nat Methods. 2009; 6: 257–261. doi: 10.1038/nmeth.1312 19333241
16. Kafkafi N, Golani I, Jaljuli I, Morgan H, Sarig T, Würbel H, et al. Addressing reproducibility in single-laboratory phenotyping experiments. Nat Methods. 2016; 14: 462–464.
17. Karp NA, Speak AO, White JK, Adams DJ, Hrabe´ de Angelis M, Hérault Y, Mott RF, Impact of temporal variation on design and analysis of mouse knockout phenotyping studies. PLoS ONE. 2014; 9: e111239. doi: 10.1371/journal.pone.0111239 25343444
18. Schoener TW, The newest synthesis: Understanding ecological dynamics. Science. 2011; 331: 426–429.
19. Würbel H, Behavioral phenotyping enhanced—beyond (environmental) standardization. Genes Brain Behav. 2002; 1: 3–8. 12886944
20. Llovera G, Hofmann K, Roth S, Salas-Pérdomo A, Ferrer-Ferrer M, Perego M, et al. Results of a preclinical randomized controlled multicenter trial (pRCT): Anti-CD49d treatment for acute brain ischemia. Sci Transl Med. 2015; 7: 299ra121–299ra121. doi: 10.1126/scitranslmed.aaa9853 26246166
21. Wagenmakers EJ, Beek T, Dijkhoff L, Gronau QF, Acosta A, Adams RB, Jret al. Registered replication report: Strack, Martin, and Stepper (1988). Persp Psychol Sci. 2016; 11: 917–28.
22. Hagger MS, Chatzisarantis NL, Alberts H, Anggono CO, Batailler C, Birt AR, et al. A multilab preregistered replication of the ego-depletion effect. Persp Psychol Sci. 2016; 11: 546–73.
23. Kahan BC, Morris TP, Analysis of multicentre trials with continuous outcomes: When and how should we account for centre effects?Stat Med. 2013; 32: 1136–1149. doi: 10.1002/sim.5667 23112128
24. Cornfield J, Randomization by group: A formal analysis. Am J Epidemiol. 1978; 108: 100–102. 707470
25. Localio AR, Berlin JA, Ten Have TR, Kimmel SE, Adjustments for center in multicenter studies: An overview. Ann Intern Med. 2001; 135: 112–123. 11453711
26. Lefer DJ, Bolli R, Development of an NIH consortium for preclinicAl AssESsment of CARdioprotective therapies (CAESAR): A paradigm shift in studies of infarct size limitation. J Cardiovasc Pharmacol Ther. 2011; 16: 332–339. doi: 10.1177/1074248411414155 21821536
27. Multi-Part website. www.multi-part.org. Accessed on 10 July 2017.
28. Collaborative Approach to Meta Analysis and Review of Animal Data from Experimental Studies (CAMARADES) website. www.camarades.info. Accessed on 30 September 2015.
29. Van Der Worp HB, Sena ES, Donnan GA, Howells DW, Macleod MR, Hypothermia in animal models of acute ischaemic stroke: A systematic review and meta-analysis. Brain. 2007; 130: 3063–3074. doi: 10.1093/brain/awm083 17478443
30. Wu T-C, Grotta JC, Hypothermia for acute ischaemic stroke. Lancet Neurol. 2013; 12: 275–284. doi: 10.1016/S1474-4422(13)70013-9 23415567
31. Reith J, Jørgensen HS, Pedersen PM, Nakamaya H, Jeppesen LL, Olsen TS, Raaschou HO, Body temperature in acute stroke: Relation to stroke severity, infarct size, mortality, and outcome. Lancet. 1996; 347: 422–425. 8618482
32. Forstmeier W, Wagenmakers E-J, Parker TH, Detecting and avoiding likely false-positive findings—a practical guide. Biol Rev. 2017; 92: 1941–1968. doi: 10.1111/brv.12315 27879038
33. Rooke ED, Vesterinen HM, Sena ES, Egan KJ, Macleod MR, Dopamine agonists in animal models of Parkinson’s disease: a systematic review and meta-analysis. Parkinsonism Relat D. 2011; 17: 313–320.
34. Sena ES, Van Der Worp HB, Bath PM, Howells DW, Macleod MR, Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 2010; 8: e1000344. doi: 10.1371/journal.pbio.1000344 20361022
35. InHout J, Ioannidis JPA, Borm GF, Obtaining evidence by a single well-powered trial or several modestly powered trials. Stat Meth Med Res. 2012; 25: 538–552.
36. Food and Drug AdministrationGuidance for Industry: Providing evidence of effectiveness for human drug and biological products. Maryland: United States Food and Drug Administration;1998.
37. Committee for Proprietary Medicinal Products. Points to consider on application with 1. Meta-analyses 2. One pivotal study. London: European Agency for the Evaluation of Medicinal Products; 2001.
38. Sorge RE, Martin LJ, Isbester KA, Sotocinal SG, Rosen S, Tuttle AH, et al. Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nat Methods. 2014; 11: 629–632. doi: 10.1038/nmeth.2935 24776635
39. Chesler EJ, Wilson SG, Lariviere WR, Rodriguez-Zas SL, Mogil JS, Influences of laboratory environment on behavior. Nat Neurosci. 2012; 5: 1101–1102.
40. Franklin CL, Ericsson AC, Microbiota and reproducibility of rodent models. LabAnimal. 2017; 46: 114–122.
41. Johnson VE, Revised standards for statistical evidence. Proc Natl Acad Sci USA. 2013; 110: 19313–19317. doi: 10.1073/pnas.1313476110 24218581
42. Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E-J, Berk R, et al. Redefine statistical significance. Nat Hum Behav. 2017; 1: 1.
43. Richter SH, Garner JP, Zipser B, Lewejohann L, Sachser N, Schindler B, et al. Effect of population heterogenization on the reproducibility of mouse behavior: A multi-laboratory study. PLoS ONE. 2011; 6: e16461.
44. Russell WMS, Burch RL, The principles of humane experimental technique. London: Methuen; 1959.
45. Würbel H, More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research. Lab Anim. (NY). 2017; 46: 164–166.
46. Sena E, van der Worp HB, Howells D, Macleod M, How can we improve the pre-clinical development of drugs for stroke?Trends Neurosci. 2007; 30: 433–439. doi: 10.1016/j.tins.2007.06.009 17765332
47. Viechtbauer W, Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010; 36: 1–48.
48. Cornfield J, Randomization by group: A formal analysis. Am J Epidemiol. 1978; 108: 100–102. 707470
49. Vierron E, Giraudeau B, Design effect in multicenter studies: Gain or loss of power?BMC Med Res Methodol. 2009; 9: 39. doi: 10.1186/1471-2288-9-39 19538744
50. Reitsma A, Chu R, Thorpe J, McDonald S, Thabane L, Hutton E, Accounting for center in the early external cephalic version trials: An empirical comparison of statistical methods to adjust for center in a multicenter trial with binary outcomes. Trials. 2014; 15: 377. doi: 10.1186/1745-6215-15-377 25257928
51. Raudenbush SW, Edwards L, Hierarchical linear models as generalizations of certain common experimental design models. In: Applied analysis of variance in behavioral science. New York: Dekker; 1993. pp. 459–496.
52. Moerbeek M, Van Breukelen GJP, Berger MPF, A comparison between traditional methods and multilevel regression for the analysis of multicenter intervention studies. J Clin Epidemiol. 2003; 56: 341–350. 12767411
53. Kahan BC, Morris TP, Assessing potential sources of clustering in individually randomised trials. BMC Med Res Methodol. 2013; 13: 58. doi: 10.1186/1471-2288-13-58 23590245