Methodological Rigor in Preclinical Cardiovascular Studies

Circulation Research

Volumn 120, Issue 12, 2017, Pages 1916-1926

  Ramirez, F Daniel ^a,b,c   Motazedian, Pouya ^a   Jung, Richard G ^b,d   Di Santo, Pietro ^a,b   Macdonald, Zachary D ^a,b   Moreland, Robert ^b,f   Simard, Trevor ^b,d   Clancy, Aisling A ^a,f   Russo, Juan J ^a,b   Welch, Vivian A ^e,g,h   Wells, George A ^a,d,e   Hibbert, Benjamin ^b,d  

^a Division of Cardiology   (Canada)

^b CAPITAL Research Group   (Canada)

^c Vascular Biology and Experimental Medicine Laboratory   (Canada)

^d UNIVERSITY OF OTTAWA HEART INSTITUTE   (Canada)

^e Public Health and Preventive Medicine   (Canada)

^f Department of Radiology ^*  (Canada)

^g BRUYÈRE RESEARCH INSTITUTE   (Canada)

^h UNIVERSITY OF OTTAWA   (Canada)

Author keywords

animal models; bias; biomedical research; cardiovascular diseases; reproducibility of results

Indexed keywords

ARTICLE; CARDIOVASCULAR DISEASE; CEREBROVASCULAR ACCIDENT; GENETIC ANALYSIS; HUMAN; IN VIVO STUDY; NONHUMAN; OUTCOME ASSESSMENT; PATHOPHYSIOLOGY; PRACTICE GUIDELINE; PRECLINICAL STUDY; PREVALENCE; PRIORITY JOURNAL; REPRODUCIBILITY; SAMPLE SIZE; SCIENTIFIC LITERATURE; STUDY DESIGN; TRANSLATIONAL RESEARCH; ANIMAL; CARDIOVASCULAR DISEASES; DISEASE MODEL; METHODOLOGY; PATHOLOGY; PROCEDURES; STANDARDS;

ANIMALS; CARDIOVASCULAR DISEASES; DISEASE MODELS, ANIMAL; DRUG EVALUATION, PRECLINICAL; HUMANS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; TRANSLATIONAL MEDICAL RESEARCH;

EID: 85017183663     PISSN: 00097330     EISSN: 15244571     Source Type: Journal    
DOI: 10.1161/CIRCRESAHA.117.310628     Document Type: Article

References (75)

1. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014; 383: 166-175. doi: 10. 1016/S0140-6736(13)62227-8.
2. Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014; 505: 612-613.
3. Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med. 2013; 10: e1001489. doi: 10. 1371/journal. pmed. 1001489.
4. Pammolli F, Magazzini L, Riccaboni M. The productivity crisis in pharmaceutical R&D. Nat Rev Drug Discov. 2011; 10: 428-438. doi: 10. 1038/ nrd3405.
5. Bunnage ME. Getting pharmaceutical R&D back on target. Nat Chem Biol. 2011; 7: 335-339. doi: 10. 1038/nchembio. 581.
6. van der Worp HB, Howells DW, Sena ES, Porritt MJ, Rewell S, O'Collins V, Macleod MR. Can animal models of disease reliably inform human studies? PLoS Med. 2010; 7: e1000245. doi: 10. 1371/journal. pmed. 1000245.
7. Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012; 490: 187-191. doi: 10. 1038/nature11556.
8. Scott S, Kranz JE, Cole J, Lincecum JM, Thompson K, Kelly N, Bostrom A, Theodoss J, Al-Nakhala BM, Vieira FG, Ramasubbu J, Heywood JA. Design, power, and interpretation of studies in the standard murine model of ALS. Amyotroph Lateral Scler. 2008; 9: 4-15. doi: 10. 1080/17482960701856300.
9. Perel P, Roberts I, Sena E, Wheble P, Briscoe C, Sandercock P, Macleod M, Mignini LE, Jayaram P, Khan KS. Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ. 2007; 334: 197. doi: 10. 1136/bmj. 39048. 407928. BE.
10. Muhlhausler BS, Bloomfield FH, Gillman MW. Whole animal experiments should be more like human randomized controlled trials. PLoS Biol. 2013; 11: e1001481. doi: 10. 1371/journal. pbio. 1001481.
11. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011; 10: 712. doi: 10. 1038/nrd3439-c1.
12. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012; 483: 531-533. doi: 10. 1038/483531a.
13. Begley CG, Ioannidis JP. Reproducibility in science: improving the standard for basic and preclinical research. Circ Res. 2015; 116: 116-126. doi: 10. 1161/CIRCRESAHA. 114. 303819.
14. Baker M. 1, 500 scientists lift the lid on reproducibility. Nature. 2016; 533: 452-454. doi: 10. 1038/533452a.
15. Unreliable research: Trouble at the lab. The Economist. 2013.
16. Lehrer J. The truth wears off: Is there something wrong with the scientific method? The New Yorker. 2010.
17. Freedman DH. Lies, damned lies, and medical science. The Atlantic. 2010.
18. Osherovich L. Hedging against academic risk. Science-Business eXchange. 2011; 4: doi: 10. 1038/scibx. 2011. 1416
19. Begley CG. Six red flags for suspect work. Nature. 2013; 497: 433-434. doi: 10. 1038/497433a.
20. Horn J, de Haan RJ, Vermeulen M, Luiten PG, Limburg M. Nimodipine in animal model experiments of focal cerebral ischemia: a systematic review. Stroke. 2001; 32: 2433-2438.
21. Macleod MR, van der Worp HB, Sena ES, Howells DW, Dirnagl U, Donnan GA. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke. 2008; 39: 2824-2829. doi: 10. 1161/STROKEAHA. 108. 515957.
22. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005; 2: e124. doi: 10. 1371/journal. pmed. 0020124.
23. Hackam DG, Redelmeier DA. Translation of research evidence from animals to humans. JAMA. 2006; 296: 1731-1732. doi: 10. 1001/ jama. 296. 14. 1731.
24. Hay M, Thomas DW, Craighead JL, Economides C, Rosenthal J. Clinical development success rates for investigational drugs. Nat Biotechnol. 2014; 32: 40-51. doi: 10. 1038/nbt. 2786.
25. National Institutes of Health. Principles and guidelines for reporting preclinical research. https: //www. nih. gov/research-training/rigor-reproducibility/ principles-guidelines-reporting-preclinical-research. Accessed December 2, 2016.
26. Clayton JA, Collins FS. Policy: NIH to balance sex in cell and animal studies. Nature. 2014; 509: 282-283.
27. Daugherty A, Hegele RA, Mackman N, Rader DJ, Schmidt AM, Weber C. Complying with the national institutes of health guidelines and principles for rigor and reproducibility: refutations. Arterioscler Thromb Vasc Biol. 2016; 36: 1303-1304. doi: 10. 1161/ATVBAHA. 116. 307906.
28. Vahidy F, Schäbitz WR, Fisher M, Aronowski J. Reporting standards for preclinical studies of stroke therapy. Stroke. 2016; 47: 2435-2438. doi: 10. 1161/STROKEAHA. 116. 013643.
29. Ramirez FD, Motazedian P, Jung RG, Di Santo P, MacDonald Z, Simard T, Clancy AA, Russo JJ, Welch V, Wells GA, Hibbert B. Sex bias is increasingly prevalent in preclinical cardiovascular research: implications for translational medicine and health equity for women: a systematic assessment of leading cardiovascular journals over a 10-year period. Circulation. 2017; 135: 625-626. doi: 10. 1161/CIRCULATIONAHA. 116. 026668.
30. Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol. 2014; 12: e1001863. doi: 10. 1371/journal. pbio. 1001863.
31. Unger EF. All is not well in the world of translational research. J Am Coll Cardiol. 2007; 50: 738-740. doi: 10. 1016/j. jacc. 2007. 04. 067.
32. Minnerup J, Zentsch V, Schmidt A, Fisher M, Schäbitz WR. Methodological quality of experimental stroke studies published in the stroke journal: time trends and effect of the basic science checklist. Stroke. 2016; 47: 267-272. doi: 10. 1161/STROKEAHA. 115. 011695.
33. Ranasinghe I, Shojaee A, Bikdeli B, Gupta A, Chen R, Ross JS, Masoudi FA, Spertus JA, Nallamothu BK, Krumholz HM. Poorly cited articles in peer-reviewed cardiovascular journals from 1997 to 2007: analysis of 5-year citation rates. Circulation. 2015; 131: 1755-1762. doi: 10. 1161/ CIRCULATIONAHA. 114. 015080.
34. Danthi N, Wu CO, Shi P, Lauer M. Percentile ranking and citation impact of a large cohort of National Heart, Lung, and Blood Institute-funded cardiovascular R01 grants. Circ Res. 2014; 114: 600-606. doi: 10. 1161/ CIRCRESAHA. 114. 302656.
35. Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior publication productivity, grant percentile ranking, and topic-normalized citation impact of NHLBI cardiovascular R01 grants. Circ Res. 2014; 115: 617-624. doi: 10. 1161/CIRCRESAHA. 115. 304766.
36. Lauer MS, Danthi NS, Kaltman J, Wu C. Predicting productivity returns on investment: thirty years of peer review, grant funding, and publication of highly cited papers at the National Heart, Lung, and Blood Institute. Circ Res. 2015; 117: 239-243. doi: 10. 1161/CIRCRESAHA. 115. 306830.
37. Fabsitz RR, Papanicolaou GJ, Sholinsky P, Coady SA, Jaquish CE, Nelson CR, Olson JL, Puggal MA, Purkiser KL, Srinivas PR, Wei GS, Wolz M, Sorlie PD. Impact of National Heart, Lung, and Blood Institute-Supported Cardiovascular Epidemiology Research, 1998 to 2012. Circulation. 2015; 132: 2028-2033. doi: 10. 1161/CIRCULATIONAHA. 114. 014147.
38. Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009; 302: 1092-1096. doi: 10. 1001/ jama. 2009. 1307.
39. Steen RG, Casadevall A, Fang FC. Why has the number of scientific retractions increased? PLoS One. 2013; 8: e68397. doi: 10. 1371/journal. pone. 0068397.
40. Concato J, Peduzzi P, Holford TR, Feinstein AR. Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy. J Clin Epidemiol. 1995; 48: 1495-1501.
41. Peduzzi P, Concato J, Feinstein AR, Holford TR. Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol. 1995; 48: 1503-1510.
42. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996; 49: 1373-1379.
43. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014; 383: 267-276. doi: 10. 1016/ S0140-6736(13)62228-X.
44. Freedman LP, Cockburn IM, Simcoe TS. The economics of reproducibility in preclinical research. PLoS Biol. 2015; 13: e1002165. doi: 10. 1371/ journal. pbio. 1002165.
45. Macleod MR, O'Collins T, Howells DW, Donnan GA. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke. 2004; 35: 1203-1208. doi: 10. 1161/01. STR. 0000125719. 25853. 20.
46. Macleod MR, O'Collins T, Horky LL, Howells DW, Donnan GA. Systematic review and metaanalysis of the efficacy of FK506 in experimental stroke. J Cereb Blood Flow Metab. 2005; 25: 713-721. doi: 10. 1038/sj. jcbfm. 9600064.
47. Pound P, Ebrahim S, Sandercock P, Bracken MB, Roberts I; Reviewing Animal Trials Systematically (RATS) Group. Where is the evidence that animal research benefits humans? BMJ. 2004; 328: 514-517. doi: 10. 1136/ bmj. 328. 7438. 514.
48. Clayton JA. Studying both sexes: a guiding principle for biomedicine. FASEB J. 2016; 30: 519-524. doi: 10. 1096/fj. 15-279554.
49. Macleod MR, Lawson McLean A, Kyriakopoulou A, et al. Risk of bias in reports of in vivo research: a focus for improvement. PLoS Biol. 2015; 13: e1002273. doi: 10. 1371/journal. pbio. 1002273.
50. Savovi J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012; 157: 429-438. doi: 10. 7326/0003-4819-157-6-201209180-00537.
51. Tzoulaki I, Siontis KC, Ioannidis JP. Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study. BMJ. 2011; 343: d6829.
52. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999; 282: 1061-1066.
53. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006; 174: 469-476. doi: 10. 1503/cmaj. 050090.
54. Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, Hutton J, Altman DG. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One. 2009; 4: e7824. doi: 10. 1371/journal. pone. 0007824.
55. Beery AK, Zucker I. Sex bias in neuroscience and biomedical research. Neurosci Biobehav Rev. 2011; 35: 565-572. doi: 10. 1016/j. neubiorev. 2010. 07. 002.
56. Yoon DY, Mansukhani NA, Stubbs VC, Helenowski IB, Woodruff TK, Kibbe MR. Sex bias exists in basic science and translational surgical research. Surgery. 2014; 156: 508-516. doi: 10. 1016/j. surg. 2014. 07. 001.
57. Reardon S. A mouse's house may ruin experiments. Nature. 2016; 530: 264. doi: 10. 1038/nature. 2016. 19335.
58. Fisher M, Feuerstein G, Howells DW, Hurn PD, Kent TA, Savitz SI, Lo EH; STAIR Group. Update of the stroke therapy academic industry roundtable preclinical recommendations. Stroke. 2009; 40: 2244-2250. doi: 10. 1161/STROKEAHA. 108. 541128.
59. Hooijmans CR, Rovers MM, de Vries RB, Leenaars M, Ritskes-Hoitinga M, Langendam MW. SYRCLE's risk of bias tool for animal studies. BMC Med Res Methodol. 2014; 14: 43. doi: 10. 1186/1471-2288-14-43.
60. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 2010; 8: e1000412. doi: 10. 1371/journal. pbio. 1000412.
61. Announcement: Reducing our irreproducibility. Nature. 2013; 496
62. Hooijmans CR, Leenaars M, Ritskes-Hoitinga M. A gold standard publication checklist to improve the quality of animal studies, to fully integrate the Three Rs, and to make systematic reviews more feasible. Altern Lab Anim. 2010; 38: 167-182.
63. Baker D, Lidster K, Sottomayor A, Amor S. Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for preclinical animal studies. PLoS Biol. 2014; 12: e1001756. doi: 10. 1371/journal. pbio. 1001756.
64. Sandercock P, Roberts I. Systematic reviews of animal experiments. Lancet. 2002; 360: 586. doi: 10. 1016/S0140-6736(02)09812-4.
65. Hirst JA, Howick J, Aronson JK, Roberts N, Perera R, Koshiaris C, Heneghan C. The need for randomization in animal trials: an overview of systematic reviews. PLoS One. 2014; 9: e98856. doi: 10. 1371/journal. pone. 0098856.
66. van Luijk J, Bakker B, Rovers MM, Ritskes-Hoitinga M, de Vries RB, Leenaars M. Systematic reviews of animal studies; missing link in translational research? PLoS One. 2014; 9: e89981. doi: 10. 1371/journal. pone. 0089981.
67. Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS One. 2012; 7: e35621. doi: 10. 1371/journal. pone. 0035621.
68. Stroke Therapy Academic Industry Roundtable (STAIR). Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke. 1999; 30: 2752-2758
69. O'Collins VE, Macleod MR, Donnan GA, Horky LL, van der Worp BH, Howells DW. 1, 026 experimental treatments in acute stroke. Ann Neurol. 2006; 59: 467-477. doi: 10. 1002/ana. 20741.
70. Crossley NA, Sena E, Goehler J, Horn J, van der Worp B, Bath PM, Macleod M, Dirnagl U. Empirical evidence of bias in the design of experimental stroke studies: a metaepidemiologic approach. Stroke. 2008; 39: 929-934. doi: 10. 1161/STROKEAHA. 107. 498725.
71. Sena E, van der Worp HB, Howells D, Macleod M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 2007; 30: 433-439. doi: 10. 1016/j. Tins. 2007. 06. 009.
72. Perrin S. Preclinical research: Make mouse studies work. Nature. 2014; 507: 423-425. doi: 10. 1038/507423a.
73. García-Berthou E, Alcaraz C. Incongruence between test statistics and P values in medical papers. BMC Med Res Methodol. 2004; 4: 13. doi: 10. 1186/1471-2288-4-13.
74. Johnson VE. Revised standards for statistical evidence. Proc Natl Acad Sci U S A. 2013; 110: 19313-19317. doi: 10. 1073/pnas. 1313476110.
75. Ioannidis JP. How to make more published research true. PLoS Med. 2014; 11: e1001747. doi: 10. 1371/journal. pmed. 1001747.