Evidence-based clinical practice: Overview of threats to the validity of evidence and how to minimise them

European Journal of Internal Medicine

Volumn 32, Issue , 2016, Pages 13-21

  Garattini, Silvio ^a   Jakobsen, Janus C ^b,c   Wetterslev, Jørn ^b   Bertelé, Vittorio ^a   Banzi, Rita ^a   Rath, Ana ^d   Neugebauer, Edmund A M ^e   Laville, Martine ^f   Masson, Yvonne ^f   Hivert, Virginie ^f   Eikermann, Michaela ^e   Aydin, Burc ^g   Ngwabyt, Sandra ^d   Martinho, Cecilia ^h   Gerardi, Chiara ^a   Szmigielski, Cezary A ⁱ   Demotes Mainard, Jacques ^j   Gluud, Christian ^b  

^a MARIO NEGRI INSTITUTE FOR PHARMACOLOGICAL RESEARCH   (Italy)

^b Rigshospitalet   (Denmark)

^c HOLBÆK HOSPITAL   (Denmark)

^d INSERM   (France)

^e UNIVERSITY OF WITTEN HERDECKE   (Germany)

^f UNIVERSITÉ DE LYON   (France)

^g DOKUZ EYLUL UNIVERSITY   (Turkey)

^h PORTUGUESE ONCOLOGY INSTITUTE   (Portugal)

ⁱ MEDICAL UNIVERSITY OF WARSAW   (Poland)

^j European Clinical Research Infrastructures Network (ECRIN)   (France)

more..

Author keywords

Evidence based clinical practice; Evidence based medicine; Meta analysis; Randomised clinical trial; Systematic review; Trial Sequential Analysis

Indexed keywords

CLINICAL PRACTICE; CLINICAL RESEARCH; CONTROLLED CLINICAL TRIAL; HUMAN; MEDICINE; META ANALYSIS; OBSERVATIONAL STUDY; RANDOM ERROR; RANDOMIZED CONTROLLED TRIAL; SEQUENTIAL ANALYSIS; SYSTEMATIC ERROR; SYSTEMATIC REVIEW; TRUST; VALIDITY; EVIDENCE BASED MEDICINE; EVIDENCE BASED PRACTICE; LITERATURE; META ANALYSIS (TOPIC); RANDOMIZED CONTROLLED TRIAL (TOPIC); REPRODUCIBILITY;

EVIDENCE-BASED MEDICINE; EVIDENCE-BASED PRACTICE; HUMANS; META-ANALYSIS AS TOPIC; RANDOMIZED CONTROLLED TRIALS AS TOPIC; REPRODUCIBILITY OF RESULTS; REVIEW LITERATURE AS TOPIC;

EID: 84992311807     PISSN: 09536205     EISSN: 18790828     Source Type: Journal    
DOI: 10.1016/j.ejim.2016.03.020     Document Type: Review

References (104)

1. The Library and Information Services Department The Royal College of Physicians of Edinburgh: James Lind Library [Available online at] http://wwwjameslindlibraryorg/ 2003
2. A. Opinel, U. Trohler, C. Gluud, G. Gachelin, G.D. Smith, S.H. Podolsky, and et al. Commentary: the evolution of methods to assess the effects of treatments, illustrated by the development of treatments for diphtheria, 1825-1918 Int J Epidemiol 42 3 2013 662 676
3. P. Heiberg Studier over den statistiske undersøgelsesmetode som hjælpemiddel ved terapeutiske undersøgelser [Studies on the statistical study design as an aid in therapeutic trials] (http://www.jameslindlibrary.org/illustrating/records/studier-over-den-statistiske-undersogelsesmetode-som-hjaelpemid/key-passages) Bibl Laeger 89 1897 1 40
4. C. Gluud, and D. Nikolova Likely country of origin in publications on randomised controlled trials and controlled clinical trials during the last 60 years Trials 8 2007 7
5. The Cochrane Collaboration The Cochrane Library [Avaiable at] http://wwwthecochranelibrarycom/view/0/indexhtml 2014
6. ECRIN-IA Description of the project [Available from] http://www.ecrin.org/index.php?id=141
7. Durisic S, Garattini S, Rath A, Neugebauer EAM, Laville M, Jakobsen JC, Kubiac C, DeMotes-Mainard J, Gluud C: Common barriers to the conduct of randomised clinical trials - the Europeean Clinical Research Infrastructure (ECRIN) perspective Trials, 2016 [to be submitted].
8. Rath A, Ngwabyt S, Durisic S, Garattini S, Neugebauer EAM, Laville M, Jakobsen JC, Kubiac C, DeMotes-Mainard J, Gluud C: Specific barriers to the conduct of randomised clinical trials within rare diseases - the Europeean Clinical Research Infrastructure (ECRIN) perspective Trials, 2016 [to be submitted].
9. Neugebauer EAM, Rath A, Durisic S, Garattini S, Laville M, Jakobsen JC, Kubiac C, DeMotes-Mainard J, Gluud C: Specific barriers to the conduct of randomised clinical trials on medical devices - the Europeean Clinical Research Infrastructure (ECRIN) perspective Trials, 2016 [to be submitted].
10. Laville M, Neugebauer EAM, Rath A, Durisic S, Garattini S, Jakobsen JC, Kubiac C, DeMotes-Mainard J, Gluud C: Specific barriers to the conduct of randomised clinical trials on nutrition - the Europeean Clinical Research Infrastructure (ECRIN) perspective Trials, 2016 [to be submitted].
11. J.P.T. Higgins, and S. Green The Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration 2011 [Available from] http://www.cochrane-handbook.org/
12. F. Keus, J. Wetterslev, C. Gluud, and C.J. van Laarhoven Evidence at a glance: error matrix approach for overviewing available evidence BMC Med Res Methodol 10 2010 90
13. J.C. Jakobsen, and C. Gluud The necessity of randomized clinical trials Br J Med Res 3 4 2013 1453 1468
14. J.J. Deeks, J. Dinnes, R. D'Amico, A.J. Sowden, C. Sakarovitch, F. Song, and et al. Evaluating non-randomised intervention studies Health Technol Assess 7 27 2003 1 173 [iii-x]
15. K. Thorlund, G. Imberger, M. Walsh, R. Chu, C. Gluud, J. Wetterslev, and et al. The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis-a simulation study PLoS One 6 2011 e25491
16. C. Gluud, J. Brok, Y. Gong, and R.L. Koretz Hepatology may have problems with putative surrogate outcome measures J Hepatol 46 4 2007 734 742
17. R. Al-Shahi Salman, E. Beller, J. Kagan, E. Hemminki, R.S. Phillips, J. Savulescu, and et al. Increasing value and reducing waste in biomedical research regulation and management Lancet 383 9912 2014 176 185
18. I. Chalmers, M.B. Bracken, B. Djulbegovic, S. Garattini, J. Grant, A.M. Gulmezoglu, and et al. How to increase value and reduce waste when research priorities are set Lancet 383 9912 2014 156 165
19. I. Chalmers, and P. Glasziou Avoidable waste in the production and reporting of research evidence Lancet 374 9683 2009 86 89
20. P. Glasziou, D.G. Altman, P. Bossuyt, I. Boutron, M. Clarke, S. Julious, and et al. Reducing waste from incomplete or unusable reports of biomedical research Lancet 383 9913 2014 267 276
21. M.R. Macleod, S. Michie, I. Roberts, U. Dirnagl, I. Chalmers, J.P. Ioannidis, and et al. Biomedical research: increasing value, reducing waste Lancet 383 9912 2014 101 104
22. D. Moher, P. Glasziou, I. Chalmers, M. Nasser, P.M. Bossuyt, D.A. Korevaar, and et al. Increasing value and reducing waste in biomedical research: who's listening? Lancet 387 2016 1573 1586
23. J.P. Ioannidis How to make more published research true PLoS Med 11 10 2014 e1001747
24. J.P. Ioannidis Clinical trials: what a waste BMJ 349 2014 g7089
25. J.P. Ioannidis, S. Greenland, M.A. Hlatky, M.J. Khoury, M.R. Macleod, D. Moher, and et al. Increasing value and reducing waste in research design, conduct, and analysis Lancet 383 9912 2014 166 175
26. J.P.A. Ioannidis Why most published research findings are false PLoS Med 2 8 2005 e124
27. D.L. Sackett How to read clinical journals Can Med Assoc J 126 12 1982 1373
28. A.W. Jørgensen, J. Hilden, and P. Gøtzsche Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs Syst Rev 333 782 2006
29. J.C. Jakobsen, C. Gluud, P. Winkel, T. Lange, and J. Wetterslev The thresholds for statistical and clinical significance - a five-step procedure for evaluation of intervention effects in randomised clinical trials BMC Med Res Methodol 14 34 2014
30. J.C. Jakobsen, J. Wetterslev, P. Winkel, T. Lange, and C. Gluud Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods BMC Med Res Methodol 14 1 2014 120
31. S. Borzak, and P.M. Ridker Discordance between meta-analyses and large-scale randomized, controlled trials. Examples from the management of acute myocardial infarction Ann Intern Med 123 11 1995 873 877
32. C.H. Hennekens, and D. DeMets The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses JAMA 302 21 2009 2361 2362
33. J. Stegenga Is meta-analysis the platinum standard of evidence? Stud Hist Philos Biol Biomed Sci 42 4 2011 497 507
34. K.A. Thayer, M.S. Wolfe, A.A. Rooney, A.L. Boyles, J.R. Bucher, and L.S. Birnbaum Intersection of systematic review methodology with the NIH reproducibility initiative Environ Health Perspect 122 7 2014 A176 A177
35. J. Inthout, J.P. Ioannidis, and G.F. Borm Obtaining evidence by a single well-powered trial or several modestly powered trials Stat Methods Med Res 25 2016 538 552
36. E.J. Mills, K. Thorlund, and J.P. Ioannidis Demystifying trial networks and network meta-analysis BMJ 346 2013 f2914
37. A.C. Del Re, G.I. Spielmans, C. Flückiger, and B.E. Wambold Efficacy of new generation antidepressants: differences seem illusory PLoS One 8 6 2013 e63509
38. L. Trinquart, A. Abbe, and P. Ravaud Impact of reporting bias in network meta-analysis of antidepressant placebo-controlled trials PLoS One 7 4 2012 e35219
39. B. Hutton, G. Salanti, D.M. Caldwell, A. Chaimani, C.H. Schmid, C. Cameron, and et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations Ann Intern Med 162 11 2015 777 784
40. L.A. Stewart, M. Clarke, M. Rovers, R.D. Riley, M. Simmonds, G. Stewart, and et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement JAMA 313 16 2015 1657 1665
41. J.P. Ioannidis, A.B. Haidich, M. Pappa, N. Pantazis, S.I. Kokori, M.G. Tektonidou, and et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies JAMA 286 7 2001 821 830
42. L.G. Hemkens, D.G. Contopoulos-Ioannidis, and J.P. Ioannidis Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey BMJ 352 2016 i493
43. T.C. Chalmers Randomize the first patient N Engl J of Med 296 2 1977 107
44. J.P. Ioannidis Are mortality differences detected by administrative data reliable and actionable? JAMA 309 13 2013 1410 1411
45. P. Winkel, and N.F. Zhang Statistical Development of Quality in Medicine 2007 Wiley
46. J. Savović, H. Jones, D. Altman, R. Harris, P. Juni, J. Pildal, and et al. Influence of reported study design characteristics on intervention effect estimates from randomized controlled trials: combined analysis of meta-epidemiologic studies Health Technol Assess 16 35 2012 1 82
47. L. Wood, M. Egger, L.L. Gluud, K.F. Schulz, P. Juni, D.G. Altman, and et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study BMJ 336 7644 2008 601 605
48. J. Savovic, H.E. Jones, D.G. Altman, R.J. Harris, P. Juni, J. Pildal, and et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials Ann Intern Med 157 6 2012 429 438
49. R.J. Little, R. D'Agostino, M.L. Cohen, K. Dickersin, S.S. Emerson, J.T. Farrar, and et al. The prevention and treatment of missing data in clinical trials N Engl J Med 367 14 2012 1355 1360
50. J.A.C. Sterne, I.R. White, J.B. Carlin, M. Spratt, P. Royston, M.G. Kenward, and et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls BMJ 338 2009
51. K. Dwan, D.G. Altman, J.A. Arnaiz, J. Bloom, A.W. Chan, E. Cronin, and et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias PLoS One 3 8 2008 e3081
52. K. Dwan, C. Gamble, P.R. Williamson, and J.J. Kirkham Reporting bias group: systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review PLoS One 8 7 2013 e66844
53. A.W. Chan, A. Hrobjartsson, M.T. Haahr, P.C. Gotzsche, and D.G. Altman Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles JAMA 291 20 2004 2457 2465
54. A. Liberati, D.G. Altman, J. Tetzlaff, C. Mulrow, P.C. Gotzsche, J.P. Ioannidis, and et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration BMJ 339 2009 b2700
55. M. Skoog, J.M. Saarimäki, C. Gluud, M. Sheinin, K. Erlendsson, S. Aamdal, and et al. Transaprency and registration in clinical research in the Nordic countries 2015 Nordic Trial Alliance, NordForsk 1 108
56. A. Lundh, S. Sismondo, J. Lexchin, O.A. Busuioc, and L. Bero Industry sponsorship and research outcome Coch Database Syst Rev 12 2012 MR000033
57. P.R. Williamson, D.G. Altman, J.M. Blazeby, M. Clarke, D. Devane, E. Gargon, and et al. Developing core outcome sets for clinical trials: issues to consider Trials 13 2012 132
58. P. Williamson, D. Altman, J. Blazeby, M. Clarke, and E. Gargon Driving up the quality and relevance of research through the use of agreed core outcomes J Health Serv Res Policy 17 1 2012 1 2
59. P. Serrano-Aguilar, M.M. Trujillo-Martin, J.M. Ramos-Goni, V. Mahtani-Chugani, L. Perestelo-Perez, and M. Posada-de la Paz Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias Soc Sci Med 69 6 2009 920 925
60. P.J. Mease, L.M. Arnold, L.J. Crofford, D.A. Williams, I.J. Russell, L. Humphrey, and et al. Identifying the clinical domains of fibromyalgia: contributions from clinician and patient Delphi exercises Arthritis Rheum 59 7 2008 952 960
61. N.F. Griffin, C. Melanie, W. John, E. Joanne, and Carl Composite outcomes in randomized trials - greater precision but with greater uncertainty? JAMA 289 2003 2554 2559
62. G. Cordoba, L. Schwartz, S. Woloshin, H. Bae, and P.C. Gotzsche Definition, reporting, and interpretation of composite outcomes in clinical trials: systematic review BMJ 341 2010 c3920
63. A. Phillips, and V. Haudiquet ICH E9 guideline 'statistical principles for clinical trials': a case study Stat Med 22 1 2003 1 11 [discussion 13-17]
64. S. Garattini, and V. Bertele Non-inferiority trials are unethical because they disregard patients' interests Lancet 370 9602 2007 1875 1877
65. F. Song, S. Parekh, L. Hooper, Y.K. Loke, J. Ryder, A.J. Sutton, and et al. Dissemination and publication of research findings: an updated review of related biases Health Technol Assess 14 8 2010 iii [ix-xi, 1-193]
66. +AllTrials All Trials Registered, All Results Reported [avaiable at] http://www.alltrialsnet/
67. A. Stevens, L. Shamseer, E. Weinstein, F. Yazdi, L. Turner, J. Thielman, and et al. Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review BMJ 348 2014 g3804
68. J.P. Ioannidis Some main problems eroding the credibility and relevance of randomized trials Bull NYU Hosp Jt Dis 66 2 2008 135 139
69. M. Clarke, S. Hopewell, and I. Chalmers Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting Lancet 376 9734 2010 20 21
70. M. Clarke, and R. Horton Bringing it all together: Lancet-Cochrane collaborate on systematic reviews Lancet 357 9270 2001 1728
71. C. Young, and R. Horton Putting clinical trials into context Lancet 366 9480 2005 107 108
72. Directive 2001/83/EC of the European Parliament and of the Council of on the Community code relating to medicinal products for human use (Consolidated version: 16/11/2012) 6 November 2001
73. S. Garattini, and V. Bertele How can we regulate medicines better? BMJ 335 7624 2007 803 805
74. S. Garattini, and V. Bertele The scientific community should lobby to be able to apply for drug licences BMJ 344 2012 e3553
75. D.W. Light, and A.F. Maturo Good Pharma 2015 Palgrave Macmillan
76. J.S. Wislar, A. Flanagin, P.B. Fontanarosa, and C.D. Deangelis Honorary and ghost authorship in high impact biomedical journals: a cross sectional survey BMJ 343 2011 d6128
77. A.-W. Chan, J.M. Tetzlaff, D.G. Altman, A. Laupacis, P.C. Gøtzsche, K. Krleža-Jerić, A. Hróbjartsson, H. Mann, and K. Dickersin Berlin JA, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials Ann Intern Med 158 2013 200 207
78. K.F. Schulz, D.G. Altman, and D. Moher CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials Ann Int Med 152 11 2010 726 732
79. S.C. Chow, H. Wang, and J. Shao Sample size calculations in clinical research 2nd ed. Chapman & Hall/CRC Biostatistics Series 2007 Chapman and Hall/CRC
80. D.L. DeMets, and K.K. Lan Interim analysis: the alpha spending function approach Stat Med 13 13-14 1994 1341 1356
81. D.V. Lindley A statistical paradox Biometrika 44 1/2 1957 187 192
82. C. Jennison, and B.W. Turnbull Repeated confidence intervals for group sequential clinical trials Control Clin Trials 5 1 1984 33 45
83. J. Brok, K. Thorlund, C. Gluud, and J. Wetterslev Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analysis J Clin Epidemiol 61 2008 763 769
84. K. Thorlund, J. Engstrøm, J. Wetterslev, J. Brok, G. Imberger, and C. Gluud User manual for trial sequential analysis (TSA) 2011 Copenhagen Trial Unit, Centre for Clinical Intervention Research Copenhagen, Denmark 1 115 [Available from http://www.ctu.dk/tsa ]
85. J. Wetterslev, K. Thorlund, J. Brok, and C. Gluud Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis J Clin Epidemiol 61 1 2008 64 75
86. R.M. Turner, S.M. Bird, and J.P. Higgins The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews PLoS One 8 3 2013 e59202
87. I. Roberts, K. Ker, P. Edwards, D. Beecher, D. Manno, and E. Sydenham The knowledge system underpinning healthcare is not fit for purpose and must change BMJ 350 2015 h2463
88. J. Wetterslev, K. Thorlund, J. Brok, and C. Gluud Estimating required information size by quantifying diversity in random-effects model meta-analyses BMC Med Res Methodol 9 2009 86
89. C. Gluud, J.C. Jakobsen, G. Imberger, T. Lange, and J. Wetterslev Re: the knowledge system underpinning healthcare is not fit for purpose and must change - reponses to the opposing viewpoints of Roberts and colleagues and Tovey and colleagues BMJ 2015
90. J.P.T. Higgins, A. Whitehead, and M. Simmonds Sequential methods for random-effects meta-analysis Stat Med 30 9 2011 903 921
91. G.K.K. Lan, and D.L. DeMets Discrete sequential boundaries for clinical trials Biometrika 70 3 1983 659 663
92. J. Brok, K. Thorlund, C. Gluud, and J. Wetterslev Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analysis J Clin Epidemiol 61 2008 763 769
93. D. Moher, D.J. Cook, S. Eastwood, I. Olkin, D. Rennie, and D.F. Stroup Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses Lancet 354 9193 1999 1896 1900
94. M. Egger, G. Davey Smith, M. Schneider, and C. Minder Bias in meta-analysis detected by a simple, graphical test BMJ 315 7109 1997 629 634
95. C.B. Begg, and M. Mazumdar Operating characteristics of a rank correlation test for publication bias Biometrics 50 4 1994 1088 1101
96. S. Duval, and R. Tweedie Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis Biometrics 56 2 2000 455 463
97. A.J. Sutton, S.J. Duval, R.L. Tweedie, K.R. Abrams, and D.R. Jones Empirical assessment of effect of publication bias on meta-analyses BMJ 320 7249 2000 1574 1577
98. D. Moher, J. Tetzlaff, A.C. Tricco, M. Sampson, and D.G. Altman Epidemiology and reporting characteristics of systematic reviews PLoS Med 4 3 2007 e78
99. D. Pieper, S.L. Antoine, T. Mathes, E.A. Neugebauer, and M. Eikermann Systematic review finds overlapping reviews were not mentioned in every other overview J Clin Epidemiol 67 4 2014 368 375
100. L.P. Moja, E. Telaro, R. D'Amico, I. Moschetti, L. Coe, and A. Liberati Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study BMJ 330 7499 2005 1053
101. G. Bjelakovic, D. Nikolova, L.L. Gluud, R.G. Simonetti, and C. Gluud Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases Cochrane Database Syst Rev 3 2012 CD007176
102. M.A. Puhan, H.J. Schunemann, M.H. Murad, T. Li, R. Brignardello-Petersen, J.A. Singh, and et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis BMJ 349 2014 g5630
103. C. Gluud, and T.I. Sorensen New developments in the conduct and management of multi-center trials: an international review of clinical trial units Fundam Clin Pharmacol 9 3 1995 284 289
104. P. McCulloch, D.G. Altman, W.B. Campbell, D.R. Flum, P. Glasziou, J.C. Marshall, and et al. No surgical innovation without evaluation: the IDEAL recommendations Lancet 374 9695 2009 1105 1112