Empirical evidence of study design biases in randomized trials: Systematic review of meta-epidemiological studies

PLoS ONE

Volumn 11, Issue 7, 2016, Pages

  Page, Matthew J ^a,b   Higgins, Julian P T ^a   Clayton, Gemma ^a   Sterne, Jonathan A C ^a   Hróbjartsson, Asbjørn ^c   Savović, Jelena ^a,d  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b MONASH UNIVERSITY   (Australia)

^c UNIVERSITY OF SOUTHERN DENMARK   (Denmark)

^d UNIVERSITY HOSPITALS BRISTOL NHS FOUNDATION TRUST   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CONTROLLED CLINICAL TRIAL; EMBASE; HUMAN; MEDLINE; META ANALYSIS; MORTALITY; ODDS RATIO; RANDOMIZED CONTROLLED TRIAL; STUDY DESIGN; SYSTEMATIC REVIEW; EPIDEMIOLOGY; PROCEDURES; RANDOMIZED CONTROLLED TRIAL (TOPIC); STATISTICAL BIAS;

BIAS (EPIDEMIOLOGY); EPIDEMIOLOGIC STUDIES; HUMANS; RANDOMIZED CONTROLLED TRIALS AS TOPIC;

EID: 84978859510     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0159267     Document Type: Review

References (58)

1. Guyatt G, Cairns J, Churchill D, Cook D, Haynes B, Hirsh J, et al. Evidence-based medicine: A new approach to teaching the practice of medicine. JAMA. 1992; 268(17):2420-5. PMID: 1404801
2. NHMRC. How to use the evidence: assessment and application of scientific evidence. Canberra, Australia: National Health and Medical Research Council; 2000.
3. OCEBM Levels of Evidence Working Group. "The Oxford 2011 Levels of Evidence". Oxford Centre for Evidence-Based Medicine. Available: http://www.cebm.net/index.aspx?o=5653.
4. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available: www.cochrane-handbook.org.
5. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995; 273(5):408-12. PMID: 7823387
6. Berkman ND, Santaguida PL, Viswanathan M, Morton SC. AHRQ Methods for Effective Health Care. The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014.
7. Mills EJ, Ayers D, Chou R, Thorlund K. Are current standards of reporting quality for clinical trials sufficient in addressing important sources of bias? Contemp Clin Trials. 2015; 45(Pt A):2-7. doi: 10.1016/j.cct.2015.07.019 PMID: 26232560
8. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009; 6(7):e1000097. doi: 10.1371/journal.pmed.1000097 PMID: 19621072
9. Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002; 21(11):1513-24. PMID: 12111917
10. Savovic J, Jones H, Altman D, Harris R, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012; 16(35):1-82. PMID: 22989478
11. Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ. 2012; 344:e1119. doi: 10.1136/bmj.e1119 PMID: 22371859
12. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials. 1996; 17(1):1-12. PMID: 8721797
13. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011; 343:d5928. doi: 10.1136/bmj.d5928 PMID: 22008217
14. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2012;12:MR000033.
15. Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013; 346:f2304. doi: 10.1136/bmj.f2304 PMID: 23616031
16. Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P. Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study. BMJ. 2012; 344:e813. doi: 10.1136/bmj.e813 PMID: 22334559
17. Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study. Ann Intern Med. 2011; 155(1):39-51. doi: 10.7326/0003-4819-155-1-201107050-00006 PMID: 21727292
18. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010; 303(12):1180-7. doi: 10.1001/jama.2010.310 PMID: 20332404
19. Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JP. Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment. BMJ. 2013; 346:f707. doi: 10.1136/bmj.f707 PMID: 23403829
20. Jacobs WC, Kruyt MC, Moojen WA, Verbout AJ, Oner FC. No evidence for intervention-dependent influence of methodological features on treatment effect. J Clin Epidemiol. 2013; 66(12):1347-55.e3. doi: 10.1016/j.jclinepi.2013.06.007 PMID: 24018348
21. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012; 157(6):429-38. PMID: 22945832
22. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7(3):177-88. PMID: 3802833
23. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003; 327(7414):557-60. PMID: 12958120
24. Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med. 2000; 19:3127-31. PMID: 11113947
25. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008; 336(7644):601-5. doi: 10.1136/bmj.39465.451748.AD PMID: 18316340
26. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003; 7(1):1-76. PMID: 12583822
27. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001; 135(11):982-9. PMID: 11730399
28. Als-Nielsen B, Chen W, Gluud LL, Siersma V, Hilden J, Gluud C. Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials [abstract]. 12th Cochrane Colloquium: Bridging the Gaps; 2004 Oct 2-6; Ottawa, Ontario, Canada [Internet]. 2004:[102-3 pp.]. Available: http://onlinelibrary.wiley.com/o/cochrane/clcmr/articles/CMR-6643/frame.html.
29. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002; 287(22):2973-82. PMID: 12052127
30. Contopoulos-Ioannidis DG, Gilbody SM, Trikalinos TA, Churchill R, Wahlbeck K, Ioannidis JP. Comparison of large versus smaller randomized trials for mental health-related interventions. Am J Psychiatry. 2005; 162(3):578-84. PMID: 15741476
31. Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007; 36(4):847-57. PMID: 17517809
32. Abraha I, Cherubini A, Cozzolino F, De Florio R, Luchetta ML, Rimland JM, et al. Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study. BMJ. 2015; 350:h2445. doi: 10.1136/bmj.h2445 PMID: 26016488
33. Armijo-Olivo S, Saltaji H, da Costa BR, Fuentes J, Ha C, Cummings GG. What is the influence of randomisation sequence generation and allocation concealment on treatment effects of physical therapy trials? A meta-epidemiological study. BMJ Open. 2015; 5(9):e008562. doi: 10.1136/bmjopen-2015-008562 PMID: 26338841
34. Bialy L, Vandermeer B, Lacaze-Masmonteil T, Dryden DM, Hartling L. A meta-epidemiological study to examine the association between bias and treatment effects in neonatal trials. Evid Based Child Health. 2014; 9(4):1052-9. doi: 10.1002/ebch.1985 PMID: 25504975
35. Chaimani A, Vasiliadis HS, Pandis N, Schmid CH, Welton NJ, Salanti G. Effects of study precision and risk of bias in networks of interventions: a network meta-epidemiological study. Int J Epidemiol. 2013; 42(4):1120-31. doi: 10.1093/ije/dyt074 PMID: 23811232
36. Gluud LL, Thorlund K, Gluud C, Woods L, Harris R, Sterne JA. Correction: reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2008; 149(3):219. PMID: 18942172
37. Hartling L, Hamm MP, Fernandes RM, Dryden DM, Vandermeer B. Quantifying bias in randomized controlled trials in child health: a meta-epidemiological study. PLoS One. 2014; 9(2):e88008. doi: 10.1371/journal.pone.0088008 PMID: 24505351
38. Herbison P, Hay-Smith J, Gillespie WJ. Different methods of allocation to groups in randomized trials are associated with different levels of bias. A meta-epidemiological study. J Clin Epidemiol. 2011; 64(10):1070-5. doi: 10.1016/j.jclinepi.2010.12.018 PMID: 21474279
39. Hrobjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ. 2013; 185(4):E201-11. doi: 10.1503/cmaj.120744 PMID: 23359047
40. Hrobjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Rasmussen JV, Hilden J, et al. Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. Int J Epidemiol. 2014; 43(3):937-48. doi: 10.1093/ije/dyt270 PMID: 24448109
41. Hrobjartsson A, Emanuelsson F, Skou Thomsen AS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. Int J Epidemiol. 2014; 43(4):1272-83. doi: 10.1093/ije/dyu115 PMID: 24881045
42. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998; 352(9128):609-13. PMID: 9746022
43. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess. 1999; 3(12):i-iv, 1-98. PMID: 10374081
44. Nuesch E, Reichenbach S, Trelle S, Rutjes AW, Liewald K, Sterchi R, et al. The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum. 2009; 61(12):1633-41. doi: 10.1002/art.24894 PMID: 19950329
45. Nuesch E, Trelle S, Reichenbach S, Rutjes AW, Burgi E, Scherer M, et al. The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ. 2009; 339: b3244. doi: 10.1136/bmj.b3244 PMID: 19736281
46. Papageorgiou SN, Xavier GM, Cobourne MT. Basic study design influences the results of orthodontic clinical investigations. J Clin Epidemiol. 2015.
47. Siersma V, Als-Nielsen B, Chen W, Hilden J, Gluud LL, Gluud C. Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med. 2007; 26(14):2745-58. PMID: 17117373
48. Unverzagt S, Prondzinsky R, Peinemann F. Single-center trials tend to provide larger treatment effects than multicenter trials: a systematic review. J Clin Epidemiol. 2013; 66(11):1271-80. doi: 10.1016/j.jclinepi.2013.05.016 PMID: 23972520
49. Armijo-Olivo S, Fuentes J, Rogers T, Hartling L, Saltaji H, Cummings GG. How should we evaluate the risk of bias of physical therapy trials?: a psychometric and meta-epidemiological approach towards developing guidelines for the design, conduct, and reporting of RCTs in Physical Therapy (PT) area: a study protocol. Systematic reviews. 2013; 2:88. doi: 10.1186/2046-4053-2-88 PMID: 24070072
50. Dossing A, Tarp S, Furst DE, Gluud C, Beyene J, Hansen BB, et al. Interpreting trial results following use of different intention-to-treat approaches for preventing attrition bias: A meta-epidemiological study protocol. BMJ Open. 2014; 4:e005297. doi: 10.1136/bmjopen-2014-005297 PMID: 25260368
51. Feys F, Bekkering GE, Singh K, Devroey D. Do randomized clinical trials with inadequate blinding report enhanced placebo effects for intervention groups and nocebo effects for placebo groups? A protocol for a meta-epidemiological study of PDE-5 inhibitors. Systematic reviews. 2012; 1:54. doi: 10.1186/2046-4053-1-54 PMID: 23151403
52. Hansen JB, Juhl CB, Boutron I, Tugwell P, Ghogomu EAT, Pardo JP, et al. Assessing bias in osteoarthritis trials included in Cochrane reviews: Protocol for a meta-epidemiological study. BMJ Open. 2014; 4:e005491. doi: 10.1136/bmjopen-2014-005491 PMID: 25280805
53. Saltaji H, Armijo-Olivo S, Cummings GG, Amin M, Flores-Mir C. Methodological characteristics and treatment effect sizes in oral health randomised controlled trials: Is there a relationship? Protocol for a meta-epidemiological study. BMJ Open. 2014; 4:e004527. doi: 10.1136/bmjopen-2013-004527 PMID: 24568962
54. Hróbjartsson A, Boutron I, Turner L, Altman DG, Moher D. Assessing risk of bias in randomised clinical trials included in Cochrane Reviews: the why is easy, the how is a challenge [editorial]. Cochrane Database of Systematic Reviews. 2013; 4:ED000058. doi: 10.1002/14651858.ED000058 PMID: 23728703
55. Giraudeau B, Higgins JP, Tavernier E, Trinquart L. Sample size calculation for meta-epidemiological studies. Stat Med. 2015; 35(2):239-50. doi: 10.1002/sim.6627 PMID: 26286683
56. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004; 57(12):1232-6. PMID: 15617948
57. Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ. 2013; 346:f1798. doi: 10.1136/bmj.f1798 PMID: 23610376
58. Mhaskar R, Djulbegovic B, Magazin A, Soares HP, Kumar A. Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols. J Clin Epidemiol. 2012; 65(6):602-9. doi: 10.1016/j.jclinepi.2011.10.016 PMID: 22424985