Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: A comparison study

BMC Research Notes

Volumn 8, Issue 1, 2015, Pages

  O'Connor, Seán R ^a,c   Tully, Mark A ^a   Ryan, Brigid ^b   Bradley, Judy M ^c   Baxter, George D ^b   McDonough, Suzanne M ^a,c  

^a QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

^b UNIVERSITY OF OTAGO   (New Zealand)

^c UNIVERSITY OF ULSTER   (United Kingdom)

Author keywords

Quality assessment; Risk of bias; Systematic review methods

Indexed keywords

SAISSETIA OLEAE;

CHI SQUARE DISTRIBUTION; CHRONIC PAIN; COMPARATIVE STUDY; HUMAN; KINESIOTHERAPY; METHODOLOGY; MUSCULOSKELETAL PAIN; ODDS RATIO; PATHOPHYSIOLOGY; RANDOMIZED CONTROLLED TRIAL (TOPIC); REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICAL BIAS; STATISTICAL MODEL; STATISTICS AND NUMERICAL DATA; TREATMENT OUTCOME;

BIAS (EPIDEMIOLOGY); CHI-SQUARE DISTRIBUTION; CHRONIC PAIN; DATA INTERPRETATION, STATISTICAL; EXERCISE THERAPY; HUMANS; MODELS, STATISTICAL; MUSCULOSKELETAL PAIN; ODDS RATIO; RANDOMIZED CONTROLLED TRIALS AS TOPIC; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; TREATMENT OUTCOME;

EID: 84939266587     PISSN: None     EISSN: 17560500     Source Type: Journal    
DOI: 10.1186/s13104-015-1181-1     Document Type: Review

References (34)

1. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP et al (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 151(4):W65-W94
2. Kirkham JJ, Gargon E, Clarke M, Williamson PR (2013) Can a core outcome set improve the quality of systematic reviews? - a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials 14:21
3. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R et al (2010) The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340:c365
4. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J et al (2009) Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 339:b4012
5. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG et al (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336(7644):601-605
6. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P et al (2011) GRADE guidelines: 4. Rating the quality of evidence - study limitations (risk of bias). J Clin Epidemiol 64(4):407-415
7. Colle F, Rannou F, Revel M, Fermanian J, Poiraudeau S (2002) Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil 83(12):1745-1752
8. Gagnier JJ, Kellam PJ (2013) Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am 95(11):e771-e777
9. Jüni P, Tallon D, Egger M (2000) 'Garbage in-garbage out'? Assessment of the quality of controlled trials in meta-analyses published in leading journals. In: Proceedings of the 3rd symposium on systematic reviews: beyond the basics. St Catherine's College, Oxford. Centre for Statistics in Medicine, Oxford, p 19
10. Herbison P, Hay-Smith J, Gillespie WJ (2006) Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 59(12):1249-1256
11. Higgins JPT, Green S (eds) (2009) Cochrane handbook for systematic reviews of interventions version 5.0.2 [updated September 2009]. The Cochrane Collaboration
12. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM et al (2001) Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med 20(3 Suppl):21-35
13. Sawaya GF, Guirguis-Blake J, LeFevre M, Harris R, Petitti D (2007) Force USPST: update on the methods of the US Preventive Services Task Force: estimating certainty and magnitude of net benefit. Ann Intern Med 147(12):871-875
14. Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, Hill S et al (2004) Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res 4(1):38
15. O'Connor SR, Tully MA, Ryan B, Bleakley CM, Baxter GD, Bradley JM et al (2014) Walking exercise for chronic musculoskeletal pain: systematic review and meta-analysis. Arch Phys Med Rehabil. S0003-9993(14)01314-8 [pii] [Epub ahead of print]
16. Downs SH, Black N (1998) The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 52(6):377-384
17. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F et al (2003) Evaluating non-randomised intervention studies. Health Technol Assess 7(27):iii-x, 1-173
18. Richmond SA, Fukuchi RK, Ezzat A, Schneider K, Schneider G, Emery CA (2013) Are joint injury, sport activity, physical activity, obesity, or occupational activities predictors for osteoarthritis? A systematic review. J Orthop Sports Phys Ther 43(8):B515-B519
19. Morton S, Barton CJ, Rice S, Morrissey D (2014) Risk factors and successful interventions for cricket-related low back pain: a systematic review. Br J Sports Med 48(8):685-691
20. Simic M, Hinman RS, Wrigley TV, Bennell KL, Hunt MA (2011) Gait modification strategies for altering medial knee joint load: a systematic review. Arthritis Care Res (Hoboken) 63(3):405-426
21. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151:264-269
22. McDowell I (2006) Chapter 2: the theoretical and technical foundations of health measurements. In: Measuring health, 3rd edn. Oxford University Press, Oxford, pp 10-54
23. The Cochrane Collaboration (2008) Review Manager (RevMan) [Computer program]. Version 50. The Nordic Cochrane Centre, Copenhagen
24. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273(5):408-412
25. Siersma V, Als-Nielsen B, Chen W, Hilden J, Gluud LL, Gluud C (2007) Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med 26(14):2745-2758
26. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A (2005) Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 330(7499):1053
27. Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H et al (2011) Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev (4):MR000012
28. Savović J, Jones HE, Altman DG, Harris RJ, Jüni P, Pildal J et al (2012) Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 157(6):429-438
29. Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A et al (2013) Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol 66(9):982-993
30. Oremus M, Oremus C, Hall GB, McKinnon MC, Team EaCSR (2012) Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales. BMJ Open 2(4):e001368
31. Aubut JA, Marshall S, Bayley M, Teasell RW (2013) A comparison of the PEDro and Downs and Black quality assessment tools using the acquired brain injury intervention literature. NeuroRehabilitation 32(1):95-102
32. Greenland S (1994) Quality scores are useless and potentially misleading. Am J Epidemiol 140:300-302
33. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C et al (2002) Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 287(22):2973-2982
34. Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG (2012) Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract 18(1):12-18