Recommendations for assessing the risk of bias in systematic reviews of health-care interventions

Journal of Clinical Epidemiology

Volumn 97, Issue , 2018, Pages 26-34

  Viswanathan, Meera ^a   Patnode, Carrie D ^b   Berkman, Nancy D ^a   Bass, Eric B ^c   Chang, Stephanie ^d   Hartling, Lisa ^e   Murad, M Hassan ^f   Treadwell, Jonathan R ^g   Kane, Robert L ^h  

^a UNIVERSITY OF NORTH CAROLINA   (United States)

^b CENTER FOR HEALTH RESEARCH   (United States)

^c JOHNS HOPKINS UNIVERSITY   (United States)

^d AGENCY FOR HEALTHCARE RESEARCH AND QUALITY   (United States)

^e UNIVERSITY OF ALBERTA   (Canada)

^f MAYO CLINIC   (United States)

^g ECRI INSTITUTE   (United States)

^h UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Critical appraisal; Evidence based practice; Health care interventions; Meta analyses; Risk of bias guidance; Systematic reviews

Indexed keywords

ACCURACY; ARTICLE; CLINICAL EVALUATION; CLINICAL OUTCOME; EXTERNAL VALIDITY; HEALTH CARE; HUMAN; INTERNAL VALIDITY; METHODOLOGY; OUTCOME ASSESSMENT; PRIORITY JOURNAL; PUBLISHING; REPRODUCIBILITY; RISK; RISK ASSESSMENT; STATISTICAL BIAS; STUDY DESIGN; SYSTEMATIC REVIEW (TOPIC); EVIDENCE BASED MEDICINE; PRACTICE GUIDELINE;

BIAS; EVIDENCE-BASED MEDICINE; GUIDELINES AS TOPIC; HUMANS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; RISK ASSESSMENT; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85042288596     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2017.12.004     Document Type: Article

References (62)

1. Higgins, J.P.T., Green, S., Cochrane handbook for systematic reviews of interventions. [updated March 2011] Higgins, J.P.T., Green, S., (eds.) The Cochrane Collaboration, 2011 Available at http://handbook.cochrane.org Version 5.1.0.
2. Institute of Medicine. Finding what works in health care: standards for systematic reviews. Available at http://www.nap.edu/openbook.php?record_id=13059&page=R1.
3. Armijo Olivo, S., Ospina, M., da Costa, B.R., Egger, M., Saltaji, H., Fuentes, J., et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One, 9, 2014, e96920.
4. Savovic, J., Jones, H.E., Altman, D.G., Harris, R.J., Juni, P., Pildal, J., et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 157 (2012), 429–438.
5. Hempel, S., Suttorp, M.J., Miles, J.N.V., Wang, Z., Maglione, M., Morton, S.C., et al. Empirical evidence of associations between trial quality and effect sizes. Methods research report. (Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-2007-10062-I). AHRQ Publication No. 11-EHC045-EF, 2011, Agency for Healthcare Research and Quality, Rockville, MD Available at http://effectivehealthcare.ahrq.gov.
6. Berkman, N.D., Santaguida, P.L., Viswanathan, M., Morton, S.C., The empirical evidence of bias in trials measuring treatment differences. Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, Contract No. 290-2007-10056-I. AHRQ Publication No. 14-EHC050-EF, 2014, RTI-UNC Evidence-based Practice Center, Research Triangle Park, NC, Rockville, MD.
7. Hempel, S., Miles, J., Suttorp, M., Wang, Z., Johnsen, B., Morton, S., et al. Detection of associations between trial quality and effect sizes. 2012, Agency for Healthcare Research and Quality, Rockville, MD.
8. Hartling, L., Hamm, M.P., Milne, A., Vandermeer, B., Santaguida, P.L., Ansari, M., et al. Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 66 (2013), 973–981.
9. Hartling, L., Bond, K., Vandermeer, B., Seida, J., Dryden, D., Rowe, B., Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for maintenance therapy of persistent asthma, 2009, Cochrane Collaboration, Singapore; London.
10. Sterne, J.A.C., Hernán, M.A., Reeves, B.C., Savović J., Berkman, N.D., Viswanathan, M., et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 355, 2016, i4919.
11. Viswanathan, M., Patnode, C.D., Berkman, N.D., Chang, S., Hartling, L., McPheeters, M., et al. Assessing the risk of bias of individual studies in systematic reviews of health care interventions. 2017, Agency for Healthcare Research and Quality, Rockville, MD.
12. Santaguida, P.L., Riley, C.R., Matchar, D.B., Chapter 5: assessing risk of bias as a domain of quality in medical test studies. AHRQ Publication No. 12-EHC077-EF. [Chapter 5] of Methods Guide for Medical Test Reviews (AHRQ Publication No. 12-EHC017), 2012, Agency for Healthcare Research and Quality, Rockville, MD Available at www.effectivehealthcare.ahrq.gov/reports/final.cfm Also published in a special supplement to the Journal of General Internal Medicine, July 2012. Rockville, MD: 2012.
13. Hayden, J.A., van der Windt, D.A., Cartwright, J.L., Côté P., Bombardier, C., Assessing bias in studies of prognostic factors. Ann Intern Med 158 (2013), 280–286.
14. Collins, G.S., Reitsma, J.B., Altman, D.G., Moons, K.G., Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD). Ann Intern Med 162 (2015), 735–736.
15. Moons, K.G., de Groot, J.A., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D.G., et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med, 11(10), 2014, e1001744.
16. Lockwood, C., Munn, Z., Porritt, K., Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc 13:3 (2015), 179–187.
17. Deeks, J.J., Dinnes, J., D'Amico, R., Sowden, A.J., Sakarovitch, C., Song, F., et al. Evaluating non-randomised intervention studies. Health Technol Assess 7 (2003), 1–173.
18. Turner, L., Boutron, I., Hróbjartsson, A., Altman, D.G., Moher, D., The evolution of assessing bias in Cochrane systematic reviews of interventions: celebrating methodological contributions of the Cochrane Collaboration. Syst Rev, 2(1), 2013, 79.
19. Lohr, K.N., Carey, T.S., Assessing “best evidence”: issues in grading the quality of studies for systematic reviews. Jt Comm J Qual Improv 25:9 (1999), 470–479.
20. Balshem, H., Helfand, M., Schunemann, H.J., Oxman, A.D., Kunz, R., Brozek, J., et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 64 (2011), 401–406.
21. U.S. Preventive Services Task Force. U.S. Preventive Services Task Force procedure manual. AHRQ Publication No. 08-05118-EF, 2008 Available at http://www.uspreventiveservicestaskforce.org/uspstf08/methods/procmanual.htm.
22. Crowe, M., Sheppard, L., A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J Clin Epidemiol 64 (2011), 79–89.
23. Norris, S.L., Atkins, D., Bruening, W., Fox, S., Johnson, E., Kane, R., et al. Observational studies in systemic reviews of comparative effectiveness: AHRQ and the effective health care program. J Clin Epidemiol 64 (2011), 1178–1186.
24. Atkins, D., Best, D., Briss, P.A., Eccles, M., Falck-Ytter, Y., Flottorp, S., et al. Grading quality of evidence and strength of recommendations. BMJ, 328, 2004, 1490.
25. Berkman, N.D., Lohr, K.N., Ansari, M.T., Balk, E.M., Kane, R., McDonagh, M., et al. Grading the strength of a body of evidence when assessing health care interventions: an EPC update. J Clin Epidemiol 68 (2015), 1312–1324.
26. Guyatt, G.H., Oxman, A.D., Kunz, R., Atkins, D., Brozek, J., Vist, G., et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol 64 (2011), 395–400.
27. Guyatt, G.H., Oxman, A.D., Kunz, R., Brozek, J., Alonso-Coello, P., Rind, D., et al. GRADE guidelines 6. Rating the quality of evidence-imprecision. J Clin Epidemiol 64 (2011), 1283–1293.
28. Guyatt, G.H., Oxman, A.D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., et al. GRADE guidelines: 8. Rating the quality of evidence-indirectness. J Clin Epidemiol 64 (2011), 1303–1310.
29. Guyatt, G.H., Oxman, A.D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., et al. GRADE guidelines: 7. Rating the quality of evidence-inconsistency. J Clin Epidemiol 64 (2011), 1294–1302.
30. Guyatt, G.H., Oxman, A.D., Montori, V., Vist, G., Kunz, R., Brozek, J., et al. GRADE guidelines: 5. Rating the quality of evidence-publication bias. J Clin Epidemiol 64 (2011), 1277–1282.
31. Guyatt, G.H., Oxman, A.D., Schunemann, H.J., Tugwell, P., Knottnerus, A., GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol 64 (2011), 380–382.
32. Guyatt, G.H., Oxman, A.D., Sultan, S., Glasziou, P., Akl, E.A., Alonso-Coello, P., et al. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol 64 (2011), 1311–1316.
33. Guyatt, G.H., Oxman, A.D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., et al. GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias). J Clin Epidemiol 64 (2011), 407–415.
34. Shea, B.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J., et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62 (2009), 1013–1020.
35. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 62 (2009), 1006–1012.
36. Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P., et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ, 339, 2009, b2700.
37. Marshall, I., Kuiper, J., Wallace, B.C., Automating risk of bias assessment for clinical trials. Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, 2014, ACM, Newport Beach, California, 88–95.
38. Marshall, I.J., Kuiper, J., Wallace, B.C., RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc 23 (2016), 193–201.
39. Olivo, S.A., Macedo, L.G., Gadotti, I.C., Fuentes, J., Stanton, T., Magee, D.J., Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 88 (2008), 156–175.
40. Sanderson, S., Tatt, I.D., Higgins, J.P., Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 36 (2007), 666–676.
41. Whiting, P., Rutjes, A.W., Dinnes, J., Reitsma, J.B., Bossuyt, P.M., Kleijnen, J., A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol 58 (2005), 1–12.
42. West, S.L., King, V., Carey, T.S., Lohr, K.N., McKoy, N., Sutton, S.F., et al. Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47. AHRQ Pub. No. 02–E016, 2002, Agency for Healthcare Research and Quality, Rockville, MD.
43. Zeng, X., Zhang, Y., Kwong, J.S., Zhang, C., Li, S., Sun, F., et al. The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med 8:1 (2015), 2–10.
44. Sun, X., Briel, M., Busse, J.W., You, J.J., Akl, E.A., Mejza, F., et al. Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ, 344, 2012, e1553.
45. Sun, X., Ioannidis, J.P., Agoritsas, T., Alba, A.C., Guyatt, G., How to use a subgroup analysis: users’ guide to the medical literature. JAMA 311 (2014), 405–411.
46. Whitlock, E.P., Eder, M., Thompson, J.H., Jonas, D.E., Evans, C.V., Guirguis-Blake, J., et al. An approach to addressing subpopulation considerations in systematic reviews. Syst Rev, 6, 2017, 41.
47. Sun, X., Briel, M., Walter, S.D., Guyatt, G.H., Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ, 340, 2010, c117.
48. Robinson, K.A., Chou, R., Berkman, N.D., Newberry, S.J., Fu, R., Hartling, L., et al. Twelve recommendations for integrating existing systematic reviews into new reviews: EPC guidance. J Clin Epidemiol 70 (2016), 38–44.
49. Robinson, K., Chou, R., Berkman, N., Newberry, S.J., Fu, R., Hartling, L., et al. Integrating bodies of evidence: existing systematic reviews and primary studies. 2008, Agency for Healthcare Research and Quality (US), Rockville, MD.
50. Murad, M.H., Montori, V.M., Ioannidis, J.P., Jaeschke, R., Devereaux, P.J., Prasad, K., et al. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA 312 (2014), 171–179.
51. Alkin, M.C., Evaluation roots: tracing theorists’ views and influences. 2004, Sage Publications, Thousand Oaks, CA.
52. Kung, J., Chiappelli, F., Cajulis, O.O., Avezova, R., Kossan, G., Chew, L., et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J 4:1 (2010), 84–91.
53. Pieper, D., Buechter, R.B., Li, L., Prediger, B., Eikermann, M., Systematic review found AMSTAR, but not R (evised)-AMSTAR, to have good measurement properties. J Clin Epidemiol 68 (2015), 574–583.
54. Higgins, J., Lane, P.W., Anagnostelis, B., Anzures-Cabrera, J., Baker, N.F., Cappelleri, J.C., et al. A tool to assess the quality of a meta-analysis. Res Synth Methods 4:4 (2013), 351–366.
55. Donegan, S., Williamson, P., Gamble, C., Tudur-Smith, C., Indirect comparisons: a review of reporting and methodological quality. PLoS One, 5, 2010, e11054.
56. Whiting, P., Savović J., Higgins, J.P., Caldwell, D.M., Reeves, B.C., Shea, B., et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69 (2016), 225–234.
57. Faggion, C.M. Jr., Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol, 15, 2015, 63.
58. Shea, B.J., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, 2017, j4008.
59. Balk, E.M., Bonis, P.A., Moskowitz, H., Schmid, C.H., Ioannidis, J.P., Wang, C., et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 287 (2002), 2973–2982.
60. Fu, R., Gartlehner, G., Grant, M., Shamliyan, T., Sedrakyan, A., Wilt, T.J., et al. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol 64 (2011), 1187–1197.
61. Higgins, J.P., Thompson, S.G., Deeks, J.J., Altman, D.G., Measuring inconsistency in meta-analyses. BMJ 327 (2003), 557–560.
62. Herbison, P., Hay-Smith, J., Gillespie, W.J., Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 59 (2006), 1249–1256.