Tools for assessing risk of reporting biases in studies and syntheses of studies: A systematic review

BMJ Open

Volumn 8, Issue 3, 2018, Pages

  Page, Matthew J ^a,b   McKenzie, Joanne E ^a   Higgins, Julian P T ^b  

^a MONASH UNIVERSITY   (Australia)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

bias (epidemiology); checklist; publication bias; review literature as topic

Indexed keywords

CHECKLIST; DECISION MAKING; EMBASE; HUMAN; LITERATURE; MEDLINE; OUTCOME ASSESSMENT; PSYCINFO; PUBLISHING; REVIEW; RISK ASSESSMENT; SELECTION BIAS; SYNTHESIS; SYSTEMATIC REVIEW; INFORMATION DISSEMINATION; MEDICAL RESEARCH; STANDARDS; STATISTICAL BIAS;

BIAS; BIOMEDICAL RESEARCH; HUMANS; INFORMATION DISSEMINATION; OUTCOME ASSESSMENT (HEALTH CARE); RISK ASSESSMENT;

EID: 85044241657     PISSN: None     EISSN: 20446055     Source Type: Journal    
DOI: 10.1136/bmjopen-2017-019703     Document Type: Review

References (70)

1. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: Addressing inaccessible research. Lancet 2014;383:257-66.
2. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: An updated review of related biases. Health Technol Assess 2010;14:8.
3. Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.
4. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919.
5. Schmucker C, Schell LK, Portalupi S, et al. Extent of nonpublication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One 2014;9:e114023.
6. Jones CW, Keil LG, Holland WC, et al. Comparison of registered and published outcomes in randomized controlled trials: A systematic review. BMC Med 2015;13:282.
7. Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLoS Med 2016;13:e1002028.
8. Koletsi D, Valla K, Fleming PS, et al. Assessment of publication bias required improvement in oral health systematic reviews. J Clin Epidemiol 2016;76:118-24.
9. Hedin RJ, Umberham BA, Detweiler BN, et al. Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-Analyses in Anesthesiology Journals. Anesth Analg 2016;123:1018-25.
10. Ziai H, Zhang R, Chan AW, et al. Search for unpublished data by systematic reviewers: An audit. BMJ Open 2017;7:e017737.
11. Page MJ, Higgins JP. Rethinking the assessment of risk of bias due to selective reporting: A cross-sectional study. Syst Rev 2016;5:108.
12. Moher D, Jadad AR, Nichol G, et al. Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials 1995;16:62-73.
13. Olivo SA, Macedo LG, Gadotti IC, et al. Scales to assess the quality of randomized controlled trials: A systematic review. Phys Ther 2008;88:156-75.
14. Bai A, Shukla VK, Bak G, et al. Quality Assessment Tools Project Report. Ottawa: Canadian Agency for Drugs and Technologies in Health, 2012.
15. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: A systematic review and annotated bibliography. Int J Epidemiol 2007;36:666-76.
16. Whiting P, Rutjes AW, Dinnes J, et al. A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol 2005;58:1-12.
17. Whiting P, Davies P, Savovic J, et al. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews, September 2013. 2013 https://www.researchgate.net/publication/303312018-Evidence-To-inform-The-development-of-ROBIS-A-new-Tool-To-Assess-The-risk-of-bias-in-systematic-reviews (accessed 1 Aug 2017).
18. Mueller KF, Meerpohl JJ, Briel M, et al. Methods for detecting, quantifying, and adjusting for dissemination bias in meta-Analysis are described. J Clin Epidemiol 2016;80:25-33.
19. Jin ZC, Zhou XH, He J. Statistical methods for dealing with publication bias in meta-Analysis. Stat Med 2015;34:343-60.
20. Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-Analyses of randomised controlled trials. BMJ 2011;343:d4002.
21. Higgins JP, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.
22. Sterne JAC, Egger M, Moher D. Chapter 10: Addressing reporting biases. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions Version 510. Chichester, UK: John Wiley & Sons, 2011.
23. Whiting P, Savović J, Higgins JP, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225-34.
24. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
25. Haddaway NR, Collins AM, Coughlin D, et al. The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS One 2015;10:e0138237.
26. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37-46.
27. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.
28. Balshem H, Stevens A, Ansari M, et al. Finding grey literature evidence and assessing for outcome and analysis reporting biases when comparing medical interventions: AHRQ and the Effective Health Care Program. (Prepared by the Oregon Health and Science University and the University of Ottawa Evidence-based Practice Centers under Contract Nos. 290-2007-10057-I and 290-2007-10059-I.) AHRQ Publication No. 13(14)-EHC096-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013.
29. Berkman ND, Lohr KN, Ansari M, et al. Chapter 15 Appendix A: A Tool for Evaluating the Risk of Reporting Bias (in Chapter 15: Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update). Methods Guide for Comparative Effectiveness Reviews (Prepared by the RTI-UNC Evidence-based Practice Center under Contract No. 290-2007-10056-I). AHRQ Publication No. 13(14)-EHC130-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013.
30. Downes MJ, Brennan ML, Williams HC, et al. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458.
31. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52:377-84.
32. Dwan K, Gamble C, Kolamunnage-Dona R, et al. Assessing the potential for outcome reporting bias in a review: A tutorial. Trials 2010;11:52.
33. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-6.
34. Guyatt GH, Oxman AD, Vist G, et al. GRADE guidelines: 4. Rating the quality of evidence-study limitations (risk of bias). J Clin Epidemiol 2011;64:407-15.
35. Guyatt GH, Oxman AD, Montori V, et al. GRADE guidelines: 5. Rating the quality of evidence-publication bias. J Clin Epidemiol 2011;64:1277-82.
36. Schönemann H, Brozek J, Guyatt G, et al. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. http://gdt. guidelinedevelopment. org/app/handbook/handbook. html.
37. Santesso N, Carrasco-Labra A, Langendam M, et al. Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. J Clin Epidemiol 2016;74.
38. Hayden JA, van der Windt DA, Cartwright JL, et al. Assessing bias in studies of prognostic factors. Ann Intern Med 2013;158:280-6.
39. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Chichester (UK: John Wiley & Sons, 2008:187-.
40. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. London: The Cochrane Collaboration, 2011.
41. Higgins JPT, Savović J, Page MJ, et al. Revised Cochrane risk of bias tool for randomized trials (RoB 2.0), Version 20. 2016 https://sites. google. com/site/riskofbiastool/(accessed 19 Sep 2017).
42. Higgins JPT, Sterne JAC, Savović J, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Methods Cochrane Database of Systematic Reviews 2016;10(Suppl 1):29-31.
43. Hooijmans CR, Rovers MM, de Vries RB, et al. SYRCLE's risk of bias tool for animal studies. BMC Med Res Methodol 2014;14:43.
44. Kim SY, Park JE, Lee YJ, et al. Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. J Clin Epidemiol 2013;66:408-14.
45. Meader N, King K, Llewellyn A, et al. A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation. Syst Rev 2014;3:82.
46. Stewart GB, Higgins JP, Schönemann H, et al. The use of Bayesian networks to assess the quality of evidence from research synthesis: 1. PLoS One 2015;10:e0114497.
47. Reid EK, Tejani AM, Huan LN, et al. Managing the incidence of selective reporting bias: A survey of Cochrane review groups. Syst Rev 2015;4:85.
48. Saini P, Loke YK, Gamble C, et al. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ 2014;349:g6501.
49. Salanti G, Del Giovane C, Chaimani A, et al. Evaluating the quality of evidence from a network meta-Analysis. PLoS One 2014;9:e99682.
50. Higgins JP, Del Giovane C, Chaimani A, et al. Evaluating the Quality of Evidence from a Network Meta-Analysis. Value Health 2014;17:A324.
51. Viswanathan M, Berkman ND. Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol 2012;65:163-78.
52. Viswanathan M, Berkman ND, Dryden DM, et al. AHRQ Methods for Effective Health Care. Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank. Rockville (MD: Agency for Healthcare Research and Quality (US), 2013.
53. Armijo-Olivo S, Stiles CR, Hagen NA, et al. Assessment of study quality for systematic reviews: A comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract 2012;18:12-18.
54. Armijo-Olivo S, Ospina M, da Costa BR, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One 2014;9:e96920.
55. Bilandzic A, Fitzpatrick T, Rosella L, et al. Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool. PLoS Med 2016;13:e1001987.
56. Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009;339:b4012.
57. Hartling L, Bond K, Vandermeer B, et al. Applying the risk of bias tool in a systematic review of combination long-Acting beta-Agonists and inhaled corticosteroids for persistent asthma. PLoS One 2011;6:e17242.
58. Hartling L, Hamm M, Milne A, et al. AHRQ Methods for Effective Health Care. Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments. Rockville (MD: Agency for Healthcare Research and Quality (US), 2012.
59. Hartling L, Hamm MP, Milne A, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 2013;66:973-81.
60. Jordan VM, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol 2017;81:72-6.
61. Kumar A, Miladinovic B, Guyatt GH, et al. GRADE guidelines system is reproducible when instructions are clearly operationalized even among the guidelines panel members with limited experience with GRADE. J Clin Epidemiol 2016;75:115-8.
62. Llewellyn A, Whittington C, Stewart G, et al. The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 2. Inter-Rater Reliability and Comparison with Standard GRADE Assessment. PLoS One 2015;10:e0123511.
63. Mustafa RA, Santesso N, Brozek J, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol 2013;66:736-42.
64. Norris SL, Holmer HK, Ogden LA, et al. AHRQ Methods for Effective Health Care. Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness. Rockville (MD): Agency for Healthcare Research and Quality (US), 2012.
65. O'Connor SR, Tully MA, Ryan B, et al. Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: A comparison study. BMC Res Notes 2015;8:224.
66. Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ 2013;346:f1798.
67. Whiting PF, Rutjes AW, Westwood ME, et al. A systematic review classifies sources of bias and variation in diagnostic test accuracy studies. J Clin Epidemiol 2013;66:1093-104.
68. Sterne JAC, Egger M, Moher D, et al. Chapter 10: Addressing reporting biases. In: Higgins JPT, Churchill R, Chandler J, eds. Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0. Chichester, UK: The Cochrane Collaboration, 2017.
69. Atakpo P, Vassar M. Publication bias in dermatology systematic reviews and meta-Analyses. J Dermatol Sci 2016;82:69-74.
70. Lundh A, Gøtzsche PC. Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol 2008;8:22.