Three risk of bias tools lead to opposite conclusions in observational research synthesis

Journal of Clinical Epidemiology

Volumn 101, Issue , 2018, Pages 61-72

  Losilla, Josep Maria ^a   Oliveras, Isabel ^a   Marin Garcia, Juan A ^b   Vives, Jaume ^a  

^a UNIVERSITAT AUTÒNOMA DE BARCELONA   (Spain)

^b UNIVERSITAT POLITÈCNICA DE VALÈNCIA   (Spain)

Author keywords

Domain of bias; Meta analysis; Methodological quality; Quality tool; Risk of bias; Systematic review

Indexed keywords

ARTICLE; CONFOUNDING VARIABLE; CONSENSUS; DATA EXTRACTION; EFFECT SIZE; INTERRATER RELIABILITY; MATHEMATICAL AND STATISTICAL PROCEDURES; NEWCASTLE-OTTAWA SCALE; OBSERVATIONAL STUDY; PRIORITY JOURNAL; QUALITY CONTROL; QUALITY OF COHORT STUDIES; RISK ASSESSMENT; RISK OF BIAS IN NONRANDOMIZED STUDIES OF INTERVENTIONS; SYSTEMATIC ERROR; BEHAVIORAL MEDICINE; COMPARATIVE STUDY; HUMAN; META ANALYSIS (TOPIC); METHODOLOGY; PROCEDURES; STATISTICAL BIAS;

BEHAVIORAL MEDICINE; BIAS; HUMANS; META-ANALYSIS AS TOPIC; OBSERVATIONAL STUDIES AS TOPIC; RESEARCH DESIGN; RISK ASSESSMENT; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85048892898     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2018.05.021     Document Type: Article

References (57)

1. Hartling, L., Ospina, M., Liang, Y., Dryden, D.M., Hooton, N., Krebs Seida, J., et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ, 339, 2009, b4012.
2. Johnson, B.T., Low, R.E., MacDonald, H.V., Panning for the gold in health research: incorporating studies’ methodological quality in meta-analysis. Psychol Health 30 (2014), 135–152.
3. Higgins, J., Green, S., Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available at: http://training.cochrane.org/handbook. (Accessed 12 December 2017)
4. Jüni, P., Witschi, A., Bloch, R., Egger, M., The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 (1999), 1054–1060.
5. Centre for Reviews and Dissemination, Systematic reviews: CRD's guidance for undertaking reviews in health care. CRD, University of York; 2009. Available at: https://www.york.ac.uk/crd/guidance/. (Accessed 30 May 2017)
6. Jüni, P., Altman, D.G., Egger, M., Systematic reviews in health care—assessing the quality of controlled clinical trials. Br Med J 323 (2001), 42–46.
7. Hootman, J.M., Driban, J.B., Sitler, M.R., Harris, K.P., Cattano, N.M., Reliability and validity of three quality rating instruments for systematic reviews of observational studies. Res Synth Methods 2 (2011), 110–118.
8. Margulis, A., Pladevall, M., Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle–Ottawa scale and the RTI item bank. Clin Epidemiol, 2014, 359–368.
9. Oliveras, I., Losilla, J.-M., Vives, J., Methodological quality is underrated in systematic reviews and meta-analyses in health psychology. J Clin Epidemiol 86 (2017), 59–70.
10. Deeks, J.J., Dinnes, J., D'Amico, R., Sowden, A.J., Sakarovitch, C., Song, F., et al. Evaluating non-randomised intervention studies. Health Technol Assess, 7, 2003 iii–x, 1–173.
11. Sanderson, S., Tatt, I.D., Higgins, J.P.T., Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 36 (2007), 666–676.
12. Jarde, A., Losilla, J.M., Vives, J., Methodological quality assessment tools of non-experimental studies: a systematic review. An Psicol 28 (2012), 617–628.
13. Ijaz, S., Verbeek, J.H., Mischke, C., Ruotsalainen, J., Inclusion of nonrandomized studies in Cochrane systematic reviews was found to be in need of improvement. J Clin Epidemiol 67 (2014), 645–653.
14. Armijo-Olivo, S., Stiles, C.R., Hagen, N.A., Biondo, P.D., Cummings, G.G., Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract 18 (2012), 12–18.
15. Herbison, P., Hay-Smith, J., Gillespie, W.J., Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 59 (2006), 1249–1256.
16. O'Connor, S.R., Tully, M.A., Ryan, B., Bradley, J.M., Baxter, G.D., McDonough, S.M., Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study. BMC Res Notes, 8, 2015, 224.
17. Colle, F., Rannou, F., Revel, M., Fermanian, J., Poiraudeau, S., Impact of quality scales on levels of evidence inferred from a systematic review of exercise therapy and low back pain. Arch Phys Med Rehabil 83 (2002), 1745–1752.
18. Jarde, A., Losilla, J.M., Vives, J., Suitability of three different tools for the assessment of methodological quality in ex post facto studies. Int J Clin Heal Psychol 12 (2012), 97–108.
19. Whiting, P., Rutjes, A.W.S., Dinnes, J., Reitsma, J.B., Bossuyt, P.M.M., Kleijnen, J., A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol 58 (2005), 1–12.
20. Higgins, J.P.T., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 343, 2011, d5928.
21. Jarde, A., Losilla, J.M., Vives, J., Rodrigo, M.F., Q-Coh: a tool to screen the methodological quality of cohort studies in systematic reviews and meta-analyses. Int J Clin Heal Psychol 13 (2013), 138–146.
22. Sterne, J.A., Higgins, J.P.T., Elbers, R.G., Reeves, B.C., The development group for ROBINS-I. Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I): detailed guidance. Available at: www.riskofbias.info. (Accessed 30 May 2017)
23. Whiting, P.F., Rutjes, A.W.S., Westwood, M.E., Mallett, S., Deeks, J.J., Reitsma, J.B., et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155 (2011), 529–536.
24. Wells, G., Shea, B., O'Connell, D., Peterson, J., The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses 2000. Available at: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. (Accessed 30 May 2017)
25. Sterne, J.A., Hernán, M.A., Reeves, B.C., Savović J., Berkman, N.D., Viswanathan, M., et al. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ, 355, 2016, i4919.
26. Hartling, L., Milne, A., Hamm, M.P., Vandermeer, B., Ansari, M., Tsertsvadze, A., et al. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol 66 (2013), 982–993.
27. Lo, C.K.-L., Mertz, D., Loeb, M., Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments. BMC Med Res Methodol, 14, 2014, 45.
28. Oremus, M., Oremus, C., Hall, G.B.C., McKinnon, M.C., Inter-rater and test–retest reliability of quality assessments by novice student raters using the Jadad and Newcastle–Ottawa Scales. BMJ Open, 2, 2012, e001368.
29. Pan, A., Sun, Q., Okereke, O.I., Rexrode, K.M., Hu, F.B., Depression and risk of stroke morbidity and mortality: a meta-analysis and systematic review. JAMA 306 (2011), 1241–1249.
30. Jike, M., Itani, O., Watanabe, N., Buysse, D.J., Kaneita, Y., Long sleep duration and health outcomes: a systematic review, meta-analysis and meta-regression. Sleep Med Rev 39 (2018), 25–36.
31. Porcelli, B., Pozza, A., Bizzaro, N., Fagiolini, A., Costantini, M.C., Terzuoli, L., et al. Association between stressful life events and autoimmune diseases: a systematic review and meta-analysis of retrospective case-control studies. Autoimmun Rev 15 (2016), 325–334.
32. Xue, J., Chen, W., Chen, L., Gaudet, L., Moher, D., Walker, M., et al. Significant discrepancies were found in pooled estimates of searching with Chinese indexes versus searching with English indexes. J Clin Epidemiol 70 (2016), 246–253.
33. Zheng, Y., Wu, X., Lin, X., Lin, H., The prevalence of depression and depressive symptoms among eye disease patients: a systematic review and meta-analysis. Sci Rep 7 (2017), 1–9.
34. Cohen, J.A., Coefficient of agreement for nominal scales. Educ Psychol Meas 20 (1960), 37–46.
35. Fleiss, J.L., Measuring nominal scale agreement among many raters. Psychol Bull 76 (1971), 378–382.
36. Feinstein, A.R., Cicchetti, D.V., High agreement but low Kappa: I. The problems of two paradoxes. J Clin Epidemiol 43 (1990), 543–549.
37. Lantz, C.A., Nebenzahl, E., Behavior and interpretation of the κ statistic: resolution of the two paradoxes. J Clin Epidemiol 49 (1996), 431–434.
38. Cicchetti, D.V., Feinstein, A.R., High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 43 (1990), 551–558.
39. Uebersax, J., Statistical methods for diagnostic agreement. 2015 Available at: http://www.john-uebersax.com/stat/agree.htm. (Accessed 25 October 2017)
40. Byrt, T., Bishop, J., Carlin, J.B., Bias, prevalence and kappa. J Clin Epidemiol 46 (1993), 423–429.
41. Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics, 33, 1977, 159.
42. Gamer, M., Lemon, J., Fellows, I., Sing, P., irr: various coefficients of interrater reliability and agreement. 2012, R package version 0.84 Available at: https://cran.r-project.org/package=irr.
43. Kendall, M.G., A new measure of rank correlation. Biometrika, 30, 1938, 81.
44. Hartung, J., Knapp, G., Sinha, B.K., Statistical meta-analysis with applications. 2008, John Wiley & Sons, Inc, Hoboken, NJ, USA.
45. Verhagen, A.P., de Vet, H.C.W., Vermeer, F., Widdershoven, J.W.M.G., de Bie, R.A., Kessels, A.G.H., et al. The influence of methodologic quality on the conclusion of a landmark meta-analysis on thrombolytic therapy. Int J Technol Assess Health Care 18 (2002), 11–23.
46. Balk, E.M., Bonis, P.A.L., Moskowitz, H., Schmid, C.H., Ioannidis, J.P.A., Wang, C., et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 287 (2002), 2973–2982.
47. Hartling, L., Hamm, M., Milne, A., Vandermeer, B., Santaguida, P.L., Ansari, M., et al. Validity and inter-rater reliability testing of quality assessment instruments. 2012, Agency for Healthcare Research and Quality (US), Rockville, MD Available at: https://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0077153/.
48. Moher, D., Pham, B., Jones, A., Cook, D.J., Jadad, A.R., Moher, M., et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet 352 (1998), 609–613.
49. Schulz, K.F., Chalmers, I., Hayes, R.J., Altman, D.G., Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273 (1995), 408–412.
50. Egger, M., Juni, P., Bartlett, C., Holenstein, F., Sterne, J., How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 7 (2003), 1–76.
51. Ahn, S., Becker, B.J., Incorporating quality scores in meta-analysis. J Educ Behav Stat 36 (2011), 555–585.
52. Conn, V.S., Rantz, M.J., Focus on research methods: research methods: managing primary study quality in meta-analyses. Res Nurs Health 26 (2003), 322–333.
53. Morgan, R., Sterne, J., Higgins, J., Thayer, K., Schunemann, H., Rooney, A.T.K., A new instrument to assess risk of bias in non-randomised studies of exposures (ROBINS-E): application to studies of environmental exposure. Abstr. Glob. Evid. Summit. 2017, Cochrane Database of Systematic Reviews, Cape Town, 10.1002/14651858.CD201702.
54. Turner, R.M., Spiegelhalter, D.J., Smith, G.C.S., Thompson, S.G., Bias modelling in evidence synthesis. J R Stat Soc Ser A Stat Soc 172 (2009), 21–47.
55. Hernán, M.A., With great data comes great responsibility. Epidemiology 22 (2011), 290–291.
56. Faggion, C.M., The rationale for rating risk of bias should be fully reported. J Clin Epidemiol, 76, 2016, 238.
57. Bhindi, B., Wallis, C.J.D., Boorjian, S.A., Thompson, R.H., Farrell, A., Kim, S.P., et al. The role of lymph node dissection in the management of renal cell carcinoma: a systematic review and meta-analysis. BJU Int 121 (2018), 684–698.