Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool

Journal of Clinical Epidemiology

Volumn 96, Issue , 2018, Pages 54-62

  Gates, Allison ^a   Vandermeer, Ben ^a   Hartling, Lisa ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

Automation; Bias; Clinical trials; Evaluation; Evidence based medicine; Machine learning

Indexed keywords

ARTICLE; CROSS-SECTIONAL STUDY; HUMAN; MACHINE LEARNING; PRIORITY JOURNAL; RELIABILITY; RISK ASSESSMENT; ROBOTREVIEWER; SENSITIVITY AND SPECIFICITY; STATISTICAL BIAS; SYSTEMATIC REVIEW (TOPIC); PROSPECTIVE STUDY; REPRODUCIBILITY; TECHNOLOGY;

BIAS; CROSS-SECTIONAL STUDIES; HUMANS; MACHINE LEARNING; PROSPECTIVE STUDIES; REPRODUCIBILITY OF RESULTS; SYSTEMATIC REVIEWS AS TOPIC; TECHNOLOGY;

EID: 85043382213     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2017.12.015     Document Type: Article

References (39)

1. McNair, P., Lewis, G., Levels of evidence in medicine. Int J Sports Phys Ther 7:5 (2012), 474–481.
2. Askie, L., Offringa, M., Systematic reviews and meta-analysis. Semin Fetal Neonatal Med 20:6 (2015), 403–409.
3. Elliott, J.H., Turner, T., Clavisi, O., Thomas, J., Higgins, J.P., Mavergames, C., et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med, 11(2), 2014, e1001603.
4. Bastian, H., Glasziou, P., Chalmers, I., Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?. PLoS Med, 7(9), 2010, e1000326.
5. Crequit, P., Trinquart, L., Yavchitz, A., Ravaud, P., Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC Med, 14, 2016, 8.
6. Millard, L.A., Flach, P.A., Higgins, J.P., Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol 45 (2016), 266–277.
7. Shojania, K.G., Sampson, M., Ansari, M.T., Ji, J., Doucette, S., Moher, D., How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147 (2007), 224–233.
8. Wallace, B.C., Dahabreh, I.J., Schmid, C.H., Lau, J., Trikalinos, T.A., Modernizing the systematic review process to inform comparative effectiveness: tools and methods. J Comp Eff Res 2:3 (2013), 273–282.
9. Tsertsvadze, A., Chen, Y.F., Moher, D., Sutcliffe, P., McCarthy, N., How to conduct systematic reviews more expeditiously?. Syst Rev, 4, 2015, 160.
10. The Cochrane Collaboration, Chapter 8: Assessing risk of bias in included studies. Higgins, J.P.T., Green, S., (eds.) Cochrane handbook of systematic reviews, 2011, Cochrane, London, UK Version 5.1.0.
11. Savovic, J., Weeks, L., Sterne, J.A., Turner, L., Altman, D.G., Moher, D., et al. Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev, 3, 2014, 37.
12. Marshall, I.J., Kuiper, J., Wallace, B.C., RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc 23 (2016), 193–201.
13. Mintz, M., Bills, S., Snow, R., Jurafsky, D., Distant supervision for relation extraction without labeled data. Proceedings of the 47th meeting of the ACL and the 4th IJCNLP of the AFNLP, 2009, Association for Computational Linguistics, Suntec, Singapore, 1003–1011.
14. Ling, X., Jiang, J., He, X., Mei, Q., Zhai, C., Schatz, B., Automatically generating gene summaries from biomedical literature. Pac Symp Biocomput, 2006, 40–51.
15. Banko, M., Brill, E., Scaling to very very large corpora for natural language disambiguation. Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, 2001, Association for Computational Linguistics, Toulouse, France, 26–33.
16. Richer, L., Billinghurst, L., Linsdell, M.A., Russell, K., Vandermeer, B., Crumley, E.T., et al. Drugs for the acute treatment of migraine in children and adolescents. Cochrane Database Syst Rev, 4, 2016 CD005220.
17. Sumamo Schellenberg, E., Dryden, D.M., Pasichnyk, D., Ha, C., Vandermeer, B., Friedman, B.W., et al. Acute migraine treatment in emergency settings. 2012, Agency for Healthcare Research and Quality, Rockville, MD.
18. Seida, J.C., Schouten, J.R., Mousavi, S.S., Hamm, M., Beaith, A., Vandermeer, B., et al. First- and second-generation antipsychotics for children and young adults. 2012, Agency for Healthcare Research and Quality, Rockville, MD.
19. Abou-Setta, A.M., Mousavi, S.S., Spooner, C., Schouten, J.R., Pasichnyk, D., Armijo-Olivo, S., et al. First-generation versus second-generation antipsychotics in adults: comparative effectiveness. 2012, Agency for Healthcare Research and quality, Rockville, MD.
20. Hartling, L., Bialy, L.M., Vandermeer, B., Tjosvold, L., Johnson, D.W., Plint, A.C., et al. Epinephrine for bronchiolitis. Cochrane Database Syst Rev, 2011 CD003123.
21. Fernandes, R.M., Hartling, L., Glucocorticoids for acute viral bronchiolitis in infants and young children. JAMA 311 (2014), 87–88.
22. Pillay, J., Armstrong, M.J., Butalia, S., Donovan, L.E., Sigal, R.J., Chordiya, P., et al. Behavioral programs for type 1 diabetes mellitus: a systematic review and meta-analysis. Ann Intern Med 163 (2015), 836–847.
23. Pillay, J., Armstrong, M.J., Butalia, S., Donovan, L.E., Sigal, R.J., Vandermeer, B., et al. Behavioral programs for type 2 diabetes mellitus: a systematic review and network meta-analysis. Ann Intern Med 163 (2015), 848–860.
24. Freedman, S.B., Pasichnyk, D., Black, K.J.L., Fitzpatrick, E., Gouin, S., Milne, A., et al., Pediatric Emergency Research Canada Gastroenteritis Study Group. Gastroenteritis therapies in developed countries: systematic review and meta-analysis. PLoS One, 12, 2015, e0176741.
25. Abou-Setta, A.M., Beaupre, L.A., Jones, C.A., Rashiq, S., Hamm, M.P., Sadowski, C.A., et al. Pain management interventions for hip fracture. 2011, Agency for Healthcare Research and Quality, Rockville, MD.
26. Gates, A., Hartling, L., Vandermeer, B., Caldwell, P., Contopoulos-Ioannidis, D.G., Curtis, S., et al. The conduct and reporting of child health research: an analysis of randomized controlled trials published in 2012 and evaluation of change over 5 years. J Pediatr 193 (2018), 237–244 e37.
27. Hartling, L., Hamm, M.P., Fernandes, R.M., Dryden, D.M., Vandermeer, B., Quantifying bias in randomized controlled trials in child health: a meta-epidemiological study. PLoS One, 9, 2014, e88008.
28. Hartling, L., Hamm, M., Milne, A., Vandermeer, B., Santaguida, P.L., Ansari, M., et al. Validity and inter-rater reliability testing of quality assessment instruments. 2012, Agency for Healthcare Research and Quality, Rockville, MD.
29. Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics 33 (1977), 159–174.
30. Marshall, C., The systematic review toolbox. Available at http://systematicreviewtools.com/index.php, 2017. (Accessed 25 April 2017)
31. Armijo-Olivo, S., Ospina, M., da Costa, B.R., Egger, M., Saltaji, H., Fuentes, J., et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One, 9, 2014, e96920.
32. Hartling, L., Hamm, M.P., Milne, A., Vandermeer, B., Santaguida, P.L., Ansari, M., et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 66 (2013), 973–981.
33. Katikireddi, S.V., Egan, M., Petticrew, M., How do systematic reviews incorporate risk of bias assessements into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69 (2015), 189–195.
34. Wood, L., Egger, M., Gluud, L.L., Schulz, K.F., Jüni, P., Altman, D.G., et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336 (2008), 601–605.
35. Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D.M., Vandermeer, B., Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol, 17, 2017, 64.
36. Pham, B., Klassen, T.P., Lawson, M.L., Moher, D., Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 58 (2005), 769–776.
37. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med, 6(7), 2009, e1000097.
38. Zhang, Y., Marshall, I., Wallace, B.C., Rationale-augmented convolutional neural networks for text classification. Proc Conf Empir Methods Nat Lang Process 2016 (2016), 795–804.
39. Marshall, I.J., Kuiper, J., Banner, E., Wallace, B.C., Automating biomedical evidence synthesis: RobotReviewer. Proceeding of the 55th Annual Meeting of the Association for Computational Linguistics-System Demonstrations, 2017, Association for Computational Linguistics, Vancouver, Canada, 7–12.