Trial2rev: Combining machine learning and crowd-sourcing to create a shared space for updating systematic reviews

JAMIA Open

Volumn 2, Issue 1, 2019, Pages 15-22

  Martin, Paige ^a   Surian, Didi ^a   Bashir, Rabia ^a   Bourgeois, Florence T ^b,c   Dunn, Adam G ^a  

^a MACQUARIE UNIVERSITY   (Australia)

^b BOSTON CHILDREN S HOSPITAL   (United States)

^c HARVARD MEDICAL SCHOOL   (United States)

Author keywords

bibliographic databases; databases as topic; review literature as topic; semi supervised learning

Indexed keywords

ARTICLE; CROWDSOURCING; HUMAN; MACHINE LEARNING; META ANALYSIS (TOPIC); MONITORING; PREDICTION; PRIORITY JOURNAL; REGISTRATION; SOFTWARE; SPACE; SYSTEMATIC REVIEW (TOPIC); TRAINING;

EID: 85063343750     PISSN: None     EISSN: 25742531     Source Type: Journal    
DOI: 10.1093/jamiaopen/ooy062     Document Type: Article

References (42)

1. Garner P, Hopewell S, Chandler J, et al. When and how to update system-atic reviews: consensus and checklist. BMJ 2016; 354: i3507.
2. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep op? PLoS Med 2010; 7 (9): e1000326.
3. Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey. PLoS One 2010; 5 (4): e9914.
4. Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in sys-tematic review preparation using automated citation classification. J Am Med Inform Assoc 2006; 13 (2): 206-19.
5. Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc 2009; 16 (5): 690-704.
6. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Sys-tematic review automation technologies. Syst Rev 2014; 3 (1): 74.
7. O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 2015; 4 (1): 5.
8. Shemilt I, Simon A, Hollands GJ, et al. Pinpointing needles in giant hay-stacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Methods 2014; 5 (1): 31-49.
9. Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Res Synth Methods 2011; 2 (1): 1-14.
10. Miwa M, Thomas J, O'Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform 2014; 51: 242-53.
11. Shekelle PG, Shetty K, Newberry S, Maglione M, Motala A. Machine learning versus standard techniques for updating searches for systematic reviews: a diagnostic accuracy study. Ann Intern Med 2017; 167 (3): 213-5.
12. Ji X, Ritter A, Yen P-Y. Using ontology-based semantic similarity to facilitate the article screening process for systematic reviews. J Biomed Inform 2017; 69: 33-42.
13. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics 2010; 11 (1): 55.
14. Schmucker C, Schell LK, Portalupi S, et al. Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One 2014; 9 (12): e114023.
15. Jones CW, Handler L, Crowell KE, Keil LG, Weaver MA, Platts-Mills TF. Non-publication of large randomized clinical trials: cross sectional analysis. BMJ2013; 347: f6104.
16. Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med 2010; 153 (3): 158-66.
17. Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias-an updated review. PLoS One 2013; 8 (7): e66844.
18. Saito H, Gill CJ. How frequently do the results from completed US clinical trials enter the public domain? A statistical analysis of the Clinical Trials.-gov database. PLoS One 2014; 9 (7): e101826.
19. Song F, Parekh S, Hooper L, et al. Dissemination and publication of re-search findings: an updated review of related biases. Health Technol Assess 2010; 14 (8): 1-193.
20. Vawdrey DK, Hripcsak G. Publication bias in clinical trials of electronic health records. J Biomed Inform 2013; 46 (1): 139-41.
21. Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med 2012; 9 (3): e1001189.
22. Chan A-W, Hrobjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004; 291 (20): 2457-65.
23. Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol 2013; 66 (5): 524-37.
24. Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome report-ing bias in randomised controlled trials on a cohort of systematic reviews. BMJ2010; 340: c365.
25. Dickersin K, Rennie D. Registering clinical trials. JAMA 2003; 290 (4): 516-23.
26. De Angelis C, Drazen JM, Frizelle FA, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med 2004; 351 (12): 1250-1.
27. Trinquart L, Dunn AG, Bourgeois FT. Registration of published randomized trials: a systematic review and meta-analysis. BMC Med 16 (1): 173.
28. Chen R, Desai NR, Ross JS, et al. Publication and reporting of clinical trial results: cross sectional analysis across academic medical centers. BMJ 2016; 352:i637.
29. Shamliyan TA, Kane RL. Availability of results from clinical research: failing policy efforts. J Epidemiol Glob Health 2014; 4 (1): 1-12.
30. Elliott JH, Synnot A, Turner T, et al. Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol 2017; 91: 23-30.
31. Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med 2014; 11 (2): e1001603.
32. Mortensen ML, Adam GP, Trikalinos TA, Kraska T, Wallace BC. An exploration of crowdsourcing citation screening for systematic reviews. Res Synth Methods 2017; 8 (3): 366-86.
33. Krivosheev E, Casati F, Caforio V, Benatallah B. Crowdsourcing paper screening in systematic literature reviews. arXiv: 2017; 170905168.
34. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc 2017; 24 (6): 1165-8.
35. Surian D, Dunn AG, Orenstein L, Bashir R, Coiera E, Bourgeois FT. A shared latent space matrix factorisation method for recommending new trial evidence for systematic review updates. J Biomed Inform 2018; 79: 32-40.
36. Bierer BE, Li R, Barnes M, Sim I. A global, neutral platform for sharing trial data. N Engl J Med 2016; 374 (25): 2411-3.
37. DeVito NJ, Bacon S, Goldacre B. FDAAA TrialsTracker: a live informatics tool to monitor compliance with FDA requirements to report clinical trial results. bioRxiv 2018: 266452.
38. Coens C, Bogaerts J, Collette L. Comment on the TrialsTracker: Automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions. F1000Res 2017; 6: 71.
39. Dunn AG, Coiera E, Bourgeois FT. Unreported links between trial regis-trations and published articles were identified using document similarity measures in a cross-sectional analysis of ClinicalTrials.gov. J Clin Epidemiol 2018; 95: 94-101.
40. Takwoingi Y, Hopewell S, Tovey D, Sutton AJ. A multicomponent deci-sion tool for prioritising the updating of systematic reviews. BMJ 2013; 347: f7191.
41. Bashir R, Surian D, Dunn A. An empirically-defined decision tree to predict systematic reviews at risk of change in conclusion. In: Cochrane Colloquium. Edinburgh, Scotland: Cochrane; 2018.
42. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008; 24 (6): 574-94.