Prioritising references for systematic reviews with RobotAnalyst: A user study

Research Synthesis Methods

Volumn 9, Issue 3, 2018, Pages 470-488

  Przybyła, Piotr ^a   Brockmeier, Austin J ^a   Kontonatsios, Georgios ^a   Le Pogam, Marie Annick ^b   McNaught, John ^a   von Elm, Erik ^b   Nolan, Kay ^c   Ananiadou, Sophia ^a  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

^b UNIVERSITY OF LAUSANNE   (Switzerland)

^c NATIONAL INSTITUTE FOR HEALTH AND CARE EXCELLENCE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; CLUSTER ANALYSIS; DATA MINING; HUMAN; LITERATURE; MACHINE LEARNING; PROCEDURES; REPRODUCIBILITY; RETROSPECTIVE STUDY; SOFTWARE; STATISTICAL ANALYSIS; SWITZERLAND;

ALGORITHMS; CLUSTER ANALYSIS; DATA INTERPRETATION, STATISTICAL; DATA MINING; HUMANS; MACHINE LEARNING; REPRODUCIBILITY OF RESULTS; RETROSPECTIVE STUDIES; REVIEW LITERATURE AS TOPIC; SOFTWARE; SWITZERLAND;

EID: 85051042447     PISSN: None     EISSN: 17592887     Source Type: Journal    
DOI: 10.1002/jrsm.1311     Document Type: Review

References (85)

1. Higgins JP, Deeks JJ. Selecting studies and collecting data; 2011. Version 5.1.0 [updated March 2011].
2. NICE | The National Institute for Health and Care Excellence. https://www.nice.org.uk; Accessed November 11, 2017.
3. The Campbell Collaboration. http://www.campbellcollaboration.org. Accessed November 11, 2017.
4. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.
5. Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med. 2002;21(11):1635-1640.
6. Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. J Med Libr Assoc. 2017;105(1):84-87.
7. Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev. 2016;5(140).
8. Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Syst Rev. 2013;2(78).
9. Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc. 2006;13(2):206-219.
10. Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2):e1001603.
11. Thomas J, Noel-Storr A, Marshall I, et al. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017;91.
12. Cohen AM, Ambert K, McDonagh M. Studying the potential impact of automated document classification on scheduling a systematic review update. BMC Med Inform Decis Mak. 2012;12(1):33.
13. Wallace BC, Small K, Brodley CE, et al. Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med. 2012;14(7):663-669.
14. O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(5).
15. Cohen AM. Optimizing feature representation for automated systematic review work prioritization. In: AMIA Annual Symposium Proceedings, Vol. 2008; 2008:121-125.
16. Bekhuis T, Demner-Fushman D. Towards automating the initial screening phase of a systematic review. In: World congress on medical informatics (medinfo) Safran C, Reti S, Marin HF, eds., Stud Health Technol Inform, vol. 160; 2010:146-150.
17. Bekhuis T, Demner-Fushman D. Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers. Artif Intell Med. 2012;55(3):197-207.
18. Bekhuis T, Tseytlin E, Mitchell KJ, Demner-Fushman D. Feature engineering and a proposed decision-support system for systematic reviewers of medical evidence. PLOS ONE. 2014;9(1):e86277.
19. Matwin S, Kouznetsov A, Inkpen D, Frunza O, O'Blenis P. A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc. 2010;17(4):446-53.
20. Frunza O, Inkpen D, Matwin S. Building systematic reviews using automatic text classification techniques. In: International Conference on Computational Linguistics; 2010:303-311.
21. Frunza O, Inkpen D, Matwin S, Klement W, O'Blenis P. Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med. 2011;51(1):17-25.
22. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinf. 2010;11(1):55.
23. Small K, Wallace B, Trikalinos T, Brodley CE. The constrained weight space SVM: learning with ranked features. In: International conference on machine learning (ICML); 2011:865-872.
24. Jonnalagadda S, Petitti D. A new iterative method to reduce workload in systematic review process. Int J Comput Biol Drug Des. 2013;6(1-2):5-17.
25. Dalal SR, Shekelle PG, Hempel S, Newberry SJ, Motala A, Shetty KD. A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating. Med Decis Making. 2013;33(3):343-355.
26. Miwa M, Thomas J, O'Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242-53.
27. Timsina P, Liu J, El-Gayar O. Advanced analytics for the automation of medical systematic reviews. Inf Syst Front. 2016;18(2):237-252.
28. Khabsa M, Elmagarmid A, Ilyas I, Hammady H, Ouzzani M. Learning to identify relevant studies for systematic reviews using random forest and external information. Mach Learn. 2016;102(3):465-482.
29. Hashimoto K, Kontonatsios G, Miwa M, Ananiadou S. Topic detection using paragraph vectors to support active learning in systematic reviews. J Biomed Inform. 2016;62:59-65.
30. Liu J, Timsina P, El-Gayar O. A comparative analysis of semi-supervised learning: the case of article selection for medical systematic reviews. Inf Syst Front. 2016:1–.
31. Kontonatsios G, Brockmeier AJ, Przybyła P, McNaught J, Mu T, Goulermas JY, Ananiadou S. A semi-supervised approach using label propagation to support citation screening. J Biomed Inform. 2017;72:67-76.
32. Cohen AM, Ambert K, McDonagh M. A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. In: AMIA Annual Symposium Proceedings, Vol. 2010; 2010:121.
33. Wallace BC, Small K, Brodley CE, Lau J, Trikalinos Ta. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In: ACM SIGHIT Symposium on International Health Informatics. 2012:819.
34. Thomas J, Brunton J, Graziosi S. EPPI-Reviewer 4.0: Software for Research Synthesis. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2011.
35. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(210).
36. Howard BE, Phillips J, Miller K, et al. SWIFT-review: a text-mining workbench for systematic review. Syst Rev. 2016;5(87).
37. Khodambashi S, Nytrø Ø. A systematic literature review on evaluation of digital tools for authoring evidence-based clinical guidelines. In: Australian National Health Informatics Conference (HIC), 2017;239:48.
38. Cutting DR, Karger DR, Pedersen JO, Tukey JW. Scatter/gather: a cluster-based approach to browsing large document collections. In: ACM SIGIR Conference on Research and Development in Information Retrieval; 1992:318-329.
39. Carpineto C, Osiński S, Romano G, Weiss D. A survey of web clustering engines. ACM Comput Surv. 2009;41(3):1-38.
40. Mu T, Goulermas JY, Korkontzelos I, Ananiadou S. Descriptive document clustering via discriminant learning in a co-embedded space of multilevel similarities. J Assoc Inf Sci Technol. 2016;67(1):106-33.
41. Brockmeier AJ, Mu T, Ananiadou S, Goulermas JY. Self-tuned descriptive document clustering using a predictive network. IEEE Trans Knowl Data Eng. https://doi.org./10.1109/TKDE.2017.2781721
42. Hofmann T. Probabilistic latent semantic indexing. In: ACM SIGIR conference on research and development in information retrieval; 1999:50-57.
43. Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3:993-1022.
44. Griffiths TL, Steyvers M. Finding scientific topics. Proc Natl Acad Sci. 2004;101(suppl 1):5228-5235.
45. Soleimani H, Miller DJ. Parsimonious topic models with salient word discovery. IEEE Trans Knowl Data Eng. 2015;27(3):824-837.
46. Mei Q, Shen X, Zhai C. Automatic labeling of multinomial topic models; 2007:490-99.
47. Wilbur WJ. A thematic analysis of the AIDS literature. In: Pacific Symposium on Biocomputing; 2001:386-397.
48. Hearst MA, Pedersen JO. Reexamining the cluster hypothesis: scatter/gather on retrieval results. In: ACM SIGIR Conference on Research and Development in Information Retrieval; 1996:76-84.
49. Aletras N, Baldwin T, Lau JH, Stevenson M. Representing topics labels for exploring digital libraries. In: ACM/IEEE-CS Joint Conference on Digital Libraries; 2014:239-248.
50. Aphinyanaphongs Y, Aliferis CF. Text categorization models for retrieval of high quality articles in internal medicine. In: AMIA annual symposium proceedings, Vol. 2003; 2003:31-35.
51. Aphinyanaphongs Y, Tsamardinos I, Statnikov A, Hardin D, Aliferis CF. Text categorization models for high-quality article retrieval in internal medicine. J Am Med Inform Assoc. 2005;12(2):207-216.
52. Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal search strategies for detecting clinically sound studies in medline. J Am Med Inform Assoc. 1994;1(6):447-458.
53. Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc. 2009;16(5):690-704.
54. Lewis DD, Gale WA. A sequential algorithm for training text classifiers. In: ACM SIGIR conference on research and development in information retrieval; 1994:3-12.
55. Tong S, Koller D. Support vector machine active learning with applications to text classification. J Mach Learn Res. 2001;2:45-66.
56. Settles B. Active learning literature survey. Computer Sciences Technical Report, University of Wisconsin–Madison; 2009.
57. Salton G, Buckley C. Improving retrieval performance by relevance feedback. J Assoc Inf Sci Technol. 1990;41(4):288-297.
58. Frantzi K, Ananiadou S, Mima H. Automatic recognition of multi-word terms: the C-value/NC-value method. Int J Digital Libr. 2000;3(2):115-130.
59. Shemilt I, Simon A, Hollands GJ, et al. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Syn Meth. 2014;5(1):31-49.
60. Donmez P, Carbonell JG. Proactive learning: cost-sensitive active learning with multiple imperfect oracles; 2008:619-628.
61. Nguyen AT, Wallace BC, Lease M. Combining crowd and expert labels using decision theoretic active learning. In: AAAI Conference on Human Computation and Crowdsourcing; 2015.
62. Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4(80).
63. McCallum AK. MALLET: A machine learning for language toolkit. 2002.
64. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality; In: Advances in Neural Information Processing Systems; 2013:3111-119.
65. Pennington Jeffrey, Socher Richard, Manning ChristopherD. Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP); 2014:1532-1543.
66. Ananiadou S, Rea B, Okazaki N, Procter R, Thomas J. Supporting systematic reviews using text mining. Soc Sci Comput Rev. 2009;27(4):509-523.
67. Choi S, Ryu B, Yoo S, Choi J. Combining relevancy and methodological quality into a single ranking for evidence-based medicine. Inf Sci. 2012;214:76-90.
68. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2015;23(1):193-01.
69. Cohen AM, Smalheiser NR, McDonagh MS, Yu C, Adams CE, Davis JM, Yu PS. Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine. J Am Med Inform Assoc. 2015;22(3):707-717.
70. Millard LA, Flach PA, Higgins JP. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol. 2015;45(1):266-77.
71. Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Res Syn Meth. 2018.
72. Wallace BC, Marshall IJ. Extracting PICO sentences from clinical trial reports using supervised distant supervision. J Mach Learn Res. 2016;17:1-25.
73. “RIS” format documentation. Thomson Reuters - ResearchSoft http://endnote.com/sites/rm/files/m/direct_export_ris.pdf; 2008.
74. Tsuruoka Y, Tateishi Y, Kim JD, et al. Developing a robust part-of-speech tagger for biomedical text, Vol. 3746 LNCS; 2005:382-392.
75. Ng AY, Jordan MI, Weiss Y. On spectral clustering: analysis and an algorithm. In: Advances in Neural Information Processing Systems, Vol. 14; 2002:849-56.
76. Salton G, Wong A, Yang CS. A vector space model for automatic indexing. Commun ACM. 1975;18(11):613-620.
77. Bahmani B, Moseley B, Vattani A, Kumar R, Vassilvitskii S. Scalable k-means++. VLDB Endowment. 2012;5(7):622-633.
78. Fleuret F. Fast binary feature selection with conditional mutual information. J Mach Learn Res. 2004;5(11):1531-1555.
79. Schwarz G. Estimating the dimension of a model. Ann Stat. 1978;6(2):461-464.
80. Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ. LIBLINEAR: a library for large linear classification. J Mach Learn Res. 2008;9:1871-1874.
81. Kanoulas E, Li D, Azzopardi L, Spijker R. CLEF 2017 technologically assisted reviews in empirical medicine overview. In: Working notes of CLEF 2017 - conference and labs of the evaluation forum Cappellato L, Ferro N, Goeuriot L, Mandl T, eds., CEUR Workshop Proceedings, vol. 1866; 2017.
82. Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed—how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.
83. Garner Paul, Hopewell Sally, Chandler Jackie, et al. When and how to update systematic reviews: consensus and checklist. BMJ. 2016;354:i3507.
84. Cormack GV, Grossman MR. Engineering quality and reliability in technology-assisted review. In: ACM SIGIR Conference on Research and Development in Information Retrieval; 2016:75-84.
85. Lau JH, Newman D, Baldwin T. Machine reading tea leaves: automatically evaluating topic coherence and topic model quality. In: Conference of the European Chapter of the Association for Computational Linguistics (EACL); 2014April:530-539.