Reproducibility of studies on text mining for citation screening in systematic reviews: Evaluation and checklist

Journal of Biomedical Informatics

Volumn 73, Issue , 2017, Pages 1-13

  Olorisade, Babatunde Kazeem ^a   Brereton, Pearl ^a   Andras, Peter ^a  

^a KEELE UNIVERSITY   (United Kingdom)

Author keywords

Citation screening; Reproducibility; Reproducible research; Systematic review; Text mining

Indexed keywords

APPLICATION PROGRAMS; DIAGNOSIS; LEARNING ALGORITHMS; PUBLISHING; SOFTWARE ENGINEERING;

CONTEXT INDEPENDENT; REPRODUCIBILITIES; REPRODUCIBLE RESEARCH; SYSTEMATIC LITERATURE REVIEW; SYSTEMATIC REVIEW; TEXT MINING; TEXT MINING TECHNIQUES; THIRD PARTY SOFTWARE;

DATA MINING;

ACCESS TO INFORMATION; ARTICLE; AUTOMATION; CITATION ANALYSIS; DATA ANALYSIS; DATA ANALYSIS SOFTWARE; DATA MINING; INFORMATION PROCESSING; MACHINE LEARNING; MATHEMATICAL ANALYSIS; MEDICAL RESEARCH; PERCEPTRON; PRACTICE GUIDELINE; PRIORITY JOURNAL; REPRODUCIBILITY; SUPPORT VECTOR MACHINE; SYSTEMATIC REVIEW (TOPIC); CHECKLIST; HUMAN; LITERATURE; PUBLICATION;

BIOMEDICAL RESEARCH; CHECKLIST; DATA MINING; HUMANS; PUBLICATIONS; REPRODUCIBILITY OF RESULTS; REVIEW LITERATURE AS TOPIC;

EID: 85025143405     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2017.07.010     Document Type: Article

References (85)

1. Donoho, D.L., Maleki, A., Rahman, I.U., Shahram, M., Stodden, V., Reproducible research in computational harmonic analysis. Comput. Sci. Eng. 11:1 (2009), 8–18, 10.1126/science.1213847.
2. Mesirov, J.P., Accessible reproducible research. Science 327:5964 (2010), 415–416.
3. King, G., Replication, replication. PS: Polit. Sci. Polit. 28:03 (1995), 444–452.
4. Peng, R.D., Reproducible research in computational science. Science 334:6060 (2011), 1226–1227, 10.1126/science.1213847.Reproducible.
5. Sandve, G.K., Nekrutenko, A., Taylor, J., Hovig, E., Ten simple rules for reproducible computational research. PLoS Comput. Biol., 9(10), 2013, e1003285.
6. Waltemath, D., Adams, R., Bergmann, F.T., Hucka, M., Kolpakov, F., Miller, A.K., Moraru, I.I., Nickerson, D., Sahle, S., Snoep, J.L., Reproducible computational biology experiments with SED-ML-the simulation experiment description markup language. BMC Syst. Biol., 5(1), 2011, 198, 10.1186/1752–0509–5–198.
7. Fomel, S., Hennenfent, G., Reproducible computational experiments using SCons. IEEE International Conference on Acoustics, Speech and Signal Processing, 2007, ICASSP 2007, vol. 4, 2007, IEEE pp. IV–1257.
8. Fomel, S., Claerbout, J.F., Reproducible research. Comput. Sci. Eng. 11:1 (2009), 5–7.
9. Hothorn, T., Held, L., Friede, T., Biometrical journal and reproducible research. Biometrical J. 51:4 (2009), 553–555.
10. Madeyski, L., Kitchenham, B., Would wider adoption of reproducible research be beneficial for empirical software engineering research?. J. Intell. Fuzzy Syst. 32:2 (2017), 1509–1521.
11. Gupta, V., Lehal, G.S., et al. A survey of text mining techniques and applications. J. Emerg. Technol. Web Intell. 1:1 (2009), 60–76.
12. Mooney, R.J., Bunescu, R., Mining knowledge from text using information extraction. ACM SIGKDD Explor. Newslett. 7:1 (2005), 3–10.
13. Hotho, A., Nürnberger, A., Paaß, G., A brief survey of text mining. LDV Forum, vol. 20, 2005, 19–62.
14. A.-H. Tan, et al., Text mining: the state of the art and the challenges, in: Proceedings of the PAKDD 1999 Workshop on Knowledge Disocovery from Advanced Databases, vol. 8, sn, 1999, pp. 65–70.
15. Croft, W.B., Metzler, D., Strohman, T., Search Engines: Information Retrieval in Practice, vol. 283, 2010, Addison-Wesley Reading.
16. Kitchenham, B.A., Dyba, T., Jorgensen, M., Evidence-based software engineering. Proceedings of the 26th International Conference on Software Engineering, 2004, IEEE Computer Society, 273–281.
17. Higgins, J.P., Green, S., Cochrane Handbook for Systematic Reviews of Interventions, vol. 4, 2011, John Wiley & Sons.
18. Kitchenham, B.A., Budgen, D., Brereton, P., Evidence-Based Software Engineering and Systematic Reviews, vol. 4, 2015, CRC Press.
19. Hassler, E., Carver, J.C., Kraft, N.A., Hale, D., Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, 2014, ACM, 31.
20. Olorisade, B.K., de Quincey, E., Brereton, P., Andras, P., A critical analysis of studies that address the use of text mining for citation screening in systematic reviews. Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, 2016, ACM, 14.
21. Aphinyanaphongs, Y., Tsamardinos, I., Statnikov, A., Hardin, D., Aliferis, C.F., Text categorization models for high-quality article retrieval in internal medicine. J. Am. Med. Infor. Assoc. 12:2 (2005), 207–216.
22. O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., Ananiadou, S., Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst. Rev., 4(1), 2015, 5.
23. Miller, J., Replicating software engineering experiments: a poisoned chalice or the holy grail. Inform. Softw. Technol. 47:4 (2005), 233–244.
24. Davison, A., Automated capture of experiment context for easier reproducibility in computational research. Comput. Sci. Eng. 14:4 (2012), 48–56.
25. T. Crick, B.A. Hall, S. Ishtiaq, Can I Implement Your Algorithm?: A Model for Reproducible Research Software, Available from: .
26. Ioannidis, J.P., Allison, D.B., Ball, C.A., Coulibaly, I., Cui, X., Culhane, A.C., Falchi, M., Furlanello, C., Game, L., Jurman, G., Repeatability of published microarray gene expression analyses. Nat. Genet. 41:2 (2009), 149–155.
27. Rung, J., Brazma, A., Reuse of public genome-wide gene expression data. Nat. Rev. Genet. 14:2 (2013), 89–99.
28. Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L.B., Bourne, P.E., et al. The fair guiding principles for scientific data management and stewardship. Sci. Data, 3, 2016, 160018.
29. Comeau, D.C., Islamaj Doğan, R., Ciccarese, P., Cohen, K.B., Krallinger, M., Leitner, F., Lu, Z., Peng, Y., Rinaldi, F., Torii, M., BioC: a minimalist approach to interoperability for biomedical text processing. Database, 2013, 2013, bat064.
30. Zobel, J., Webber, W., Sanderson, M., Moffat, A., Principles for robust evaluation infrastructure. Proceedings of the 2011 Workshop on Data Infrastructures for Supporting Information Retrieval Evaluation, 2011, ACM, 3–6.
31. Agosti, M., Di Buccio, E., Ferro, N., Masiero, I., Peruzzo, S., Silvello, G., Directions: design and specification of an IR evaluation infrastructure. CLEF, 2012, Springer, 88–99.
32. Freire, J., Fuhr, N., Rauber, A., Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041). Dagstuhl Reports, vol. 6, 2016, Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
33. Ferro, N., Reproducibility challenges in information retrieval evaluation. J. Data Inform. Qual. (JDIQ), 8(2), 2017, 8.
34. Hersh, W., Evaluation of biomedical text-mining systems: lessons learned from information retrieval. Briefings Bioinformatics 6:4 (2005), 344–356.
35. Wilkinson, M.D., Verborgh, R., da Silva Santos, L.O.B., Clark, T., Swertz, M.A., Kelpin, F.D., Gray, A.J., Schultes, E.A., van Mulligen, E.M., Ciccarese, P., Interoperability and fairness through a novel combination of web technologies. Peer J. Comput. Sci., 3, 2017, e110.
36. Cohen, A.M., Hersh, W.R., Peterson, K., Yen, P.-Y., Reducing workload in systematic review preparation using automated citation classification. J. Am. Med. Infor. Assoc. 13:2 (2006), 206–219.
37. Cohen, A.M., Optimizing feature representation for automated systematic review work prioritization. AMIA Annual Symposium Proceedings, vol. 2008, 2008, American Medical Informatics Association, 121.
38. Kim, S., Choi, J., Improving the performance of text categorization models used for the selection of high quality articles. Healthc. Infor. Res. 18:1 (2012), 18–28.
39. Cohen, A.M., Ambert, K., McDonagh, M., Studying the potential impact of automated document classification on scheduling a systematic review update. BMC Med. Infor. Decis. Mak., 12(1), 2012, 33.
40. Cohen, A.M., An effective general purpose approach for automated biomedical document classification. AMIA Annual Symposium Proceedings, vol. 2006, 2006, American Medical Informatics Association, 161.
41. Cohen, A.M., Ambert, K., McDonagh, M., Cross-topic learning for work prioritization in systematic review creation and update. J. Am. Med. Infor. Assoc. 16:5 (2009), 690–704.
42. Choi, S., Ryu, B., Yoo, S., Choi, J., Combining relevancy and methodological quality into a single ranking for evidence-based medicine. Inform. Sci. 214 (2012), 76–90.
43. Martinez, D., Karimi, S., Cavedon, L., Baldwin, T., Facilitating biomedical systematic reviews using ranked text retrieval and classification. Australasian Document Computing Symposium (ADCS), 2008, 53–60.
44. Cohen, A.M., Ambert, K., McDonagh, M., A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. AMIA Annual Symposium Proceedings, vol. 2010, 2010, American Medical Informatics Association, 121.
45. Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., O'blenis, P., A new algorithm for reducing the workload of experts in performing systematic reviews. J. Am. Med. Infor. Assoc. 17:4 (2010), 446–453.
46. Bekhuis, T., Demner-Fushman, D., Towards automating the initial screening phase of a systematic review. Stud. Health Technol. Inform. 160:PART 1 (2010), 146–150, 10.3233/978-1-60750-588-4-146.
47. Khabsa, M., Elmagarmid, A., Ilyas, I., Hammady, H., Ouzzani, M., Learning to identify relevant studies for systematic reviews using random forest and external information. Mach. Learn. 102:3 (2016), 465–482.
48. Howard, B.E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M.R., Holmgren, S., Pelch, K.E., Walker, V., Rooney, A.A., Swift-review: a text-mining workbench for systematic review. Syst. Rev., 5(1), 2016, 87.
49. Frunza, O., Inkpen, D., Matwin, S., Building systematic reviews using automatic text classification techniques. Proceedings of the 23rd International Conference on Computational Linguistics: Posters, 2010, Association for Computational Linguistics, 303–311.
50. Frunza, O., Inkpen, D., Matwin, S., Klement, W., Oblenis, P., Exploiting the systematic review protocol for classification of medical abstracts. Artif. Intell. Med. 51:1 (2011), 17–25.
51. Razavi, A.H., Matwin, S., Inkpen, D., Kouznetsov, A., Parameterized contrast in second order soft co-occurrences: a novel text representation technique in text mining and knowledge extraction. IEEE International Conference on Data Mining Workshops, 2009, ICDMW'09, 2009, IEEE, 471–476.
52. Kouznetsov, A., Matwin, S., Inkpen, D., Razavi, A.H., Frunza, O., Sehatkar, M., Seaward, L., OBlenis, P., Classifying biomedical abstracts using committees of classifiers and collective ranking techniques. Canadian Conference on Artificial Intelligence, 2009, Springer, 224–228.
53. Kouznetsov, A., Japkowicz, N., Using classifier performance visualization to improve collective ranking techniques for biomedical abstracts classification. Canadian Conference on Artificial Intelligence, 2010, Springer, 299–303.
54. Wallace, B.C., Small, K., Brodley, C.E., Trikalinos, T.A., Who should label what? Instance allocation in multiple expert active learning. Proceedings of the 2011 SIAM International Conference on Data Mining, 2011, SIAM, 176–187.
55. Wallace, B.C., Small, K., Brodley, C.E., Trikalinos, T.A., Active learning for biomedical citation screening. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2010, ACM, 173–182.
56. Wallace, B.C., Trikalinos, T.A., Lau, J., Brodley, C., Schmid, C.H., Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics, 11(1), 2010, 55.
57. Miwa, M., Thomas, J., OMara-Eves, A., Ananiadou, S., Reducing systematic review workload through certainty-based screening. J. Biomed. Infor. 51 (2014), 242–253.
58. Wallace, B.C., Small, K., Brodley, C.E., Lau, J., Trikalinos, T.A., Modeling annotation time to reduce workload in comparative effectiveness reviews. Proceedings of the 1st ACM International Health Informatics Symposium, 2010, ACM, 28–35.
59. Cohen, A.M., Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@ 95 measure. J. Am. Med. Infor. Assoc., 18(1), 2011 pp. 104-104.
60. Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., O'blenis, P., Performance of SVM and Bayesian classifiers on the systematic review classification task. J. Am. Med. Infor. Assoc. 18:1 (2011), 104–105.
61. Matwin, S., Sazonova, V., Direct comparison between support vector machine and multinomial naive Bayes algorithms for medical abstract classification. J. Am. Med. Infor. Assoc., 19(5), 2012 pp. 917-917.
62. Wallace, B.C., Small, K., Brodley, C.E., Lau, J., Schmid, C.H., Bertram, L., Lill, C.M., Cohen, J.T., Trikalinos, T.A., Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet. Med. 14:7 (2012), 663–669.
63. Z. Yu, N.A. Kraft, T. Menzies, How to read less: Better machine assisted reading methods for systematic literature reviews. Available from: .
64. Mo, Y., Kontonatsios, G., Ananiadou, S., Supporting systematic reviews using LDA-based document representations. Syst. Rev., 4(1), 2015, 172.
65. González-Barahona, J.M., Robles, G., On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Softw. Eng. 17:1-2 (2012), 75–89, 10.1007/s10664–011–9181–9.
66. Fayyad, U., Piatetsky-Shapiro, G., Smyth, P., From data mining to knowledge discovery in databases. AI Mag., 17(3), 1996, 37.
67. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12:October (2011), 2825–2830.
68. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H., The WEKA data mining software: an update. ACM SIGKDD Explor. Newslett. 11:1 (2009), 10–18.
69. Sokolova, M., Lapalme, G., A systematic analysis of performance measures for classification tasks. Inform. Process. Manage. 45:4 (2009), 427–437.
70. Bekhuis, T., Demner-Fushman, D., Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers. Artif. Intell. Med. 55:3 (2012), 197–207.
71. Bekhuis, T., Tseytlin, E., Mitchell, K.J., Demner-Fushman, D., Feature engineering and a proposed decision-support system for systematic reviewers of medical evidence. PloS One, 9(1), 2014, e86277.
72. Dalal, S.R., Shekelle, P.G., Hempel, S., Newberry, S.J., Motala, A., Shetty, K.D., A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating. Med. Decis. Mak. 33:3 (2013), 343–355.
73. Felizardo, K.R., Andery, G.F., Paulovich, F.V., Minghim, R., Maldonado, J.C., A visual analysis approach to validate the selection review of primary studies in systematic reviews. Inform. Softw. Technol. 54:10 (2012), 1079–1091.
74. Felizardo, K.R., Salleh, N., Martins, R.M., Mendes, E., MacDonell, S.G., Maldonado, J.C., Using visual text mining to support the study selection activity in systematic literature reviews. Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on, 2011, IEEE, 77–86.
75. Felizardo, K.R., Souza, S.R., Maldonado, J.C., The use of visual text mining to support the study selection activity in systematic literature reviews: a replication study. 2013 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER), 2013, IEEE, 91–100.
76. Adeva, J.G., Atxa, J.P., Carrillo, M.U., Zengotitabengoa, E.A., Automatic text classification to support systematic reviews in medicine. Expert Syst. Appl. 41:4 (2014), 1498–1508.
77. Jonnalagadda, S., Petitti, D., A new iterative method to reduce workload in systematic review process. Int. J. Comput. Biol. Drug Des. 6:1–2 (2013), 5–17.
78. Malheiros, V., Hohn, E., Pinho, R., Mendonca, M., A visual text mining approach for systematic reviews. First International Symposium on Empirical Software Engineering and Measurement, 2007. ESEM 2007, 2007, IEEE, 245–254.
79. Shemilt, I., Simon, A., Hollands, G.J., Marteau, T.M., Ogilvie, D., O'Mara-Eves, A., Kelly, M.P., Thomas, J., Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res. Synth. Methods 5:1 (2014), 31–49.
80. Tomassetti, F., Rizzo, G., Vetro, A., Ardito, L., Torchiano, M., Morisio, M., Linked data approach for selection process automation in systematic reviews. 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE 2011), 2011, IET, 31–35.
81. Small, K., Wallace, B., Trikalinos, T., Brodley, C.E., The constrained weight space SVM: learning with ranked features. Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, 865–872.
82. Yu, W., Clyne, M., Dolan, S.M., Yesupriya, A., Wulf, A., Liu, T., Khoury, M.J., Gwinn, M., GAPscreener: an automatic tool for screening human genetic association literature in PubMed using the support vector machine technique. BMC Bioinformatics, 9(1), 2008, 205.
83. Gentleman, R.C., Carey, V.J., Bates, D.M., Bolstad, B., Dettling, M., Dudoit, S., Ellis, B., Gautier, L., Ge, Y., Gentry, J., Bioconductor: open software development for computational biology and bioinformatics. Genome Biol., 5(10), 2004, R80.
84. Shull, F., Mendoncça, M.G., Basili, V., Carver, J., Maldonado, J.C., Fabbri, S., Travassos, G.H., Ferreira, M.C., Knowledge-sharing issues in experimental software engineering. Empirical Softw. Eng. 9:1 (2004), 111–137.
85. Vegas, S., Juristo, N., Moreno, A., Solari, M., Letelier, P., Analysis of the influence of communication between researchers on experiment replication. Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering, 2006, ACM, 28–37.