Community challenges in biomedical text mining over 10 years: Success, failure and the future

Briefings in Bioinformatics

Volumn 17, Issue 1, 2016, Pages 132-144

  Huang, Chung Chi ^a   Lu, Zhiyong ^a  

^a NATIONAL INSTITUTES OF HEALTH   (United States)

Author keywords

Biomedical natural language processing (BioNLP); BioNLP challenges; BioNLP shared tasks; Critical assessment; Text mining

Indexed keywords

HUMAN; HUMAN EXPERIMENT; MINING; NATURAL LANGUAGE PROCESSING; WORKFLOW; ALGORITHM; BIOLOGY; DATA MINING; GENETIC DATABASE; PROCEDURES; STATISTICS AND NUMERICAL DATA; TRENDS;

ALGORITHMS; COMPUTATIONAL BIOLOGY; DATA MINING; DATABASES, GENETIC; HUMANS; NATURAL LANGUAGE PROCESSING;

EID: 84960109752     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbv024     Document Type: Article

References (120)

1. Lu Z. PubMed and beyond: a survey of web tools for searching biomedical literature. Database (Oxford) 2011;2011: baq036.
2. Khare R, Leaman R, Lu Z. Accessing biomedical literature in the current information landscape. Methods Mol Biol 2014; 1159:11-31.
3. Islamaj Dogan R, Murray GC, Neveol A, et al. Understanding PubMed user search behavior through log analysis. Database (Oxford) 2009;2009:bap018.
4. Jensen LJ, Saric J, Bork P. Literature mining for the biologist: from information retrieval to biological discovery. Nat Rev Genet 2006;7:119-29.
5. Shatkay H, Feldman R. Mining the biomedical literature in the genomic era: an overview. J Comput Biol 2003;10(6):821-55.
6. Hunter L, Cohen KB. Biomedical language processing: what's beyond PubMed? Mol Cell 2006;21(5):589-94.
7. Berner ES, Moss J. Informatics challenges for the impending patient information explosion. J Am Med Inform Assoc 2005; 12(6):614-17.
8. Linder JA, Ma J, Bates DW, et al. Electronic health record use and the quality of ambulatory care in the United States. Arch Intern Med 2007;167:1400-5.
9. Cohen KB, Hunter L. Natural language processing and systems biology. Artif Intell Syst Biol 2004:147-75.
10. Hersh W, Bhuptiraju RT, Ross L, et al. TREC 2004 genomics track overview. In: Proceedings of the 13th Text Retrieval Conference, 2004
11. Chen D, Muller HM, Sternberg PW. Automatic document classification of biological literature. BMC Bioinformatics 2006;7:370.
12. Krallinger M, Vazquez M, Leitner F, et al. The Protein- Protein Interaction tasks of BioCreative III: classification/ ranking of articles and linking bio-ontology concepts to full text. BMC Bioinformatics 2011;12 (Suppl 8):S3.
13. Wang P, Morgan AA, Zhang Q, et al. Automating document classification for the Immune Epitope Database. BMC Bioinformatics 2007;8:269.
14. Gobeill J, Teodoro D, Patsche E, et al. Report on the TREC 2009 experiments: chemical IR track. In: Proceedings of the 18th Text Retrieval Conference, 2009.
15. Lu Z, Kim W, Wilbur WJ. Evaluation of query expansion using MeSH in PubMed. Inf Retr Boston 2009;12(1):69-80.
16. Lu Z, Kim W, Wilbur WJ. Evaluating relevance ranking strategies for MEDLINE retrieval. J Am Med Inform Assoc 2009;16(1):32-6.
17. Cohen AM, Hersh WR. The TREC 2004 genomics track categorization task: classifying full text biomedical documents. J Biomed Discov Collab 2006;1:4.
18. Segura-Bedmar I, Martínez P, Sánchez-Cisneros D. The 1st DDIExtraction-2011 challenge task: extraction of drug-drug interactions from biomedical texts. In: Proceedings of the 1st Challenge Task on Drug-drug Interaction Extraction, 2011, 1-9.
19. Krallinger M, Leitner F, Rodriguez-Penagos C, et al. Overview of the protein-protein interaction annotation extraction task of BioCreative II. Genome Biol 2008;9 (Suppl 2):S4.
20. Ravikumar K, Liu H, Cohn JD, et al. Literature mining of protein-residue associations with graph rules learned through distant supervision. J Biomed Semantics 2012;3 (Suppl 3):S2.
21. Nédellec C. Learning language in logic - genic interaction extraction challenge. In: Proceedings of the Learning Language in Logic 2005 Workshop, 2005, 31-7.
22. Sun W, Rumshisky A, Uzuner O. Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. J Am Med Inform Assoc 2013;20(5):806-13.
23. Hersh W, Bhupatiraju RT. TREC genomics track overview. In: Proceedings of the 12th Text Retrieval Conference, 2003.
24. Kim JD, Ohta T, Pyysalo S, et al. Overview of BioNLP'09 shared task on event extraction. In: Proceedings of the Workshop on BioNLP: Shared Task, 2009.
25. Yeh AS, Hirschman L, Morgan AA. Evaluation of text data mining for database curation: lessons learned from the KDD Challenge Cup. Bioinformatics 2003;19 (Suppl 1):i331-9.
26. Wei CH, Harris BR, Li D, et al. Accelerating literature curation with text-mining tools: a case study of using PubTator to curate genes in PubMed abstracts. Database (Oxford) 2012;2012:bas041.
27. Burger JD, Doughty E, Khare R, et al. Hybrid curation of gene-mutation relations combining automated extraction and crowdsourcing. Database (Oxford) 2014;2014:bau-094.
28. Khare R, Li J, Lu Z. LabeledIn: cataloging labeled indications for human drugs. J Biomed Inform 2014;52:448-56.
29. Khare R, Burger J, Aberdeen J, et al. Scaling drug indication curation through crowdsourcing. Database (Oxford) 2015; 2015:bav016.
30. Spasic I, Ananiadou S, McNaught J, et al. Text mining and ontologies in biomedicine: making sense of raw text. Brief Bioinform 2005;6(3):239-51.
31. Ananiadou S, Pyysalo S, Tsujii J, et al. Event extraction for systems biology by text mining the literature. Trends Biotechnol 2010;28(7):381-90.
32. Ananiadou S. Advances of Biomedical Text Mining for Semantic Search. Web Science in the Medical Domain 2011;5.
33. Wei CH, Kao HY, Lu Z. PubTator: a web-based text mining tool for assisting biocuration. Nucleic Acids Res 2013;41: W518-22.
34. Salgado D, Krallinger M, Depaule M, et al. MyMiner: a web application for computer-assisted biocuration and text annotation. Bioinformatics 2012;28(17):2285-7.
35. Moult J, Pedersen JT, Judson R, et al. A large-scale experiment to assess protein structure prediction methods. Proteins 1995;23(3):ii-v.
36. Moult J, Fidelis K, Kryshtafovych A, et al. Critical assessment of methods of protein structure prediction (CASP)-round x. Proteins 2014;82 (Suppl 2):1-6.
37. Cohen AM, Hersh WR. A survey of current work in biomedical text mining. Brief Bioinform 2005;6(1):57-71.
38. Simpson MS, Demner-Fushman D. Biomedical text mining: a survey of recent progress. In: Biomedical Text Mining: A Survey of Recent Progress, 2012, 465-517.
39. Yeh A, Hirschman L, Morgan A. Background and Overview for KDD Cup 2002 Task 1: information extraction from biomedical articles. ACM SIGKDD Explor Newsl 2002;4(2):87-9.
40. Hersh W, Cohen A, Yang J, et al. TREC 2005 Genomics Track Overview. In: Proceedings of the 14th Text Retrieval Conference, 2005.
41. Lu Z, Cohen KB, Hunter L. Finding GeneRIFs via gene ontology annotations. Pac Symp Biocomput 2006:52-63.
42. Lu Z, Cohen KB, Hunter L. GeneRIF quality assurance as summary revision. Pac Symp Biocomput 2007:269-80.
43. Hersh W, Cohen AM, Roberts P, et al. TREC 2006 Genomics Track Overview. In: Proceedings of the 15th Text Retrieval Conference, 2006.
44. Hersh W, Cohen A, Ruslen L, et al. TREC 2007 Genomics Track Overview. In: Proceedings of the 16th Text Retrieval Conference, 2007.
45. Lupu M, Piroi F, Huang X, et al. Overview of the TREC 2009 chemical IR track. In: Proceedings of the 18th Text Retrieval Conference, 2009.
46. Lupu M, Tait J, Huang J, et al. TREC-CHEM 2010:Notebook report. In: Proceedings of the 19th Text Retrieval Conference, 2010.
47. Lupu MH, Gurulingappa I. Filippov, et al. Overview of the TREC 2011 Chemical IR Track. In: Proceedings of the 20th Text Retrieval Conference, 2011.
48. Farkas R, Vincze V, Mó ra G, et al. The CoNLL-2010 shared task: learning to detect hedges and their scope in natural language text. In: Proceedings of the 14th Conference on Computational Natural Language Learning: Shared Task, 2010.
49. Yeh A, Morgan A, Colosimo M, et al. BioCreAtIvE task 1A: gene mention finding evaluation. BMC Bioinformatics 2005;6 (Suppl 1):S2.
50. Hirschman L, Yeh A, Blaschke C, et al. Overview of BioCreAtIvE: critical assessment of information extraction for biology. BMC Bioinformatics 2005;6 (Suppl 1):S1.
51. Kim JD, Ohta T, Tsuruoka Y, et al. Introduction to the Bio- Entity Recognition Task at JNLPBA. In: Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, 2004, 28-29.
52. Hirschman L, Colosimo M, Morgan A, et al. Overview of BioCreAtIvE task 1B: normalized gene lists. BMC Bioinformatics 2005;6 (Suppl 1):S11.
53. Morgan AA, Lu Z, Wang X, et al. Overview of BioCreative II gene normalization. Genome Biol 2008;9 (Suppl 2):S3.
54. Lu Z, Kao HY, Wei CH, et al. The gene normalization task in BioCreative III. BMC Bioinformatics 2011;12 (Suppl 8):S2.
55. Wiegers TC, Davis AP, Mattingly CJ. Collaborative biocuration-text-mining development task for document prioritization for curation. Database (Oxford) 2012;2012: bas037.
56. Krallinger M, Leitner F, Rabal O, et al. Overview of the chemical compound and drug name recognition (CHEMDNER) task. In: Proceedings of the Fourth BioCreative Challenge Evaluation Workshop, 2013, 2-33.
57. Krallinger M, Rabal O, Leitner F, et al. The CHEMDNER corpus of chemicals and drugs and its annotation principles. J Cheminform 2015;7:S2.
58. Rebholz-Schuhmann D, Jimeno Yepes A, Li C, et al. Assessment of NER solutions against the first and second CALBC Silver Standard Corpus. J Biomed Semantics 2011;2 (Suppl 5):S11.
59. Rebholz-Schuhmann D, Jimeno Yepes AJ, Van Mulligen EM, et al. CALBC silver standard corpus. J Bioinform Comput Biol 2010;8(1):163-79.
60. Blaschke C, Leon EA, Krallinger M, et al. Evaluation of BioCreAtIvE assessment of task 2. BMC Bioinformatics 2005;6 (Suppl 1):S16.
61. Mao Y, Van Auken K, Li D, et al. Overview of the gene ontology task at BioCreative IV. Database (Oxford) 2014;2014: bau086.
62. Van Auken K, Schaeffer ML, McQuilton P, et al. BC4GO: a full-text corpus for the BioCreative IV GO task. Database (Oxford) 2014;2014:bau074.
63. Leitner F, Krallinger M, Cesareni G, et al. The FEBS Letters SDA corpus: a collection of protein interaction articles with high quality annotations for the BioCreative II.5 online challenge and the text mining community. FEBS Lett 2010;584(19):4129-30.
64. Segura-Bedmar I, Martínez P, Herrero-ZazoM. SemEval-2013 Task 9:Extraction of Drug-Drug Interactions fromBiomedical Texts (DDIExtraction 2013). In: Proceedings of the Seventh International Workshop on Semantic Evaluation, 2013, 341-50.
65. Huang M, Neveol A, Lu Z. Recommending MeSH terms for annotating biomedical articles. J Am Med Inform Assoc 2011; 18(5):660-7.
66. Partalas I, Gaussier E, Ngomo AN. Results of the First BioASQ Workshop. In: Proceedings of the first Workshop on BioASQ, 2013.
67. Balikas G, Partalas I, Ngomo ACN, et al. Results of the BioASQ Track of the Question Answering Lab at CLEF 2014. In: Results of the BioASQ Track of the Question Answering Lab at CLEF 2014. 2014. 1181-93.
68. Kim JD, Wang Y, Takagi T, et al. Overview of Genia Event Task in BioNLP Shared Task 2011. In: Proceedings of the BioNLP Shared Task 2011 Workshop, 2011, 7-15.
69. Kim JD, Wang Y, Yasunori Y. The Genia Event Extraction Shared Task, 2013 Edition - Overview. In: Proceedings of the BioNLP Shared Task 2013 Workshop, 2013, 8-15.
70. Bossy R, Jourde J, Manine AP, et al. BioNLP shared task- The bacteria track. BMC Bioinformatics 2012;13 (Suppl 11): S3.
71. Bossy R, Golik W, Ratkovic Z, et al. BioNLP shared Task 2013 - An Overview of the Bacteria Biotope Task. In: Proceedings of the BioNLP Shared Task 2013 Workshop, 2013, 74-82.
72. Pyysalo S, Ohta T, Rak R, et al. Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011. BMC Bioinformatics 2012;13 (Suppl 11):S2.
73. Pyysalo S, Ohta T, Ananiadou S. Overview of the Cancer Genetics (CG) task of BioNLP Shared Task 2013. In: Proceedings of the BioNLP Shared Task 2013 Workshop, 2013, 58-66.
74. Ohta T, Pyysalo S, Rak R, et al. Overview of the Pathway Curation (PC) task of BioNLP Shared Task 2013. In: Proceedings of the BioNLP Shared Task 2013 Workshop, 2013, 67-75.
75. Kim JD, Nguyen N, Wang Y, et al. The Genia Event and Protein Coreference tasks of the BioNLP shared task 2011. BMC Bioinformatics 2012;13 (Suppl 11):S1.
76. Jourde J, Manine AP, Veber P, et al. BioNLP Shared Task 2011 - Bacteria Gene Interactions and Renaming. In: Proceedings of BioNLP Shared Task 2011 Workshop, 2011.
77. Nédellec C, Bossy R, Kim JD, et al. Overview of BioNLP Shared Task 2013. In: Proceedings of the BioNLP Shared Task 2013 Workshop, 2013, 1-7.
78. Jin F, Huang M, Lu Z, et al. Towards automatic generation of gene summary. In: Proceedings of BioNLP workshop, 2009.
79. Uzuner O, Luo Y, Szolovits P. Evaluating the state-of-theart in automatic de-identification. J Am Med Inform Assoc 2007;14(5):550-63.
80. Uzuner O, Goldstein I, Luo Y, et al. Identifying patient smoking status from medical discharge records. J Am Med Inform Assoc 2008;15(1):14-24.
81. Uzuner O. Recognizing obesity and comorbidities in sparse data. J Am Med Inform Assoc 2009;16(4):61-70.
82. Xu Y, Wang Y, Liu J, et al. Suicide note sentiment classification: a supervised approach augmented by web data. Biomed Inform Insights 2012;5(Suppl. 1):31-41.
83. Uzuner O, Solti I, Cadag E. Extracting medication information from clinical text. J Am Med Inform Assoc 2010;17(5): 514-8.
84. Uzuner O, South BR, Shen S, et al. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 2011;18(5):552-6.
85. Voorhees EM, Tong RM. Overview of the TREC 2011 medical records track. In: Proceedings of the Text Retrieval Conference, 2011.
86. Voorhees EM, Hersh W. Overview of the TREC 2012 medical records track. In: Proceedings of the Text Retrieval Conference, 2012.
87. Suominen H, Salantera S, Velupillai S, et al. Overview of the ShARe/CLEF eHealth evaluation lab 2013. Lect Notes Comput Sci 2013;8138:212-31.
88. Pradhan S, Elhadad N, Chapman W, et al. SemEval- 2014 task 7: analysis of clinical text. In: Proceedings of the 8th workshop on semantic evaluation, 2014, 54-62.
89. Kelly L, Goeuriot L, Suominen H, et al. Overview of the ShARe/CLEF eHealth evaluation lab 2014. In: Proceedings of the CLEF, 2014.
90. Mowery DL, Velupillai S, South BR, et al. Task 2: ShARe/ CLEF eHealth evaluation lab 2014. In: Proceedings of the CLEF, 2014.
91. Goeuriot L, Kelly L, Li W, et al. ShARe/CLEF eHealth evaluation lab 2014, Task 3: User-centred health information retrieval. In: Proceedings of the CLEF, 2014.
92. Arighi CN, Wu CH, Cohen KB, et al. BioCreative-IV virtual issue. Database (Oxford) 2014;2014:bau039.
93. Van Landeghem S, Bjorne J, Wei CH, et al. Large-scale event extraction from literature with multi-level gene normalization. PLoS One 2013;8(4):e55814.
94. Leaman R, Islamaj Dogan R, Lu Z. DNorm: disease name normalization with pairwise learning to rank. Bioinformatics 2013;29(22):2909-17.
95. Wei CH, Kao HY, Lu Z. SR4GN: a species recognition software tool for gene normalization. PLoS One 2012;7(6): e38460.
96. Leaman R, Wei CH, Lu Z. tmChem: a high performance tool for chemical named entity recognition and normalization. J Cheminform 2015;7:S3.
97. Hirschman L, Grishman R, Sager N. From text to structured information: automatic processing of medical reports. In: Proceedings of the National Computer Conference and Exposition, 1976.
98. Hersh W, Buckley C, Leone TJ, et al. OHSUMED: an interactive retrieval evaluation and new large test collection for research. In: The 17th ACM SIGIR Conference on Research and Development in Information Retrieval, 1994, 192-201.
99. Ruch P, Baud RH, Rassinoux AM, et al. Medical document anonymization with a semantic lexicon. In: Proc AMIA Symp 2000, 729-33.
100. Franzen K, Eriksson G, Olsson F, et al. Protein names and how to find them. Int J Med Inform 2002;67(1-3):49-61.
101. Maglott D, Ostell J, Pruitt KD, et al. Entrez Gene: genecentered information at NCBI. Nucleic Acids Res 2005;33: D54-8.
102. Coordinators NR. Database resources of the National Center for Biotechnology Information. Nucleic Acids Res 2013;41:D8-D20.
103. Magrane M, Consortium UniProt. UniProt Knowledgebase: a hub of integrated protein data. Database (Oxford) 2011; 2011:bar009.
104. Reference Genome Group of the Gene Ontology Consortium. The Gene Ontology's Reference Genome Project: a unified framework for functional annotation across species. PLoS Comput Biol 2009;5(7):e1000431.
105. Neveol A, Islamaj Dogan R, Lu Z. Semi-automatic semantic annotation of PubMed queries: a study on quality, efficiency, satisfaction. J Biomed Inform 2011;44(2):310-8.
106. Jessop DM, Adams SE, Willighagen EL, et al. OSCAR4: a flexible architecture for chemical text-mining. J Cheminform 2011;3(1):41.
107. Dogan RI, Leaman R, Lu Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. J Biomed Inform 2014;47:1-10.
108. Dogan RI, Lu Z. An improved corpus of disease mentions in PubMed citations. In: Proceedings of BioNLP workshop, Association for Computational Linguistics, 2012.
109. Stenetorp A, Pyysalo S, Topić G, et al. brat: a Web-based Tool for NLP-Assisted Text Annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2012.
110. Mao Y, Wei CH, Lu Z. NCBI at the 2014 BioASQ challenge task: large-scale biomedical semantic indexing and question answering. In: Proceedings of the 2014 Question Answering Lab at CLEF, 2014.
111. Comeau DC, Islamaj Dogan R, Ciccarese P, et al. BioC: a minimalist approach to interoperability for biomedical text processing. Database (Oxford) 2013;2013:bat064.
112. Comeau DC, Batista-Navarro RT, Dai HJ, et al. BioC interoperability track overview. Database (Oxford) 2014; 2014.
113. Lu Z, Hirschman L. Biocuration workflows and text mining: overview of the BioCreative 2012 Workshop Track II. Database (Oxford) 2012;2012:bas043.
114. Cejuela JM, McQuilton P, Ponting L, et al. tagtog: interactive and text-mining-assisted annotation of gene mentions in PLOS full-text articles. Database (Oxford) 2014; 2014:bau033.
115. Lu Z. Text Mining in GeneRIFs. PhD thesis, University of Colorado Denver, 2007.
116. Wei CH. The Recognition and Normalization of Biomedical and Biological Concepts. PhD thesis, National Cheng Kung University, 2012.
117. Arighi CN, Roberts PM, Agarwal S, et al. BioCreative III interactive task: an overview. BMC Bioinformatics 2011;12 (Suppl 8):S4.
118. Arighi CN, Carterette B, Cohen KB, et al. An overview of the BioCreative 2012 Workshop Track III: interactive text mining task. Database (Oxford) 2013;2013: bas056.
119. Wiegers TC, Davis AP, Mattingly CJ. Web services-based text-mining demonstrates broad impacts for interoperability and process simplification. Database (Oxford) 2014;2014: bau050.
120. Khare R, Wei CH, MaoY, et al. tmBioC: improving interoperability of text-mining tools with BioC. Database (Oxford) 2014;2014.