A corpus of full-text journal articles is a robust evaluation tool for revealing differences in performance of biomedical natural language processing tools

BMC Bioinformatics

Volumn 13, Issue 1, 2012, Pages

  Verspoor, Karin ^a   Cohen, Kevin B ^a,b   Lanfranchi, Arrick ^b   Warner, Colin ^c   Johnson, Helen L ^a   Roeder, Christophe ^a   Choi, Jinho D ^b   Funk, Christopher ^a   Malenkiy, Yuriy ^a   Eckert, Miriam ^b   Xue, Nianwen ^c   Baumgartner Jr, William A ^a   Bada, Michael ^a   Palmer, Martha ^b   Hunter, Lawrence E ^a  

^a UNIVERSITY OF COLORADO SCHOOL OF MEDICINE   (United States)

^b UNIVERSITY OF COLORADO   (United States)

^c BRANDEIS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EVALUATION TOOL; JOURNAL ARTICLES; LINGUISTIC ANNOTATIONS; NAMED ENTITY RECOGNITION; NATURAL LANGUAGE PROCESSING; NATURAL LANGUAGE PROCESSING TOOLS; POOR PERFORMANCE; RULE SET; SYNTACTIC PARSING; TOKENIZATION; TRAINABLE SYSTEM;

DATA MINING; NATURAL LANGUAGE PROCESSING SYSTEMS; PUBLISHING;

LINGUISTICS;

ARTICLE; COMPUTER PROGRAM; DATA MINING; METHODOLOGY; NATURAL LANGUAGE PROCESSING;

DATA MINING; NATURAL LANGUAGE PROCESSING; SOFTWARE;

EID: 84865060319     PISSN: None     EISSN: 14712105     Source Type: Journal    
DOI: 10.1186/1471-2105-13-207     Document Type: Article

References (73)

1. Cohen KB, Hunter L. Natural language processing and systems biology. Springer; 2004:147-174.
2. Krallinger M, Leitner F, Valencia A. Assessment of the second BioCreative PPI task: automatic extraction of protein-protein interactions. Proceedings of the Second BioCreative Challenge Evaluation Workshop 2007,
3. Raychaudhuri S, Plenge RM, Rossin EJ, Ng ACY, Purcell SM, Sklar P, Scolnick EM, Xavier RJ, Altshuler D, Daly MJ. Identifying relationships among genomic disease regions: predicting genes at pathogenic SNP associations and rare deletions. PLoS Genet 2009, 5(6):e1000534. 10.1371/journal.pgen.1000534, 2694358, 19557189.
4. Cohen KB, Johnson HL, Verspoor K, Roeder C, Hunter L. The structural and content aspects of abstracts versus bodies of full text journal articles are different. BMC Bioinf 2010, 11(492). [doi:10.1186/1471-2105-11-492].
5. Clegg A, Shepherd A. Benchmarking natural-language parsers for biological applications using dependency graphs. BMC Bioinf 2007, 8(24). [doi:10.1186/1471-2105-8-24].
6. Verspoor K, Cohen KB, Hunter L. The textual characteristics of traditional and Open Access scientific journals are similar. BMC Bioinf 2009, 10:183. [doi:10.1186/1471-2105-10-183].
7. Blaschke C, Valencia A. Can bibliographic pointers for known biological data be found automatically? Protein interactions as a case study. Comp Funct Genomics 2001, 2(4):196-206. 10.1002/cfg.91, 2447212, 18628915.
8. Shah PK, Perez-Iratxeta C, Bork P, Andrade MA. Information extraction from full text scientific articles: where are the keywords?. BMC Bioinf 2003, 4:20. [doi:10.1186/1471-2105-4-20].
9. Corney DP, Buxton BF, Langdon WB, Jones DT. BioRAT: extracting biological information from full-length papers. Bioinformatics 2004, 20(17):3206-3213. 10.1093/bioinformatics/bth386, 15231534.
10. Friedman C, Kra P, Yu H, Krauthammer M, Rzhetsky A. GENIES: a natural-language processing system for the extraction of molecular pathways from journal articles. Bioinformatics 2001, 17(Suppl 1):S74-S82. 10.1093/bioinformatics/17.suppl_1.S74, 11472995.
11. Tanabe L, Wilbur WJ. Tagging gene and protein names in biomedical text. Bioinformatics 2002, 18(8):1124-1132. 10.1093/bioinformatics/18.8.1124, 12176836.
12. Rzhetsky A, Iossifov I, Koike T, Krauthammer M, Kra P, Morris M, Yu H, Duboue PA, Weng W, Wilbur WJ, Hatzivassiloglou V, Friedman C. GeneWays: a system for extracting, analyzing, visualizing, and integrating molecular pathway data. J Biomed Inf 2004, 37:43-53.
13. Arighi C, Cohen K, Hirschman L, Krallinger M, Lu Z, Valencia A, Wilbur J, . Wu C (Eds): 2010, Wu C (Eds):.
14. Bada M, Eckert M, Evans D, Garcia K, Shipley K, Sitnikov D, Baumgartner Jr WA, Cohen KB, Verspoor K, Blake JA, Hunter LE. Concept Annotation in the CRAFT Corpus. [http://www.biomedcentral.com/1471-2105/13/161].
15. Ide NC, Loane RF, Demner-Fushman D. Essie: A concept-based search engine for structured biomedical text. J Am Med Inf Assoc 2007, 14:253-263.
16. Miyao Y, Sagae K, Saetre R, Matsuzaki T, Tsujii J. Evaluating contributions of natural language parsers to protein-protein interaction extraction. Bioinformatics 2009, 25(3):394-400. 10.1093/bioinformatics/btn631, 2639072, 19073593.
17. Kim JD, Ohta T, Pyysalo S, Kano Y, Tsujii J. Overview of BioNLP'09 shared task on event extraction. BioNLP 2009 Companion Volume: Shared Task on Entity Extraction 2009, 1-9.
18. Grover C, Lapata M, Lascarides A. A comparison of parsing techniques for the biomedical domain. Nat Language Eng 2003, 1:1-38.
19. Pyysalo S, Ginter F, Pahikkala T, Boberg J, Järvinen J, Salakoski T. Evaluation of two dependency parsers on biomedical corpus targeted at protein-protein interactions. Int J Med Inf 2006, 75(6):430-442.
20. Schneider G, Rinaldi F, Kaljurand K, Hess M. Steps towards a GENIA dependency treebank. Third Workshop on Treebanks and Linguistic Theories (TLT) 2004 2004, 137-149.
21. Ohta T, Tateisi Y, Kim JD, Mima H, ichi Tsujii J. The GENIA corpus: an annotated corpus in molecular biology. Proceedings of the Human Language Technology conference 2002,
22. Kim JD, Ohta T, Tateisi Y, Tsujii J. GENIA corpus-a semantically annotated corpus for bio-textmining. Bioinformatics 2003, 19(Suppl 1):180-182.
23. Cohen KB, Fox L, Ogren PV, Hunter L. Corpus design for biomedical natural language processing. Proceedings of the ACL-ISMB workshop on linking biological literature, ontologies and databases 2005, 38-45. Association for Computational Linguistics, Detroit.
24. Cohen KB, Fox L, Ogren PV, Hunter L. Empirical data on corpus design and usage in biomedical natural language processing. AMIA 2005 symposium proceedings 2005, 156-160.
25. Tanabe L, Xie N, Thom L, Matten W, Wilbur W. GENETAG: a tagged corpus for gene/protein named entity recognition. BMC Bioinf 2005, 6(Suppl 1):S3.
26. Pyysalo S, Ginter F, Heimonen J, Bjorne J, Boberg J, Jarvinen J, Salakoski T. BioInfer: a corpus for information extraction in the biomedical domain. BMC Bioinf 2007, 8:50.
27. Szarvas G, Vincze V, Farkas R, Csirik J. The BioScope corpus: annotation for negation, uncertainty and their scope in biomedical texts. Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing 2008, 38-45. Association for Computational Linguistics, Columbus, Ohio, [http://www.aclweb.org/anthology/W/W08/W08-0606].
28. Kim JD, Pyysalo S, Ohta T, Bossy R, Nguyen N, Tsujii J. Overview of BioNLP Shared Task 2011. Proceedings of BioNLP Shared Task 2011 Workshop 2011, 1-6. Association for Computational Linguistics, Portland, Oregon, USA, [http://www.aclweb.org/anthology/W11-1801].
29. Santorini B. Part-of-Speech Tagging Guidelines for the Penn Treebank Project, 3rd revision. 1990, [http://repository.upenn.edu/cis reports/570/].
30. Bies A, Ferguson M, Katz K, MacIntyre R. Bracketing Guidelines for Treebank II style 1995, [ftp://ftp.cis.upenn.edu/pub/treebank/doc/manual/root.ps.gz].
31. Mott J, Warner C, Bies A, Taylor A. Supplementary Guidelines for English Translation Treebank 2.0. 2009, [projects.ldc.upenn.edu/gale/task specifications/ettb guidelines.pdf].
32. Taylor A. Reconciliation of Differences Between Onto/WSJ and EXTB 2010,
33. Taylor A. Treebank 2a guidelines. 2006, [http://www-users.york.ac.uk/lang22/TB2a Guidelines.htm].
34. Warner C, Bies A, Brisson C, Mott J. Addendum to the Penn Treebank II Style Bracketing Guidelines: Biomedical Treebank Annotation 2004,
35. Hunter LE. The Processes of Life: An Introduction to Molecular Biology 2009, MIT Press, Cambridge, MA.
36. Kulick S, Bies A, Mott J. Using derivation trees for Treebank error detection. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies 2011, 693-698. Association for Computational Linguistics, Portland, Oregon, USA.
37. Eilbeck K, Lewis SE, Mungall CJ, Yandell M, Stein L, Durbin R, Ashburner M. The Sequence Ontology: a tool for the unification of genome annotations. Genome Biol 2005, 6(5):r44. [doi:10.1186/gb-2005-6-5-r44], 10.1186/gb-2005-6-5-r44, 1175956, 15892872.
38. Maglott D, Ostell J, Pruitt KD, Tatusova T. Entrez Gene: gene-centered information at NCBI. Nucleic Acids Res 2011, 39(Suppl 1):D52-D57. [http://nar.oxfordjournals.org/content/39/suppl 1/D52.abstract], 3013746, 21115458.
39. Sayers EW, Barrett T, Benson DA, Bolton E, Bryant SH, Canese K, Chetvernin V, Church DM, DiCuccio M, Federhen S, Feolo M, Fingerman IM, Geer LY, Helmberg W, Kapustin Y, Landsman D, Lipman DJ, Lu Z, Madden TL, Madej T, Maglott DR, Marchler-Bauer A, Miller V, Mizrachi I, Ostell J, Panchenko A, Phan L, Pruitt KD, Schuler GD, Sequeira E, Sherry ST, Shumway M, Sirotkin K, Slotta D, Souvorov A, Starchenko G, Tatusova TA, Wagner L, Wang Y, Wilbur WJ, Yaschenko E, Ye J. Database resources of the National Center for Biotechnology Information. Nucleic Acids Res 2011, 39(Suppl 1):D38-D51. [http://nar.oxfordjournals.org/content/39/suppl 1/D38.abstract], 3013733, 21097890.
40. Smith L, Tanabe L, Johnson nee Ando R, Kuo CJ, Chung IF, Hsu CN, Lin YS, Klinger R, Friedrich C, Ganchev K, Torii M, Liu H, Haddow B, Struble C, Povinelli R, Vlachos A, Baumgartner Jr W, Hunter L, Carpenter B, Tsai RTH, Dai HJ, Liu F, Chen Y, Sun C, Katrenko S, Adriaans P, Blaschke C, Perez RT, Neves M, Nakov P, Divoli A, Mana M, Mata-Vazquez J, Wilber J. Overview of BioCreative II gene mention recognition. Genome Biol 2008, 9(Suppl 2):S2. 10.1186/gb-2008-9-s2-s2, 2559986, 18834493.
41. Morgan AA, Cohen KB, Hirschman L, et al. Overview of BioCreative II gene normalization. Genome Biol 2008, 9(Suppl 2):S3. 10.1186/gb-2008-9-s2-s3, 2559987, 18834494.
42. Yeh A, Morgan A, Colosimo M, Hirschman L. BioCreative task 1A: gene mention finding evaluation. BMC Bioinf 2005, 6(Suppl 1).
43. Kim JD, Ohta T, Tsuruoka Y, Tateisi Y, Collier N. Introduction to the bio-entity recognition task at JNLPBA. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA) 2004, 70-75.
44. Wang Y, Kim JD, Saetre R, Pyysalo S, Ohta T, Tsujii J. Improving the inter-corpora compatibility for protein annotations. J Bioinf Comput Biol 2010, 8(5):901-916.
45. Klein D, Manning C. Fast exact inference with a factored model for natural language parsing. Adv Neural Inf Process Syst 2003, 15:3-10.
46. McClosky D, Charniak E, Johnson M. Effective self-training for parsing. Proceedings of Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics 2006, 152-159. Association for Computational Linguistics, Brooklyn, NY.
47. Cer D, de Marneffe MC, Jurafsky D, Manning CD. Parsing to Stanford Dependencies: Trade-offs between speed and accuracy. Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC'10) 2010,
48. Banko M, Brill E. Mitigating the paucity-of-data problem: exploring the effect of training corpus size on classifier performance for natural language processing. Proceedings of the First International Conference on Human Language Technology Research 2001,
49. Cohen KB, Lanfranchi A, Corvey W, Jr WAB, Roeder C, Ogren PV, Palmer M, Hunter LE. Annotation of all coreference in biomedical text: Guideline selection and adaptation. BioTxtM 2010: 2nd Workshop on Building and Evaluating Resources for Biomedical Text Mining 2010, 37-41.
50. Cohen KB, Christiansen T, Hunter LE. Parenthetically speaking: Classifying the contents of parentheses for text mining. Proceedings of the American Medical Informatics Association Fall Symposium 2011,
51. PubMed Central Open Access Collection. [http://www.ncbi.nlm.nih.gov/pmc/about/openftlist.html].
52. Sekine S, Collins MJ. The Evalb software. 1997, [http://cs.nyu.edu/cs/projects/proteus/evalb].
53. Ogren P, Bethard S. Building Test Suites for UIMA Components. Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing (SETQA-NLP 2009) 2009, 1-4. Association for Computational Linguistics, Boulder, Colorado.
54. Ogren P. Knowtator: a Protege plugin for annotated corpus construction. HLT-NAACL 2006 Companion Volume 2006a,
55. Noy N, Fridman N, Fergerson Ray W, Musen M, Mark A. The knowledge model of Protege-2000: Combining interoperability and flexibility. 2000, 17-32. Springer-Verlag, London, UK, UK.
56. Settles B. ABNER: an open source tool for automatically tagging genes, proteins, and other entity names in text. Bioinformatics 2005, 21(14):3191-3192. [http://dx.doi.org/doi:10.1093/bioinformatics/bti475], 10.1093/bioinformatics/bti475, 15860559.
57. Leaman R, Gonzalez G. BANNER: An executable survey of advances in biomedical named entity recognition. Pac Symp Biocomput 2008,
58. Carpenter B. Phrasal Queries with LingPipe and Lucene. 2004,
59. Verspoor K, Baumgartner W, Roeder C, Hunter L. Abstracting the Types away from a UIMA Type System. Von der Form aur Bedeutung: Text automatisch verarbeiten / From Form to Meaning: Processing Texts Automatically 2009, 249-256. Chiarcos C, Eckhart de Castilho R, Stede M.
60. Baldridge J, Morton T, Bierner G. The opennlp maximum entropy package. 2002, Technical report, SourceForge.
61. Ferrucci D, Lally A. Building an example application with the unstructured information management architecture. IBM Syst J 2004, 43(3):455-475.
62. Marcus MP, Marcinkiewicz MA, Santorini B. Building a large annotated corpus of English: the Penn Treebank. Comput Linguistics 1993, 19(2):313-330.
63. Apache UIMA ConceptMapper Annotator Documentation. 2009, Tech. rep., The Apache Software Foundation.
64. Petrov S, Barrett L, Thibaux R, Klein D. Learning accurate, compact, and interpretable tree annotation. Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics 2006, 433-440. Association for Computational Linguistics, Sydney.
65. Bikel D. A distributional analysis of a lexicalized statistical parsing model. Proc. of EMNLP, Volume 4 2004, 182-189.
66. Lease M, Charniak E. Parsing biomedical literature. Second International Joint Conference on Natural Language Processing 2005, 58-69.
67. Ninomiya T, Matsuzaki T, Tsuruoka Y, Miyao Y, Tsujii J. Extremely lexicalized models for accurate and fast HPSG parsing. Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing 2006, 155-163. Association for Computational Linguistics, Sydney.
68. McClosky D. Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing. 2009, PhD thesis. Brown University, Department of Computer Science.
69. Nivre J. Non-projective dependency parsing in expected linear time. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (ACL-IJCNLP'09) 2009, 351-359.
70. McDonald R, Pereira F, Ribarov K, Hajic J. Non-projective dependency parsing using spanning tree algorithms. Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT-EMNLP'05) 2005, 523-530.
71. Choi JD, Nicolov N. K-best, Locally Pruned, Transition-based Dependency Parsing Using Robust Risk Minimization. Collections of Recent Advances in Natural Language Processing V 2009, 205-216. John Benjamins.
72. Choi JD, Palmer M. Robust Constituent-to-Dependency Conversion for English. Proceedings of the 9th International Workshop on Treebanks and Linguistic Theories (TLT'9) 2010, 55-66.
73. Johansson R, Nugues P. Extended constituent-to-dependency conversion for English. Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA'07) 2007, Tartu, [http://dspace.utlib.ee/dspace/handle/10062/2560].