Building a biomedical tokenizer using the token lattice design pattern and the adapted Viterbi algorithm

BMC Bioinformatics

Volumn 12, Issue SUPPL. 3, 2011, Pages

  Barrett, Neil ^a   Weber Jahnke, Jens ^a  

^a UNIVERSITY OF VICTORIA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

BIOMEDICAL DOMAIN; CLASSIFICATION APPROACH; CLASSIFICATION TASKS; DESIGN PATTERNS; LANGUAGE PROCESSING; MACHINE LEARNING APPROACHES; REGULAR EXPRESSIONS; RULE-BASED TECHNIQUES;

PATTERN MATCHING;

LEARNING SYSTEMS;

ALGORITHM; ARTICLE; INFORMATION RETRIEVAL; METHODOLOGY; NATURAL LANGUAGE PROCESSING;

ALGORITHMS; INFORMATION STORAGE AND RETRIEVAL; NATURAL LANGUAGE PROCESSING;

EID: 79958107612     PISSN: None     EISSN: 14712105     Source Type: Journal    
DOI: 10.1186/1471-2105-12-S3-S1     Document Type: Article

References (28)

1. Jurafsky D, Martin JH. Speech and Language Processing. 2009, Prentice Hall.
2. Webster JJ, Kit C. Tokenization as the initial phase in NLP. Proceedings of the 14th conference on Computational linguistics 1992, 1106-1110. Morristown, NJ, USA: Association for Computational Linguistics.
3. Attia M. Arabic Tokenization System. Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources 2007, 65-72. Prague, Czech Republic: Association for Computational Linguistics., http://www.aclweb.org/anthology/W/W07/W07-0809
4. Jiang J, Zhai C. An empirical study of tokenization strategies for biomedical information retrieval. Inf. Retr. 2007, 10(4-5):341-363.
5. Tomanek K, Wermter J, Hahn U. Sentence and Token Splitting Based on Conditional Random Fields. PACLING 2007 - Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics, Melbourne, Australia, September 19-21, 2007 2007, 49-57. Melbourne: Pacific Association for Computational Linguistics.
6. Arens R. A preliminary look into the use of named entity information for bioscience text tokenization. Proceedings of the Student Research Workshop at HLT-NAACL 2004 on XX 2004, 37-42. HLT-NAACL '04, Morristown, NJ, USA: Association for Computational Linguistics., http://portal.acm.org/citation.cfm?id=1614038.1614045
7. He Y, Kayaalp M. A Comparison of 13 Tokenizers on MEDLINE. Tech. Rep. LHNCBC-TR-2006-003 2006, The Lister Hill National Center for Biomedical Communications.
8. Grana J, Barcala FM, Ferro JV. Formal Methods of Tokenization for Part-of-Speech Tagging. CICLing '02: Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing 2002, 240-249. London, UK: Springer-Verlag.
9. Lavelli A, Califf ME, Ciravegna F, Freitag D, Giuliano C, Kushmerick N, Romano L, Ireson N. Evaluation of machine learning-based information extraction algorithms: criticisms and recommendations. Language Resources and Evaluation 2008, 42(2):361-393.
10. Trieschnigg D, Kraaij W, de Jong F. The influence of basic tokenization on biomedical document retrieval. SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval 2007, 803-804. New York, NY, USA: ACM.
11. Hassler M, Fliedl G. Text preparation through extended tokenization. Data Mining VII: Data, Text and Web Mining and Their Business Applications 2006, 37:13-21. WIT Press/Computational Mechanics Publications, Zanasi, A and Brebbia, CA and Ebecken, NFF.
12. Grana J, Alonso MA, Vilares M. A Common Solution for Tokenization and Part-of-Speech Tagging: One-Pass Viterbi Algorithm vs. Iterative Approaches. Text, Speech and Dialogue, volume 2448 of Lecture Notes in Computer Science 2002, 3-10. Springer-Verlag.
13. Wrenn JO, Stetson PD, Johnson SB. An unsupervised machine learning approach to segmentation of clinician-entered free text. AMIA Annu Symp Proc 2007, 811-5. 2655800, 18693949.
14. McDonald R, Crammer K, Pereira F. Flexible text segmentation with structured multilabel classification. HLT '05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing 2005, 987-994. Morristown, NJ, USA: Association for Computational Linguistics.
15. Buschmann F, Henney K, Schmidt DC. Pattern Oriented Software Architecture: On Patterns and Pattern Languages. 2007, John Wiley & Sons.
16. Kiss T, Strunk J. Unsupervised multilingual sentence boundary detection. Computational Linguistics 2006, 32(4):485-525.
17. Marcus MP, Marcinkiewicz MA, Santorini B. Building a large annotated corpus of English: the penn treebank. Comput. Linguist. 1993, 19(2):313-330.
18. Brants T. TnT: a statistical part-of-speech tagger. Proceedings of the sixth conference on Applied natural language processing 2000, 224-231. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc..
19. Davey BA, Priestley HA. Introduction to Lattices and Order. 2002, Cambridge University Press, 2.
20. Guo J. Critical tokenization and its properties. Comput. Linguist. 1997, 23(4):569-596.
21. Ma Y, Way A. Bilingually motivated domain-adapted word segmentation for statistical machine translation. EACL '09: Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics 2009, 549-557. Morristown, NJ, USA: Association for Computational Linguistics.
22. The International Health Terminology Standards Development Organisation SNOMED Clinical Terms - User Guide. 2009, The International Health Terminology Standards Development Organisation.
23. Cohen J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 1960, 20:37-46.
24. 2011, http://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/textTools/current/index.html
25. Smith L, Rindflesch T, Wilbur WJ. MedPost: a part-of-speech tagger for bioMedical text. Bioinformatics 2004, 20(14):2320-1. 10.1093/bioinformatics/bth227, 15073016.
26. Loper E, Bird S. NLTK: the Natural Language Toolkit. Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics 2002, 63-70. Morristown, NJ, USA: Association for Computational Linguistics.
27. Mendenhall W, Sincich T. Statistics for the Engineering and Computer Sciences. 1984, Dellen Publishing Company.
28. Parnas DL. On the criteria to be used in decomposing systems into modules. Commun. ACM 1972, 15(12):1053-1058.