Tokenization: Returning to a long solved problem: A survey, contrastive experiment, recommendations, and toolkit

50th Annual Meeting of the Association for Computational Linguistics, ACL 2012 - Proceedings of the Conference

Volumn 2, Issue , 2012, Pages 378-382

  Dridan, Rebecca ^a   Oepen, Stephan ^a  

^a UNIVERSITY OF OSLO   (Norway)

Author keywords

[No Author keywords available]

Indexed keywords

DOMAIN SPECIFIC; RULE BASED; TOKENIZATION; TREEBANKS;

COMPUTATIONAL LINGUISTICS;

FORESTRY;

EID: 84875542948     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (9)

1. Charniak, E., & Johnson, M. (2005). Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (pp. 173-180). Ann Arbor, USA.
2. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006). Ontonotes. The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (pp. 57-60). New York City, USA.
3. Kaplan, R. M. (2005). A method for tokenizing text. Festschrift for Kimmo Koskenniemi on his 60th birthday. In A. Arppe, L. Carlson, K. Lindén, J. Piitulainen, M. Suominen, M. Vainio, H. Westerlund, & A. Yli-Jyrä (Eds.), Inquiries into words, constraints and contexts (pp. 55 - 64). Stanford, CA: CSLI Publications.
4. Levenshtein, V. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physice - Doklady, 10, 707-710.
5. Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English. The Penn Treebank. Computational Linguistics, 19, 313 - 330.
6. Øvrelid, L., Velldal, E., & Oepen, S. (2010). Syntactic scope resolution in uncertainty analysis. In Proceedings of the 23rd international conference on computational linguistics (pp. 1379 - 1387). Beijing, China.
7. Schäfer, U., Kiefer, B., Spurk, C., Steffen, J., & Wang, R. (2011). The ACL Anthology Searchbench. In Proceedings of the ACL-HLT 2011 system demonstrations (pp. 7-13). Portland, Oregon, USA.
8. Tateisi, Y., & Tsujii, J. (2006). GENIA annotation guidelines for tokenization and POS tagging (Technical Report # TR-NLP-UT-2006-4).Tokyo, Japan: Tsujii Lab, University of Tokyo.
9. Waldron, B., Copestake, A., Schäfer, U., & Kiefer, B. (2006). Preprocessing and tokenisation standards in DELPH-IN tools. In Proceedings of the 5th International Conference on Language Resources and Evaluation (pp. 2263 - 2268). Genoa, Italy.