Empirical comparison of evaluation methods for unsupervised learning of morphology

TAL Traitement Automatique des Langues

Volumn 52, Issue 2, 2011, Pages 49-90

  Virpioja, Sami ^a   Turunen, Ville T ^a   Spiegler, Sebastian ^b   Kohonen, Oskar ^a   Kurimo, Mikko ^a  

^a AALTO UNIVERSITY   (Finland)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

Evaluation; Morphology; Unsupervised learning

Indexed keywords

EID: 84861519211     PISSN: 12489433     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Review

References (88)

1. Agirre E., Di Nunzio G. M., Ferro N., Mandl T., Peters C., "CLEF 2008: Ad Hoc Track Overview", Working Notes for the CLEF 2008 Workshop, September, 2008.
2. Alkula R., "From Plain Character Strings to Meaningful Words: Producing Better Full Text Databases for Inflectional and Compounding Languages with Morphological Analysis Software", Information Retrieval, vol. 4, p. 195-208, 2001. 10.1023/A:1011942104443.
3. Alpaydin E., Introduction to Machine Learning, The MIT Press, Cambridge, MA, USA, 2004.
4. Baayen R. H., "Storage and Computation in the Mental Lexicon", in G. Jarema, G. Libben (eds), The Mental Lexicon: Core Perspectives, Elsevier, p. 81-104, 2007.
5. Baayen R. H., Piepenbrock R., Gulikers L., The CELEX Lexical Database (CD-ROM), Linguistic Data Consortium, University of Pennsylvania, Philadelphia, PA, 1995. http://wave.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC96L14.
6. Baroni M., Matiasek J., Trost H., "Unsupervised Discovery of Morphologically Related Words Based on Orthographic and Semantic Similarity", Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, Association for Computational Linguistics, Morristown, NJ, USA, p. 48-57, July, 2002.
7. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers, vol. 5152 of Lecture Notes in Computer Science, Springer, p. 873-880, 2008.
8. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 598-608, 2010.
9. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers, vol. 5152 of Lecture Notes in Computer Science, Springer, p. 881-891, 2008.
10. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 641-648, 2010.
11. Chan E., "Learning Probabilistic Paradigms for Morphology in a Latent Class Model", Proceedings of the Eighth Meeting of the ACL Special Interest Group on Computational Phonology and Morphology at HLT-NAACL 2006, Association for Computational Linguistics, New York City, USA, p. 69-78, June, 2006.
12. Chan E., Structures and Distributions in Morphology Learning, PhD thesis, Department of Computer and Information Science, University of Pennsylvania, 2008.
13. Christodoulopoulos C., Goldwater S., Steedman M., "Two Decades of Unsupervised POS Induction: How Far Have We Come?", Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Cambridge, MA, p. 575-584, October, 2010.
14. st Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Sapporo, Japan, p. 280-287, July, 2003.
15. Creutz M., Lagus K., "Unsupervised Discovery of Morphemes", Proceedings of the Workshop on Morphological and Phonological Learning of ACL-02, p. 21-30, 2002.
16. Creutz M., Lagus K., "Inducing the Morphological Lexicon of a Natural Language From Unannotated Text", Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR'05), Espoo, Finland, p. 106-113, 2005a.
17. Creutz M., Lagus K., "Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0", Technical Report no. A81, Publications in Computer and Information Science, Helsinki University of Technology, 2005b.
18. Creutz M., Lagus K., "Unsupervised Models for Morpheme Segmentation and Morphology Learning", ACM Transactions on Speech and Language Processing, January, 2007.
19. Creutz M., Lindén K., "Morpheme Segmentation Gold Standards for Finnish and English", Technical Report no. A77, Publications in Computer and Information Science, Helsinki University of Technology, 2004.
20. Dasgupta S., Ng V., "High-Performance, Language-Independent Morphological Segmentation", Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, Association for Computational Linguistics, Rochester, New York, p. 155-163, April, 2007.
21. Gao J., Johnson M., "A Comparison of Bayesian Estimators for Unsupervised Hidden Markov Model POS Taggers", Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Honolulu, Hawaii, p. 344-352, October, 2008.
22. Goldsmith J., "Unsupervised Learning of the Morphology of a Natural Language", Computational Linguistics, vol. 27, no. 2, p. 153-189, 2001.
23. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 633-640, 2010a.
24. Golénia B., Spiegler S., Ray O., Flach P., "Morphological Analysis Using Morpheme Graph and Mixed-Integer Computation of Stable Models", in M. Kurimo, S. Virpioja, V. T. Turunen (eds), Proceedings of the Morpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 25-29, September, 2010b. Technical Report TKK-ICS-R37. Extended abstract.
25. Hafer M. A., Weiss S. F., "Word Segmentation by Letter Successor Varieties", Information Storage and Retrieval, vol. 10, no. 11-12, p. 371-385, 1974.
26. Hammarström H., Borin L., "Unsupervised Learning of Morphology", Computational Linguistics, vol. 37, no. 2, p. 309-350, June, 2011.
27. Harris Z. S., "From Phoneme to Morpheme", Language, vol. 31, no. 2, p. 190-222, 1955.
28. Reprinted 1970 in Papers in Structural and Transformational Linguistics, Reidel Publishing Company, Dordrecht, Holland.
29. Hirsimäki T., Creutz M., Siivola V., Kurimo M., Virpioja S., Pylkkönen J., "Unlimited Vocabulary Speech Recognition with Morph Language Models Applied to Finnish", Computer, Speech and Language, vol. 20, no. 4, p. 515-541, 2006.
30. Hirsimäki T., Pylkkönen J., KurimoM., "Importance of High-Order N-GramModels inMorph-Based Speech Recognition", IEEE Transactions on Audio, Speech and Language Processing, vol. 17, no. 4, p. 724-732, May, 2009.
31. nd edn, Prentice Hall, New Jersey, USA, 2008.
32. Karp R. M., "Reducibility Among Combinatorial Problems", in R. E. Miller, J. W. Thatcher (eds), Complexity of Computer Computations, New York, Plenum, p. 85-103, 1972.
33. Keshava S., Pitler E., "A Simpler, Intuitive Approach to Morpheme Induction", Proceedings of the PASCAL Challenge Workshop on Unsupervised Segmentation of Words into Morphemes, PASCAL European Network of Excellence, Venice, Italy, 2006.
34. th Machine Translation Summit, Phuket, Thailand, p. 79-86, 2005.
35. Koehn P., Hoang H., Birch A., Callison-Burch C., Federico M., Bertoldi N., Cowan B., Shen W., Moran C., Zens R., Dyer C., Bojar O., Constantin A., Herbst E., "Moses: Open Source Toolkit for Statistical Machine Translation", Annual Meeting of ACL, demonstration session, Czech Republic, June, 2007.
36. th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers, vol. 5706 of Lecture Notes in Computer Science, Springer, 2009.
37. th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, Association for Computational Linguistics, Uppsala, Sweden, p. 78-86, July, 2010a.
38. Kohonen O., Virpioja S., Leppänen L., Lagus K., "Semi- Supervised Extensions to Morfessor Baseline", in M. Kurimo, S. Virpioja, V. T. Turunen (eds), Proceedings of the Morpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 30-34, September, 2010b. Technical Report TKK-ICS-R37. Extended abstract.
39. Koskenniemi K., Two-Level Morphology: A General Computational Model for Word-Form Recognition and Production, PhD thesis, University of Helsinki, 1983.
40. Kuhn H. W., "The Hungarian Method for the Assignment Problem", Naval Research Logistics Quarterly, vol. 2, p. 83-97, 1955.
41. Kumar S., Byrne W., "Minimum Bayes-Risk Decoding for Statistical Machine Translation", Proceedings of Human Language Technologies: The 2004 Annual Conference of the North American Chapter of the Association for Computational Linguistics, p. 169-176, 2004.
42. Kurimo M., Creutz M., Lagus K., "Unsupervised Segmentation of Words into Morphemes - Challenge 2005, An Introduction and Evaluation Report", Proceedings of the PASCAL Challenge Workshop on Unsupervised Segmentation of Words into Morphemes, PASCAL European Network of Excellence, Venice, Italy, 2006.
43. Kurimo M., Creutz M., Turunen V., "Unsupervised Morpheme Analysis Evaluation by IR experiments - Morpho Challenge 2007", in A. Nardi, C. Peters (eds), Working Notes for the CLEF 2007 Workshop, CLEF, September, 2007. Invited paper.
44. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers, vol. 5152 of Lecture Notes in Computer Science, Springer, p. 864-873, 2008.
45. th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers, vol. 5706 of Lecture Notes in Computer Science, Springer, 2009.
46. th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, Association for Computational Linguistics, Uppsala, Sweden, p. 87-95, July, 2010a.
47. Kurimo M., Virpioja S., Turunen V. T., "Overview and Results of Morpho Challenge 2010", Proceedings of the Morpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 7-24, September, 2010b. Technical Report TKK-ICS-R37.
48. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 578-597, 2010c.
49. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 617-624, 2010.
50. Lignos C., "Learning from Unseen Data", in M. Kurimo, S. Virpioja, V. T. Turunen (eds), Proceedings of the Morpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 35-38, September, 2010. Technical Report TKK-ICS-R37. Extended abstract.
51. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 658-665, 2010a.
52. Lignos C., Chan E., Yang C., Marcus M. P., "Evidence for a Morphological Acquisition Model from Development Data", Proceedings of BUCLD 34, Cascadilla Press, 2010b.
53. Lim H. S., Nam K., Hwang Y., "A Computational Model of Korean Mental Lexicon", in O. Gervasi, M. Gavrilova, V. Kumar, A. Laganà, H. Lee, Y.Mun, D. Taniar, C. Tan (eds), Computational Science and Its Applications-ICCSA 2005, vol. 3480 of Lecture Notes in Computer Science, Springer Berlin / Heidelberg, p. 17-26, 2005.
54. nd edn, Cambridge University Press, 1991.
55. McNamee P., Mayfield J., "N-Gram Morphemes for Retrieval", Working Notes for the CLEF 2007 Workshop, Budapest, Hungary, September, 2007.
56. Meila M., "Comparing Clusterings by the Variation of Information", in B. Schölkopf, M. K. Warmuth (eds), Learning Theory and Kernel Machines, vol. 2777 of Lecture Notes in Computer Science, Springer Berlin / Heidelberg, p. 173-187, 2003.
57. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers, vol. 5152 of Lecture Notes in Computer Science, Springer, p. 900-907, 2008.
58. th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers, vol. 5706 of Lecture Notes in Computer Science, Springer, p. 967-974, 2009.
59. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 649-657, 2010.
60. Munkres J., "Algorithms for the Assignment and Transportation Problems", Journal of the Society for Industrial and Applied Mathematics, vol. 5, no. 1, p. 32-38, 1957.
61. Nicolas L., Farré J., Molinero M. A., "Unsupervised Learning of Concatenative Morphology Based on Frequency-Related Form Occurrence", in M. Kurimo, S. Virpioja, V. T. Turunen (eds), Proceedings of the Morpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 39-43, September, 2010. Technical Report TKK-ICS-R37. Extended abstract.
62. Nießen S., Ney H., "Statistical Machine Translation With Scarce Resources Using Morpho-Syntactic Information", Computational Linguistics, vol. 30, no. 2, p. 181-204, 2004.
63. Oflazer K., El-Kahlout İ. D., "Exploring Different Representational Units in English-to-Turkish Statistical Machine Translation", Proceedings of the Statistical Machine Translation Workshop at ACL 2007, Association for Computational Linguistics, Prague, Czech Republic, p. 25-32, June, 2007.
64. Ogilvie P., Callan J., "Experiments Using the Lemur Toolkit", Proc. TREC '01, National Institute of Standards and Technology, special publication, Gaithersburg, Maryland, USA, p. 103-108, 2002.
65. th Annual Meeting on Association for Computational Linguistics (ACL'02), Association for Computational Linguistics, Morristown, NJ, USA, p. 311-318, 2002.
66. Pinker S., Ullman M. T., "The Past and Future of the Past Tense", Trends in Cognitive Sciences, vol. 6, p. 456-463, November, 2002.
67. Pirkola A., "Morphological Typology of Languages for IR", Journal of Documentation, vol. 57, no. 3, p. 330-348, 2001.
68. Poon H., Cherry C., Toutanova K., "Unsupervised Morphological Segmentation with Log-Linear Models", Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT 2009), Association for Computational Linguistics, p. 209-217, 2009.
69. Rosenberg A., Hirschberg J., "V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure", Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLPCoNLL), Association for Computational Linguistics, Prague, Czech Republic, p. 410-420, June, 2007.
70. th Conference on Computational Natural Language Learning, Association for Computational Linguistics, Morristown, NJ, USA, p. 67-72, 2000.
71. Schone P., Jurafsky D., "Knowledge-Free Induction of Inflectional Morphologies", Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, Pittsburgh, USA, 2001.
72. Shalonova K., Golenia B., Flach P., "Towards Learning Morphology for Under-Resourced Fusional and Agglutinating Languages", IEEE Transactions on Audio, Speech, and Language Processing, vol. 17, no. 5, p. 956-965, July, 2009.
73. Snover M. G., Jarosz G. E., Brent M. R., "Unsupervised Learning of Morphology Using a Novel Directed Search Algorithm: Taking the First Step", Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, Association for Computational Linguistics, p. 11-20, July, 2002.
74. Snyder B., Barzilay R., "Unsupervised Multilingual Learning for Morphological Segmentation", Proceedings of ACL-08: HLT, Association for Computational Linguistics, Columbus, Ohio, p. 737-745, June, 2008.
75. Spiegler S., Machine Learning for the Analysis of Morphologically Complex Languages, PhD thesis, Merchant Venturers School of Engineering, University of Bristol, April, 2011.
76. Spiegler S., Golénia B., Flach P. A., "DEAP: Deductive-Abductive Parsing for Morphological Analysis", in M. Kurimo, S. Virpioja, V. T. Turunen (eds), Proceedings of theMorpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 44-48, September, 2010a. Technical Report TKK-ICS-R37. Extended abstract.
77. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 625-632, 2010b.
78. Spiegler S., Golénia B., Flach P. A., "Word Decomposition With the Promodes Algorithm Family Bootstrapped on a Small Labelled Dataset", in M. Kurimo, S. Virpioja, V. T. Turunen (eds), Proceedings of the Morpho Challenge 2010 Workshop, Aalto University School of Science and Technology, Department of Information and Computer Science, Espoo, Finland, p. 49-52, September, 2010c. Technical Report TKK-ICS-R37. Extended abstract.
79. rd International Conference on Computational Linguistics (COLING), August, 2010.
80. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 666-673, 2010.
81. th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers, vol. 6241 of Lecture Notes in Computer Science, Springer, p. 609-616, 2010.
82. Virpioja S., Lehtonen M., Hultén A., Salmelin R., Lagus K., "Predicting Reaction Times in Word Recognition by Unsupervised Learning of Morphology", Proceedings of International Conference on Artificial Neural Networks (ICANN 2011), vol. 6791 of Lecture Notes in Computer Science, Springer Berlin / Heidelberg, Espoo, Finland, p. 275-282, 2011.
83. Virpioja S., Väyrynen J. J., Creutz M., Sadeniemi M., "Morphology-Aware Statistical Machine Translation Based on Morphs Induced in an Unsupervised Manner", Proceedings of the Machine Translation Summit XI, Copenhagen, Denmark, p. 491-498, September, 2007.
84. th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2006), Association for Computational Linguistics, Trento, Italy, p. 41-48, 2006.
85. th Meeting of the ACL, Association for Computational Linguistics, p. 207-216, 2000.
86. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers, vol. 5152 of Lecture Notes in Computer Science, Springer, p. 892-899, 2008.
87. th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers, vol. 5706 of Lecture Notes in Computer Science, Springer, p. 983-990, 2009.
88. Zhu X., "Semi-Supervised Learning Literature Survey", Technical Report no. 1530, Computer Sciences, University of Wisconsin-Madison, 2005.