Producing power-law distributions and damping word frequencies with two-stage language models

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 2335-2382

  Goldwater, Sharon ^a   Griffiths, Thomas L ^b   Johnson, Mark ^c  

^a Institute for Neural Computation   (United Kingdom)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c MACQUARIE UNIVERSITY   (Australia)

Author keywords

Language model; Nonparametric Bayes; Pitman Yor process; Unsupervised

Indexed keywords

BAYESIAN MODEL; DIRICHLET PROCESS; ESTIMATION PROCEDURES; FORMAL ANALYSIS; LANGUAGE MODEL; NATURAL LANGUAGES; NON-PARAMETRIC; PITMAN-YOR PROCESS; POWER LAW; POWER LAW DISTRIBUTION; PROBABILISTIC MODELS; STATISTICAL MODELS; TWO STAGE; UNSUPERVISED; WORD FREQUENCIES;

BAYESIAN NETWORKS; COMPUTATIONAL LINGUISTICS; DATA MINING; IMAGE SEGMENTATION; RANDOM PROCESSES;

FREQUENCY ESTIMATION;

EID: 80052252167     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (91)

1. Adam Albright and Bruce Hayes. Rules vs. analogy in English past tenses: a computational/experimental study. Cognition, 90:118-161, 2003.
2. David Aldous. Exchangeability and related topics. In École d'été de probabilités de Saint-Flour, XIII-1983, pages 1-198. Springer, Berlin, 1985.
3. Maria Alegre and Peter Gordon. Frequency effects and the representational status of regular inflections. Journal of Memory and Language, 40(1):41-61, 1999.
4. Charles Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of Statistics, 2:1152-1174, 1974.
5. Richard Arratia, A. D. Barbour, and Simon Tavaré. Poisson process approximations for the Ewens sampling formula. Annals of Applied Probability, 2:519-535, 1992.
6. Ricardo Baeza-Yates and Berthier Ribeiro-Neto. Modern Information Retrieval. ACM press, New York, 1999.
7. Matthew Beal, Zoubin Ghahramani, and Carl Rasmussen. The infinite hidden Markov model. In Advances in Neural Information Processing Systems 14, Cambridge, MA, 2002. MIT Press.
8. David Blackwell and James B. MacQueen. Ferguson distributions via Pólya urn schemes. Annals of Statistics, 1:353-355, 1973.
9. David Blei, Andrew Ng, and Michael Jordan. Latent Dirichlet allocation. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, Cambridge, MA, 2002. MIT Press.
10. David Blei, Andrew Ng, and Michael Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
11. David Blei, Thomas L. Griffiths, Michael Jordan, and Joshua B. Tenenbaum. Hierarchical topic models and the nested Chinese restaurant process. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing 16, Cambridge, MA, 2004. MIT Press.
12. Michael Brent. An efficient, probabilistically sound algorithm for segmentation and word discovery. Machine Learning, 34:71-105, 1999.
13. Roger Brown. A First Language: The Early Stages. Harvard University Press, Cambridge, MA, 1973.
14. Wray L. Buntine and Marcus Hutter. A Bayesian review of the Poisson-Dirichlet process. http://arxiv.org/abs/1007.0296, Jul 2010.
15. Joan Bybee. Morphology: a Study of Relation between Meaning and Form. Benjamins, Amsterdam, 1985.
16. Joan Bybee. Phonology and Language Use. Cambridge University press, Cambridge, UK, 2001.
17. Stanley F. Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Center for Research in Computing Technology, Harvard University, 1998.
18. Shay B. Cohen, David M. Blei, and Noah A. Smith. Variational inference for adaptor grammars. In Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 564-572, Los Angeles, California, 2010.
19. Trevor Cohn, Sharon Goldwater, and Phil Blunsom. Inducing compact but accurate tree-substitution grammars. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 548-556, Boulder, Colorado, 2009.
20. Trevor Cohn, Sharon Goldwater, and Phil Blunsom. Inducing tree substitution grammars. Journal of Machine Learning Research, 11:3053-3096, Nov. 2010.
21. Michael Collins, Lance Ramshaw, Jan Hajič, and Christoph Tillmann. A statistical parser for Czech. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 505-512, College Park, Maryland, USA, 1999.
22. Brooke Cowan and Michael Collins. Morphology and reranking for the statistical parsing of Spanish. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 795-802, Vancouver, 2005.
23. Philip Cowans. Probabilistic Document Modelling. PhD thesis, Cambridge University, 2006.
24. Mathias Creutz and Krista Lagus. Induction of a simple morphology for highly-inflecting languages. In Proceedings of the Seventh Meeting of the ACL Special Interest Group in Computational Phonology (SIGPHON), pages 43-51, Barcelona, Spain, 2004.
25. Mathias Creutz and Krista Lagus. Inducing the morphological lexicon of a natural language from unannotated text. In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning, pages 51-59, Espoo, Finland, 2005.
26. Charles Elkan. Clustering documents with an exponential-family approximation of the Dirichlet compound multinomial distribution. In Proceedings of the 23rd International Conference on Machine Learning, pages 289-296, Pittsburgh, Pennsylvania, 2006.
27. Micha Elsner, Eugene Charniak, and Mark Johnson. Structured generative models for unsupervised named-entity clustering. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 164-172, Boulder, Colorado, 2009.
28. Michael D. Escobar and Mike West. Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90(430):577-588, 1995.
29. Thomas S. Ferguson. A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1:209-230, 1973.
30. Jenny Finkel, Trond Grenager, and Christopher D. Manning. The infinite tree. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 272-279, Prague, Czech Republic, 2007.
31. Walter R. Gilks, Sylvia Richardson, and David J. Spiegelhalter, editors. Markov Chain Monte Carlo in Practice. Chapman and Hall, Suffolk, 1996.
32. John Goldsmith. Unsupervised learning of the morphology of a natural language. Computational Linguistics, 27:153-198, 2001.
33. John Goldsmith. An algorithm for the unsupervised learning of morphology. Journal of Natural Language Engineering, 12:353-371, 2006. (Pubitemid 44773861)
34. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. Interpolating between types and tokens by estimating power-law generators. In Advances in Neural Information Processing Systems 18, Cambridge, MA, 2006a. MIT Press.
35. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. Contextual dependencies in unsuper-vised word segmentation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Syndney, Australia, 2006b.
36. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. A Bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112(1):21-54, 2009.
37. Joseph Greenberg. Language universals. In T. A. Sebok, editor, Current Trends in Linguistics III. Mouton, The Hague, 1966.
38. Thomas L. Griffiths. Power-law distributions and nonparametric Bayes. Unpublished manuscript, 2006.
39. Thomas L. Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National Academy of Science, 101:5228-5235, 2004. (Pubitemid 38469131)
40. Dilek Z. Hakkani-Tür, Kemal Oflazer, and Gökhan Tür. Statistical morphological disambiguation for agglutinative languages. Computers and the Humanities, 36(4):381-410, 2002. (Pubitemid 44947889)
41. Zelig Harris. From phoneme to morpheme. Language, 31:190-222, 1955.
42. Jennifer Hay. Lexical frequency in morphology: Is everything relative? Linguistics, 39(6):1041-1070, 2001. (Pubitemid 33620691)
43. Jennifer Hay and R. Harald Baayen. Shifting paradigms: gradient structure in morphology. Trends in Cognitive Sciences, 9(7):342-348, 2005. (Pubitemid 40910333)
44. Songfang Huang and Steve Renals. Hierarchical Bayesian language models for conversational speech recognition. IEEE Transactions on Audio, Speech and Language Processing, 18(8):1941- 1954, 2010.
45. Hemant Ishwaran and Lancelot James. Generalized weighted Chinese restaurant processes for species sampling mixture models. Statistica Sinica, 13:1211-1235, 2003. (Pubitemid 38042014)
46. Ray Jackendoff. Foundations of Language: Brain, Meaning, Grammar, Evolution. Oxford University Press, Oxford/New York, 2002.
47. Mark Johnson. Using adaptor grammars to identify synergies in the unsupervised acquisition of linguistic structure. In Proceedings of ACL-08: HLT, Columbus, Ohio, 2008a.
48. Mark Johnson. Unsupervised word segmentation for Sesotho using adaptor grammars. In Proceedings of the Tenth Meeting of ACL Special Interest Group on Computational Morphology and Phonology (SIGMORPHON), Columbus, Ohio, 2008b.
49. Mark Johnson. PCFGs, topic models, adaptor grammars and learning topical collocations and the structure of proper names. In Proceedings of the 48th Annual Meeting of the Association of Computational Linguistics, pages 1148-1157, Uppsala, Sweden, 2010.
50. Mark Johnson and Sharon Goldwater. Improving nonparametric Bayesian inference: Experiments on unsupervised word segmentation with adaptor grammars. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Boulder, Colorado, 2009.
51. Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. Adaptor grammars: a framework for specifying compositional nonparametric Bayesian models. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, Cambridge, MA, 2007. MIT Press.
52. Aravind Joshi. Tree adjoining grammars. In Ruslan Mikkov, editor, The Oxford Handbook of Computational Linguistics, pages 483-501. Oxford University Press, Oxford, England, 2003.
53. Reinhard Kneser and Hermann Ney. Improved backing-off for n-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 181-184, Detroit, Michigan, 1995.
54. Philipp Koehn and Hieu Hoang. Factored translation models. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 868-876, Prague, Czech Republic, 2007.
55. Leah S. Larkey, Lisa Ballesteros, and Margaret Connell. Improving stemming for Arabic information retrieval: Light stemming and co-occurrence analysis. In Proceedings of the 25th International Conference on Research and Development in Information Retrieval (SIGIR), pages 275-282, Tampere, Finland, 2002.
56. Percy Liang, Slav Petrov, Michael Jordan, and Dan Klein. The infinite PCFG using hierarchical Dirichlet processes. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 688-697, Prague, Czech Republic, 2007.
57. Albert Lo. On a class of Bayesian nonparametric estimates. Annals of Statistics, 12:351-357, 1984.
58. David MacKay and Linda Bauman Peto. A hierarchical Dirichlet language model. Natural Language Engineering, 1(1), 1994.
59. Brian MacWhinney and Catharine Snow. The child language data exchange system. Journal of Child Language, 12:271-296, 1985.
60. Rasmus Madsen, David Kauchak, and Charles Elkan. Modeling word burstiness using the Dirichlet distribution. In Proceedings of the 22nd International Conference on Machine Learning, pages 545-552, Bonn, Germany, 2005. (Pubitemid 43183377)
61. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19(2):331-330, 1993.
62. Michael Mitzenmacher. A brief history of generative models for power law and lognormal distributions. Internet Mathematics, 1(2):226-251, 2004.
63. Christian Monson, Alon Lavie, Jaime Carbonell, and Lori Levin. Unsupervised induction of natural language morphology inflection classes. In Proceedings of the Seventh Meeting of the ACL Special Interest Group in Computational Phonology (SIGPHON), pages 52-61, Barcelona, Spain, 2004.
64. Radford Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9:249-265, 2000.
65. David Newman, Arthur Asuncion, Padhraic Smyth, and Max Welling. Distributed algorithms for topic models. Journal of Machine Learning Research, 10(Aug):1801-1828, 2009.
66. Hermann Ney, Ufe Essen, and Reinhard Kneser. On structuring probabilistic dependencies in stochastic language modeling. Computer, Speech, and Language, 8:1-38, 1994.
67. Timothy O'Donnell. Computation and Reuse in Language. PhD thesis, Harvard University, in preparation.
68. Timothy O'Donnell, Noah Goodman, and Joshua B. Tenenbaum. Fragment grammars: Exploring computation and reuse in language. Technical Report MIT-CSAIL-TR-2009-013, MIT, 2009.
69. Janet Pierrehumbert. Probabilistic phonology: Discrimination and robustness. In R. Bod, J. Hay, and S. Jannedy, editors, Probabilistic Linguistics. MIT Press, Cambridge, MA, 2003.
70. Jim Pitman. Exchangeable and partially exchangeable random partitions. Probability Theory and Related Fields, 102:145-158, 1995.
71. Jim Pitman. Combinatorial Stochastic Processes. Springer-Verlag, New York, 2006.
72. Jim Pitman and Marc Yor. The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator. Annals of Probability, 25:855-900, 1997.
73. David Plaut and Laura Gonnerman. Are non-semantic morphological effects incompatible with a distributed connectionist approach to lexical processing? Language and Cognitive Processes, 15: 445-485, 2000.
74. Matt Post and Daniel Gildea. Bayesian learning of a tree substitution grammar. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 45-48, Suntec, Singapore, August 2009.
75. Carl Rasmussen. The infinite Gaussian mixture model. In Advances in Neural Information Processing Systems 12, Cambridge, MA, 2000. MIT Press.
76. Terry Regier, Bryce Corrigan, Rachael Cabasaan, Amanda Woodward, Michael Gasser, and Linda Smith. The emergence of words. In Proceedings of the 23rd Annual Conference of the Cognitive Science Society, pages 815-820, Mahwah, NJ, 2001. Erlbaum.
77. Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information Processing and Management: an International Journal, 24(5):513-523, 1988.
78. Jayazam Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica, 4:639-650, 1994.
79. Herbert Simon. On a class of skew distribution functions. Biometrika, 42(3/4):425-440, 1955.
80. Matthew Snover and Michael Brent. A probabilistic model for learning concatenative morphology. In Advances in Neural Information Processing Systems 15, Cambridge, MA, 2003. MIT Press.
81. Benjamin Snyder and Regina Barzilay. Unsupervised multilingual learning for morphological segmentation. In Proceedings of ACL-08: HLT, pages 737-745, Columbus, Ohio, 2008.
82. Yee Whye Teh. A Bayesian interpretation of interpolated Kneser-Ney. Technical Report TRA2/06, National University of Singapore, School of Computing, 2006a.
83. Yee Whye Teh. A hierarchical Bayesian language model based on Pitman-Yor processes. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 985-992, Syndney, Australia, 2006b.
84. Yee Whye Teh, Michael Jordan, Matthew Beal, and David Blei. Hierarchical Dirichlet processes. In Advances in Neural Information Processing Systems 17. MIT Press, Cambridge, MA, 2005.
85. Romain Thibaux and Michael I. Jordan. Hierarchical beta processes and the Indian buffet process. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, San Juan, Puerto Rico, 2007.
86. Hanna M. Wallach. Structured Topic Models for Language. PhD thesis, University of Cambridge, 2008.
87. Mike West. Hyperparameter estimation in Dirichlet process mixture models. Technical Report 92-A03, Institute of Statistics and Decision Sciences, Duke University, 1992.
88. Frank Wood and Yee Whye Teh. A hierarchical, hierarchical Pitman Yor process language model. In Proceedings of the ICML/UAI Workshop on Nonparametric Bayes, Helsinki, Finland, 2008.
89. Frank Wood and Yee Whye Teh. A hierarchical nonparametric Bayesian approach to statistical language model domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Clearwater Beach, Florida, 2009.
90. David Yarowsky and Richard Wicentowski. Minimally supervised morphological analysis by mul-timodal alignment. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 207-216, Hong Kong, 2000.
91. George Zipf. Selective Studies and the Principle of Relative Frequency in Language. Harvard University Press, Cambridge, MA, 1932.