Hierarchical Pitman-Yor-Dirichlet language model

IEEE Transactions on Audio, Speech and Language Processing

Volumn 23, Issue 8, 2015, Pages 1259-1272

  Chien, Jen Tzung ^a  

^a NATIONAL CHIAO TUNG UNIVERSITY   (Taiwan)

Author keywords

Bayesian nonparametrics; language model; speech recognition; topic model; unsupervised learning

Indexed keywords

DATA MINING; PROBABILITY DISTRIBUTIONS; SEMANTICS; SPEECH RECOGNITION; STATISTICS; UNSUPERVISED LEARNING;

BAYESIAN NONPARAMETRICS; HIERARCHICAL DIRICHLET PROCESS; LANGUAGE MODEL; LATENT DIRICHLET ALLOCATION; POSTERIOR DISTRIBUTIONS; PROBABILISTIC DISTRIBUTION; PROBABILISTIC REPRESENTATION; TOPIC MODELING;

COMPUTATIONAL LINGUISTICS;

EID: 84930629832     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2015.2428632     Document Type: Article

References (35)

1. Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei, "Hierarchical Dirichlet processes," J. Amer. Statist. Assoc., vol. 101, no. 476, pp. 1566-1581, 2006.
2. Y. W. Teh, "A hierarchical Bayesian language model based on pitman-yor processes," in Proc. Annu. Meeting Assoc. Comput. Linguist., 2006, pp. 985-992.
3. J.-T. Chien, "Association pattern language modeling," IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 5, pp. 1719-1728, Sep. 2006.
4. G. Saon and J.-T. Chien, "Large-vocabulary continuous speech recognition systems-a look at some recent advances," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 18-33, Nov. 2012.
5. R. Kneser and H. Ney, "Improved backing-off for m-gram language modeling," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1995, pp. 181-184.
6. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling," Comput. Speech Lang., vol. 13, no. 4, pp. 359-394, 1999.
7. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model," in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2010, pp. 1045-1048.
8. R. Kuhn and R. De Mori, "A cache-based natural language model for speech recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 6, pp. 570-583, Jun. 1990.
9. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, "A neural probabilistic language model," J. Mach. Learn. Res., vol. 3, pp. 1137-1155, 2003.
10. J. Bellegarda, "Exploiting latent semantic information in statistical language modeling," Proc. IEEE, vol. 88, no. 8, pp. 1279-1296, Aug. 2000.
11. D. Gildea and T. Hofmann, "Topic-based language models using EM," in Proc. Eur. Conf. Speech Commun. Technol. (EUROSPEECH), 1999, pp. 2167-2170.
12. P. R. Clarkson and A. J. Robinson, "Language model adaptation using mixtures and an exponentially decaying cache," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1997, pp. 799-802.
13. D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent Dirichlet allocation," J. Mach. Learn. Res., vol. 3, no. 5, pp. 993-1022, 2003.
14. Y. C. Tam and T. Schultz, "Dynamic language model adaptation using variational Bayes inference," in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2005, pp. 5-8.
15. J.-T. Chien and C.-H. Chueh, "Dirichlet class language models for speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 3, pp. 482-495, Mar. 2011.
16. P. Brown, P. Desouza, R. Mercer, V. Pietra, and J. Lai, "Class-based-gram models of natural language," Comput. Linguist., vol. 18, no. 4, pp. 467-479, 1992.
17. J.-T. Chien and Y.-L. Chang, "Hierarchical Pitman-Yor and Dirichlet process for language model," in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2013, pp. 2212-2216.
18. S. Goldwater, T. L. Griffiths, and M. Johnson, "Interpolating between types and tokens by estimating power-law generators," Adv. Neural Inf. Process. Syst., vol. 18, pp. 459-466, 2006.
19. J.-T. Chien, "Hierarchical theme and topic modeling," IEEE Trans. Neural Netw. Learn. Syst., 2015, to be published.
20. J. Pitman, "Poisson-Dirichlet and GEM invariant distributions for splitand-merge transformation of an interval partition," Combinat. Probab. Comput., vol. 11, pp. 501-514, 2002.
21. S. Huang and S. Renals, "Hierarchical Bayesian language models for conversational speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no. 8, pp. 1941-1954, Nov. 2010.
22. J. Pitman and M. Yor, "The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator," Ann. Probab., vol. 25, no. 2, pp. 855-900, 1997.
23. D. J. C. MacKay and L. C. B. Peto, "A hierarchical Dirichlet language model," Nat. Lang. Eng., vol. 1, no. 3, pp. 289-308, 1995.
24. Y. Su, "Bayesian class-based language models," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2011, pp. 5564-5567.
25. F. Wood and Y. W. Teh, "A hierarchical nonparametric Bayesian approach to statistical language model adaptation," in Proc. Int. Conf. Artif. Intell. Statist., 2009, vol. 5, pp. 607-614.
26. D. Machihashi, T. Yamada, and N. Ueda, "Bayesian unsupervised word segmentation with nested pitman-yor language modeling," in Proc. Annu. Meeting Assoc. Comput. Linguist., 2009, pp. 100-108.
27. I. Sato and H. Nakagawa, "Topic models with power-law using Pitman-Yor process," in Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 2010, pp. 673-682.
28. R. V. Lindsey, "A phrase-discovering topic model using hierarchical pitman-yor processes," in Proc. Joint Conf. Empir. Meth. Nat. Lang. Process. Comput. Nat. Lang. Learn., 2012, pp. 214-222.
29. S. Yaman, J.-T. Chien, and C.-H. Lee, "Structural Bayesian language modeling and adaptation," in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2007, pp. 2365-2368.
30. H. Noji, D. Machihashi, and Y. Miyao, "Improvement to the Bayesian topic-gram models," in Proc. Conf. Empir.Meth. Nat. Lang. Process. (EMNLP), 2013, pp. 1180-1190.
31. H. Wang, B. Chen, J.-W. Kuo, and S.-S. Cheng, "MATBN: A mandarin Chinese broadcast news corpus," Int. J. Comput. Linguist. Chinese Lang. Process., vol. 10, no. 1, pp. 219-235, 2005.
32. J.-T. Chien and C.-H. Huang, "Aggregate a posteriori linear regression adaptation," IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 3, pp. 797-807, May 2006.
33. S. Young, G. Evermann, M. J. F. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. J. Odell, D. Ollason, D. Povey, and V. Valchev P. Woodland, The HTK Book, Version 3.4. Cambridge, U.K.: Cambridge Univ. Eng. Dept., 2006.
34. A. Stolcke, "SRILM-an extensible language modeling toolkit," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), 2002, pp. 901-904.
35. Y. W. Teh, "A Bayesian interpretation of interpolated Kneser-Ney," School of Comput., National Univ. of Singapore, Tech. Rep. TRA2/06, 2006.