Bayesian Recurrent Neural Network for Language Modeling

IEEE Transactions on Neural Networks and Learning Systems

Volumn 27, Issue 2, 2016, Pages 361-374

  Chien, Jen Tzung ^a   Ku, Yuan Chu ^a  

^a NATIONAL CHIAO TUNG UNIVERSITY   (Taiwan)

Author keywords

Bayesian learning; Hessian matrix; language model; rapid approximation; recurrent neural network.

Indexed keywords

BAYESIAN NETWORKS; COMPUTATIONAL LINGUISTICS; CONTINUOUS SPEECH RECOGNITION; MODELING LANGUAGES; PARAMETER ESTIMATION; RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION; UNCERTAINTY ANALYSIS;

BAYESIAN APPROACHES; BAYESIAN CLASSIFICATION; CONTINUOUS SPACES; ILL POSED PROBLEM; MARGINAL LIKELIHOOD; MAXIMUM A POSTERIORI CRITERIA; OBJECTIVE FUNCTIONS; RECURRENT NEURAL NETWORK (RNN);

KNOWLEDGE BASED SYSTEMS;

EID: 84949921270     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2015.2499302     Document Type: Article

References (41)

1. J.-T. Chien, "Association pattern language modeling, " IEEE Trans. Audio, Speech, Language Process., vol. 14, no. 5, pp. 1719-1728, Sep. 2006.
2. R. Kuhn and R. de Mori, "A cache-based natural language model for speech recognition, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 6, pp. 570-583, Jun. 1990.
3. J.-T. Chien and C.-H. Chueh, "Dirichlet class language models for speech recognition, " IEEE Trans. Audio, Speech, Language Process., vol. 19, no. 3, pp. 482-495, Mar. 2011.
4. P. F. Brown, V. J. Della Pietra, P. V. deSouza, J. C. Lai, and R. L. Mercer, "Class-based n-gram models of natural language, " Comput. Linguistics, vol. 18, no. 4, pp. 467-479, Dec. 1992.
5. R. Kneser and H. Ney, "Improved backing-off for M-gram language modeling, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Detroit, MI, USA, May 1995, pp. 181-184.
6. S. F. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling, " Comput. Speech Lang., vol. 13, no. 4, pp. 359-394, Oct. 1999.
7. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, "A neural probabilistic language model, " J. Mach. Learn. Res., vol. 3, pp. 1137-1155, Feb. 2003.
8. A. Emami and F. Jelinek, "Exact training of a neural syntactic language model, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 1. Montreal, QC, Canada, May 2004, pp. I-245-I-248.
9. H. Schwenk, "Continuous space language models, " Comput. Speech Lang., vol. 21, no. 3, pp. 492-518, Jul. 2007.
10. E. Arisoy, T. N. Sainath, B. Kingsbury, and B. Ramabhadran, "Deep neural network language models, " in Proc. North Amer. Chapter Assoc. Comput. Linguistics-Human Lang. Technol. Workshop (NAACL-HLT), Montreal, QC, Canada, Jun. 2012, pp. 20-28.
11. T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, "Recurrent neural network based language model, " in Proc. 11th Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), Chiba, Japan, Sep. 2010, pp. 1045-1048.
12. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Cernocký, "Empirical evaluation and combination of advanced language modeling techniques, " in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), Florence, Italy, Aug. 2011, pp. 605-608.
13. C. Chelba et al., "One billion word benchmark for measuring progress in statistical language modeling, " in Proc. Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), Singapore, Sep. 2014, pp. 2635-2639.
14. R. M. Neal, "Bayesian learning for neural networks, " Ph.D. dissertation, Dept. Comput. Sci., Univ. Toronto, Toronto, ON, Canada, 1995.
15. D. J. C. MacKay, "A practical Bayesian framework for backpropagation networks, " Neural Comput., vol. 4, no. 3, pp. 448-472, May 1992.
16. C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY, USA: Springer-Verlag, Feb. 2006.
17. S. Watanabe and J.-T. Chien, Bayesian Speech and Language Processing. Cambridge, U.K.: Cambridge Univ. Press, Jul. 2015.
18. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-Term dependencies with gradient descent is difficult, " IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994.
19. G. Saon and J.-T. Chien, "Large-vocabulary continuous speech recognition systems: A look at some recent advances, " IEEE Signal Process. Mag., vol. 29, no. 6, pp. 18-33, Nov. 2012.
20. H. Masataki, Y. Sagisaka, K. Hisaki, and T. Kawahara, "Task adaptation using MAP estimation in N-gram language modeling, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Munich, Germany, Apr. 1997, pp. 783-786.
21. S. F. Chen, "Shrinking exponential language models, " in Proc. North Amer. Chapter Assoc. Comput. Linguistics-Human Lang. Technol. Workshop (NAACL-HLT), Boulder, CO, USA, Jun. 2009, pp. 468-476.
22. R. Rosenfeld, "A maximum entropy approach to adaptive statistical language modelling, " Comput. Speech Lang., vol. 10, no. 3, pp. 187-228, Jul. 1996.
23. D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent Dirichlet allocation, " J. Mach. Learn. Res., vol. 3, pp. 993-1022, Jan. 2003.
24. Y. W. Teh, "A hierarchical Bayesian language model based on Pitman-Yor processes, " in Proc. 21st Int. Conf. Comput. Linguistics, 44th Annu. Meeting Assoc. Comput. Linguistics, Sydney, NSW, Australia, Jul. 2006, pp. 985-992.
25. J. Pitman and M. Yor, "The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator, " Ann. Probab., vol. 25, no. 2, pp. 855-900, Apr. 1997.
26. J.-T. Chien, "Hierarchical Pitman-Yor-Dirichlet language model, " IEEE/ACM Trans. Audio, Speech, Language Process., vol. 23, no. 8, pp. 1259-1272, Aug. 2015.
27. D. T. Mirikitani and N. Nikolaev, "Recursive Bayesian recurrent neural networks for time-series modeling, " IEEE Trans. Neural Netw., vol. 21, no. 2, pp. 262-274, Feb. 2010.
28. S. Renals and D. J. C. MacKay, "Bayesian regularisation methods in a hybrid MLP-HMM system, " in Proc. 3rd Eur. Conf. Speech Commun. Technol. (EUROSPEECH), Berlin, Germany, Sep. 1993, pp. 1719-1722.
29. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors, " Nature, vol. 323, pp. 533-536, Oct. 1986.
30. D. J. C. MacKay, "The evidence framework applied to classification networks, " Neural Comput., vol. 4, no. 5, pp. 720-736, Sep. 1992.
31. J.-T. Chien, Y.-C. Ku, and M.-Y. Huang, "Rapid Bayesian learning for recurrent neural network language model, " in Proc. 9th Int. Symp. Chin. Spoken Lang. Process. (ISCSLP), Singapore, Sep. 2014, pp. 34-38.
32. J.-T. Chien and Y.-C. Ku, "Bayesian recurrent neural network language model, " in Proc. IEEE Spoken Lang. Technol. Workshop (SLT), South Lake Tahoe, NV, USA, Dec. 2014, pp. 206-211.
33. A. Stolcke, "SRILM-An extensible language modeling toolkit, " in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), Denver, CO, USA, Sep. 2002, pp. 901-904.
34. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. H. Cernocký, "RNNLM-Recurrent neural network language modeling toolkit, " in Proc. IEEE Autom. Speech Recognit. Understand. Workshop (ASRU), Waikoloa, HI, USA, Dec. 2011, pp. 196-201.
35. J.-T. Chien, "Laplace group sensing for acoustic models, " IEEE/ACM Trans. Audio, Speech, Language Process., vol. 23, no. 5, pp. 909-922, May 2015.
36. T. Mikolov, S. Kombrink, L. Burget, J. H. Cernocký, and S. Khudanpur, "Extensions of recurrent neural network language model, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Prague, Czech Republic, May 2011, pp. 5528-5531.
37. D. Povey et al., "The Kaldi speech recognition toolkit, " in Proc. IEEE Autom. Speech Recognit. Understand. Workshop (ASRU), Waikoloa, HI, USA, Dec. 2011.
38. J.-T. Chien and T.-W. Lu, "Tikhonov regularization for deep neural network acoustic modeling, " in Proc. IEEE Spoken Lang. Technol. Workshop (SLT), South Lake Tahoe, NV, USA, Dec. 2014, pp. 147-152.
39. G. Saon, M. Padmanabhan, R. Gopinath, and S. Chen, "Maximum likelihood discriminant feature spaces, " in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 2. Istanbul, Turkey, Jun. 2000, pp. 1129-1132.
40. M. J. F. Gales, "Maximum likelihood linear transformations for HMMbased speech recognition, " Comput. Speech Lang., vol. 12, no. 2, pp. 75-98, Apr. 1998.
41. T. Mikolov and G. Zweig, "Context dependent recurrent neural network language model, " in Proc. IEEE Spoken Lang. Technol. Workshop (SLT), Miami, FL, USA, Dec. 2012, pp. 234-239.