Bayesian approaches to acoustic modeling: A review

APSIPA Transactions on Signal and Information Processing

Volumn 1, Issue , 2012, Pages

  Watanabe, Shinji ^a   Nakamura, Atsushi ^b  

^a MITSUBISHI ELECTRIC RESEARCH LABORATORIES   (United States)

^b Nippon Telegraph and Telephone Corporation   (Japan)

Author keywords

Approximate bayesian inference; Bayesian approach; Machine learning; Speech processing

Indexed keywords

ARTIFICIAL INTELLIGENCE; BAYESIAN NETWORKS; INFERENCE ENGINES; LEARNING SYSTEMS; MARKOV PROCESSES; MAXIMUM LIKELIHOOD; MONTE CARLO METHODS; SPEECH; SPEECH PROCESSING; VARIATIONAL TECHNIQUES;

APPROXIMATE BAYESIAN INFERENCE; ASYMPTOTIC APPROXIMATION; BAYESIAN APPROACHES; BAYESIAN INFORMATION CRITERION; GENERALIZATION CAPABILITY; MARKOV CHAIN MONTE-CARLO; NUMERICAL COMPUTATIONS; VARIATIONAL APPROXIMATION;

SPEECH RECOGNITION;

EID: 84887091716     PISSN: None     EISSN: 20487703     Source Type: Journal    
DOI: 10.1017/ATSIP.2012.6     Document Type: Review

References (93)

1. Dempster, A.P.; Laird, N.M.; Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc. B, 39, (1976) 1-38.
2. Jelinek, F.: Continuous speech recognition by statistical methods. Proc. IEEE, 64(4), (1976) 532-556.
3. Huang, X.D.; Ariki, Y.; Jack, M.A.: Hidden Markov Models for Speech Recognition, Edinburgh University Press, 1990.
4. Gales, M.; Young, S.: The application of hidden Markov models in speech recognition. Signal Process., 1, (3), (2007) 195-304.
5. Furui, S.: Recent advances in spontaneous speech recognition and understanding. in Proc. SSPR 2003, 2003, 1-6.
6. Berger, J.O.: Statistical Decision Theory and Bayesian Analysis, 2nd ed., Springer-Verlag, 1985.
7. Bernardo, J.M.; Smith, A.F.M.: Bayesian Theory, John Wiley & Sons Ltd, 1994.
8. Bishop, C.M.: Pattern Recognition and Machine Learning, vol. 4, Springer New York, 2006.
9. Ghahramani, Z.: Unsupervised learning. Advanced Lectures on Machine Learning, 2004, 72-112, Springer.
10. Lee, C.-H.; Lin, C.H.; Juang, B-H.: A study on speaker adaptation of the parameters of continuous density hidden Markov models. IEEE Trans.Acoust. Speech Signal Process., 39, (1991) 806-814.
11. Gauvain, J.L.; Lee, C.H.: Improved acoustic modeling with Bayesian learning. in ICASSP'92, 1, (1992) 481-484.
12. Gauvain, J.-L.; Lee, C.-H.: Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains. IEEE Trans. Speech Audio Process., 2, (1994) 291-298.
13. Reynolds, D.A.; Quatieri, T.F.; Dunn, R.B.: Speaker verification using adapted Gaussian mixture models. Digit. Signal Process., 10, (1-3), (2000) 19-41.
14. Schwarz, G.: Estimating the dimension of a model. Ann. Stat., 6, (1978) 461-464.
15. Akaike, H.: Likelihood and the Bayes procedure. in Bayesian Statistics, J.M.Bernardo, M. H. DeGroot, D. V. Lindley, andA. F.M. Smith, eds. 1980, 143-166, University Press, Valencia, Spain.
16. Shinoda, K.; Watanabe, T.: MDL-based context-dependent subword modeling for speech recognition. J. Acoust. Soc. Jpn. (E), 21, (2000) 79-86.
17. Chou, W.; Reichl, W.: Decision tree state tying based on penalized Bayesian information criterion, in Proc. ICASSP 1999, 1, (1999) 345-348.
18. Chen, S.; Gopinath, R.: Model selection in acoustic modeling. in Proc. Eurospeech 1999, 3, (1999) 1087-1090.
19. Shinoda, K.; Iso, K.: Efficient reduction of Gaussian components using MDL criterion for HMM-based speech recognition. in Proc. ICASSP 2001, 1, (2001) 869-872.
20. Watanabe, S.; Minami, Y.; Nakamura, A.; Ueda, N.: Application of Variational Bayesian Approach to Speech Recognition, NIPS 2002, MIT Press, 2002, 1261-1268.
21. Watanabe, S.; Minami, Y.; Nakamura, A.; Ueda, N.: Variational Bayesian estimation and clustering for speech recognition. IEEE Trans. Speech Audio Process., 12, (2004) 365-381.
22. Fox, E.B.; Sudderth, E.B.; Jordan, M.I.;Willsky, A.S.: An HDP-HMM for systems with state persistence. in Proc. of ICML, 2008, 312-319.
23. Tawara, N.;Watanabe, S.; Ogawa, T.;Kobayashi, T.: Speaker clustering based on utterance-oriented Dirichlet process mixture model. in Proc. Interspeech'11, 2011, 2905-2908.
24. Huo, Q.; Lee, C.-H.: On-line adaptive learning of the correlated continuous density hiddenMarkov models for speech recognition. IEEE Trans. Speech Audio Process., 6, (1998) 386-397.
25. Watanabe, S.;Nakamura, A.: Predictor-corrector adaptation by using time evolution system with macroscopic time scale. IEEE Trans. Audio Speech Lang. Process., 18, (2), (2010) 395-406.
26. Shinoda, K.; Lee, C.-H.: A structural Bayes approach to speaker adaptation. IEEE Trans. Speech Audio Process., 9, (2001) 276-287.
27. Siohan, O.;Myrvoll, T.A.; Lee, C.H.: Structural maximuma posteriori linear regression for fas HMM adaptation.Comput. Speech Lang., 16, (1), (2002) 5-24.
28. Makov, U.E.; Smith, A.F.M.: A quasi-Bayes unsupervised learning procedure for priors. IEEE Trans. Inf. Theory, 23, (1977) 761-764.
29. Huo, Q.; Chan, C.; Lee, C.-H.: On-line adaptation of the SCHMM parameters based on the segmental quasi-Bayes learning for speech recognition. IEEE Trans. Speech Audio Process., 4, (1996) 141-144.
30. Chien, J.T.: Quasi-Bayes linear regression for sequential learning of hidden Markov models. IEEE Trans. Speech Audio Process., 10, (2002) 268-278.
31. Zweig, G.; Russell, S.: Speech recognition with dynamic Bayesian networks, in Proc. Nat. Conf. Artificial Intelligence, 1998, 173-180.
32. Bilmes, J.; Zweig, G.: The Graphical Models Toolkit: An open source software system for speech and time-series processing, in Proc. ICASSP'02, 2002, vol. 4, 3916-3919.
33. Rennie, S.; Hershey, J.R.; Olsen, P.A.: Single channel multi-talker speech recognition: Graphical modeling approaches. IEEE Signal Process.Mag. Spec. Issue Graph. Models, 27, (6), (2010) 66-80.
34. Saon, G.; Chien, J.T.: Bayesian sensing hidden Markov models for speech recognition. in Proc. ICASSP'11. IEEE, 2011, 5056-5059.
35. Lee, C.-H.; Huo, Q.: On adaptive decision rules and decision parameter adaptation for automatic speech recognition. in Proc. IEEE, 88, (2000) 1241-1269.
36. Bilmes, J.; Bartels, C.: Graphical model architectures for speech recognition. IEEE Signal Process. Mag., 22, (5), (2005) 89-100.
37. Watanabe, S.; Chien, J.T.: Tutorial: Bayesian learning for speech and language processing T-10, ICASSP'12, 2012.
38. Price, P.; Fisher, W.M.; Bernstein, J.; Pallett, D.S.: The DARPA 1000- word resource management database for continuous speech recognition, in Proc. ICASSP'88, 1988, 651-654.
39. Povey, D.: Discriminative training for large vocabulary speech recognition, Ph.D. thesis, Cambridge University, 2003.
40. Young, S.J.; Odell, JJ; Woodland, PC: Tree-based state tying for high accuracy acoustic modelling, in Proc. Workshop on Human Language Technology, 1994, 307-312.
41. Hu, R.; Zhao, Y.: Knowledge-based adaptive decision tree state tying for conversational speech recognition. IEEE Trans. Audio Speech Lang. Process., 15, (7), (2007) 2160-2168.
42. Anguera Miro, X.; Bozonnet, S.; Evans, N.; Fredouille, C.; Friedland, G.; Vinyals, O.: Speaker diarization: A review of recent research. IEEE Trans. Audio Speech Lang. Process., 20, (2), (2012) 356-370.
43. Jiang, H.; Hirose, K.; Huo, Q.: Robust speech recognition based on a Bayesian prediction approach. IEEE Trans. Speech Audio Process., 7, (1999) 426-440.
44. Huo, Q.; Lee, C.-H.: A Bayesian predictive classification approach to robust speech recognition. IEEE Trans. Speech Audio Process., 8, (2000) 200-204.
45. Jordan, M.I.; Ghahramani, Z.; Jaakkola, T.S.; Saul, L.K.: An introduction to variational methods for graphical models. Mach. Learn., 37, (1997) 183-233.
46. Waterhouse, S.; MacKay, D.; Robinson, T.: Bayesian Methods for Mixtures of Experts, NIPS 7, MIT Press, 1995, 351-357.
47. Attias, H.: Inferring parameters and structure of latent variable models by variational Bayes, in Proc. Uncertainty in Artificial Intelligence (UAI) 15, 1999, 21-30.
48. Ueda, N.; Ghahramani, Z.: Bayesian model search for mixture models based on optimizing variational bounds. Neural Netw., 15, (2002) 1223-1241.
49. Watanabe, S.: Speech recognition based on a Bayesian approach, Ph.D. thesis, Waseda University, 2006.
50. Kwon, O.; Lee, T.-W.; Chan, K.: Application of variational Bayesian PCA for speech feature extraction, in Proc. ICASSP 2002, 2002, vol. 1, 825-828.
51. Valente, F.; Wellekens, C.: Variational Bayesian feature selection for Gaussian mixture models, in Proc. ICASSP 2004, 2004, vol. 1, 513-516.
52. Pettersen, S.G.S.: Robust Speech Recognition in the Presence of Additive Noise, Ph.D. thesis, Norwegian University of Science and Technology, 2008.
53. Cournapeau, D.; Watanabe, S.; Nakamura, A.; Kawahara, T.: Online unsupervised classification withmodel comparison in the variational Bayes framework for voice activity detection. IEEE J. Sel. Top. Signal Process., 4, (6), (2010) 1071-1083.
54. Zhang, Y.; Liu, P.; Chien, J.T.; Soong, F.: An evidence framework for Bayesian learning of continuous-density hidden Markov models, in Proc. ICASSP 2009, 2009, 3857-3860.
55. Chen, J.C.; Chien, J.T.: Bayesian large margin hiddenMarkov models for speech recognition, in Proc. ICASSP 2009, 2009, pp. 3765-3768.
56. Watanabe, S.; Minami, Y.; Nakamura, A.; Ueda, N.: Bayesian acoustic modeling for spontaneous speech recognition, in Proc. SSPR 2003, 2003, 47-50.
57. Valente, F.; Wellekens, C.: Variational Bayesian GMM for speech recognition, in Proc. Eurospeech 2003, 2003, 441-444.
58. Ogawa, A.; Takahashi, S.: Weighted distance measures for efficient reduction of Gaussian mixture components in HMM-based acoustic model, in Proc. ICASSP'08, 2008, 4173-4176.
59. Watanabe, S.; Nakamura, A.: Acoustic model adaptation based on coarse-fine training of transfer vectors and its application to speaker adaptation task, in Proc. ICSLP 2004, 2004, vol. 4, pp. 2933-2936.
60. Yu, K.; Gales, M.J.F.; Bayesian adaptation and adaptively trained systems, in Proc. Automatic Speech Recognition and Understanding Workshop (ASRU) 2005, 2005, pp. 209-214.
61. Watanabe, S.; Nakamura, A.; Juang, B.H.: Bayesian linear regression for hidden Markov model based on optimizing variational bounds, ' in Proc. MLSP 2011, 2011, 1-6.
62. Hahm, S.J.; Ogawa, A.; Fujimoto, M.;Hori, T.;Nakamura, A.: Speaker adaptation using variational Bayesian linear regression in normalized feature space, in Proc. of Interspeech'12, 2012.
63. Watanabe, S.; Minami, Y.; Nakamura, A.; Ueda, N.: Constructing shared-state hidden Markov models based on a Bayesian approach, in Proc. ICSLP 2002, 2002, vol. 4, 2669-2672.
64. Jitsuhiro, T.; Nakamura, S.: Automatic generation of non-uniform HMM structures based on variational Bayesian approach, in Proc. ICASSP 2004, 2004, vol. 1, 805-808.
65. Watanabe, S.; Sako, A.; Nakamura, A.: Automatic determination of acoustic model topology using variational Bayesian estimation and clustering for large vocabulary continuous speech recognition. IEEE Trans. Audio Speech Lang. Process. 14, (2006) 855-872.
66. Hashimoto, K.; Zen, H.; Nankaku, Y.; Lee, A.; Tokuda, K.: Bayesian context clustering using cross valid prior distribution for HMM based speech recognition, in Proc. Interspeech'08, 2008, 936-939.
67. Shiota, S.; Hashimoto, K.; Nankaku, Y.; Tokuda, K.: Deterministic annealing based training algorithm for Bayesian speech recognition, in Proc. Interspeech' 09, 2009, 680-683.
68. Valente, F.: Infinite models for speaker clustering, in Proc. Interspeech' 06, 2006, 1329-1332.
69. Ding, N.;Ou, Z.:Variational nonparametric Bayesian hidden Markov model, in Proc. ICASSP'10, 2010, 2098-2101.
70. Ishiguro, K.; Yamada, T.; Araki, S.; Nakatani, T.; Sawada, H.: Probabilistic speaker diarization with bag-of-words representations of speaker angle information. IEEE Trans. Audio Speech Lang. Process., 20, (2), (2012) 447-460.
71. Tawara, N.; Ogawa, T.; Watanabe, S.; Kobayashi, T.: Fully Bayesian inference of multi-mixture Gaussian model and its evaluation using speaker clustering, in Proc. ICASSP'12, 2012, 5253-5256.
72. Hashimoto, K.; Zen, H.; Nankaku, Y.; Masuko, T.; Tokuda, K.: A Bayesian approach to HMM-based speech synthesis, in Proc, ICASSP 2009, 2009, 4029-4032.
73. Hershey, J.R.; Olsen, P.A.: Approximating the Kullback Leibler divergence between Gaussian mixturemodels, in Proc. ICASSP 2007, 2007, pp. 317-320.
74. Kubo, Y.; Watanabe, S.; Nakamura, A.; Kobayashi, T.: A regularized discriminative training method of acoustic models derived by minimum relative entropy discrimination, in Proc. Interspeech 2010, 2010, 2954-2957.
75. Goldwater, S.; Griffiths, T.: A fully Bayesian approach toun super vised part-of-speech tagging, in Proc. ACL'07, 2007, 744-751.
76. Mochihashi, D.; Yamada, T.; Ueda, N.: Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling, in Proc. ACL-IJCNLP, 2009, 100-108.
77. Watanabe, S.;Mochihashi, D.;Hori, T.;Nakamura, A.:Gibbs sampling basedmulti-scale mixturemodel for speaker clustering, in ICASSP'11, 2011, 4524-4527.
78. Geman, S.;Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell., 6, (6), (1984) 721-741.
79. Hori, T.; Araki, S.; Yoshioka, T.; Fujimoto, M.; Watanabe, S.; Oba, T.; Ogawa, A.; Otsuka, K.; Mikami, D.; Kinoshita, K.; et al., : Lowlatency real-time meeting recognition and understanding using distant microphones and omni-directional camera. IEEE Trans. Audio Speech Lang. Process., 20, (2), (2012) 499.
80. Fiscus, J.; Ajot, J.; Garofolo, J.: The rich transcription 2007 meeting recognition evaluation. Multimodal Technol. Percept. Humans, 2009 373-389. http://www.springerlink.com/content/94w143777u0165v5/.
81. Hinton, G.;Deng, L.; Yu, D.;Dahl, G.;Mohamed, A.; Jaitly, N.; Senior, A.; Van houcke, V.; Nguyen, P.; Sainath, T.; Kingsbury, B.: Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process. Mag., 29, (6), (2012), 82-97.
82. Ferguson, T.S.: A Bayesian analysis of some nonparametric problems. Ann. Stat., 1, (2) (1973) 209-230.
83. Griffiths, T.; Ghahramani, Z.: Infinite latent feature models and the Indian buffet process. Tech. Rep., Gatsby Unit, 2005.
84. Teh, Y.W.; Jordan, M.I.; Beal, M.J.; Blei, D.M.: Hierarchical Dirichlet processes. J. Am. Stat. Assoc., 101, (476), (2006) 1566-1581.
85. Blei, D.M.; Griffiths, T.L.; Jordan, M.I.: The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. J. ACM, 57, (2), (2010) 7.
86. Lee, C. Y.; Glass, J.: A nonparametric Bayesian approach to acoustic model discovery, in Proc. ACL'12, 2012.
87. Neubig, G.; Mimura, M.; Mori, S.; Kawahara, T.: Learning a language model from continuous speech, in Proc. Interspeech'10, 2010, 1053-1056.
88. Hoffman, M.; Blei, D.; Cook, P.R.: Finding latent sources in recorded music with a shift-invariant HDP, in Proceedings of the International Conference on Digital Audio Effects (DAFx), 2009, 438-444.
89. Yoshii, K.; Goto, M.: Infinite latent harmonic allocation: A nonparametric Bayesian approach to multipitch analysis, in Proc. 11th Int. Conf. Music Information Retrieval (ISMIR), 2010, 309-314.
90. Nakano, M.; Le Roux, J.; Kameoka, H.; Nakamura, T.; Ono, N.; Sagayama, S.: Bayesian nonparametric spectrogram modeling based oninfinite factorial infinitehidden Markov model, in 2011 IEEE Workshop on, Applications of Signal Processing to Audio and Acoustics (WASPAA), 2011, 325-328.
91. McDermott, E.; Hazen, T.J.; Le Roux, J.; Nakamura, A.; Katagiri, S.: "Discriminative training for large-vocabulary speech recognition using minimum classification error", IEEE Trans. Audio Speech Lang. Process., 15, (1), (2007) 203-223.
92. Gales, M.; Watanabe, S.; Fossler-Lussier, E.: Structured discriminativemodels for speech recognition. IEEE Signal Process.Mag., 29, (6), (2012), 70-81.
93. Jebara, T.: Machine Learning: Discriminative and Generative, Springer, 2004.