Investigations on an EM-Style optimization algorithm for discriminative training of HMMs

IEEE Transactions on Audio, Speech and Language Processing

Volumn 21, Issue 12, 2013, Pages 2616-2626

  Heigold, Georg ^a,b   Ney, Hermann ^a   Schluter, Ralf ^a  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b GOOGLE INC   (United States)

Author keywords

discriminative training; Expectation maximization; generalized iterative scaling; hidden Markov model

Indexed keywords

DISCRIMINATIVE TRAINING; EXPECTATION MAXIMIZATION; GENERALIZED ITERATIVE SCALING; GRAPHEME-TO-PHONEME CONVERSION; HIDDEN MARKOV MODELS (HMMS); MAXIMUM MUTUAL INFORMATION; OPTIMIZATION ALGORITHMS; SPEECH RECOGNITION SYSTEMS;

HIDDEN MARKOV MODELS; ITERATIVE METHODS; MAXIMUM PRINCIPLE; OPTIMIZATION; PARAMETER ESTIMATION; SPEECH RECOGNITION; STEREOPHONIC BROADCASTING;

ALGORITHMS;

EID: 84887376734     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2013.2280234     Document Type: Article

References (44)

1. L. R. Rabiner, "A tutorial on hiddenMarkovmodels and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
2. Y. Normandin, "Hidden Markov models, maximum mutual information, and the speech recognition problem," Ph.D. dissertation, McGill Univ., Montreal, QC, Canada, 1991.
3. P. C. Woodland and D. Povey, "Large scale discriminative training of hidden Markov models for speech recognition," Comput. Speech Lang., vol. 16, no. 1, pp. 25-48, 2002.
4. D. Povey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah, "Boosted MMI for model and feature-space discriminative training," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Las Vegas, NV, USA, Apr. 2008, pp. 4057-4060.
5. E. McDermott, T. Hazen, J. L. Roux, A. Nakamura, and S. Katagiri, "Discriminative training for large vocabulary speech recognition using minimum classification error," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 203-223, Jan. 2007.
6. W. Macherey, L. Haferkamp, R. Schlüter, and H. Ney, "Investigations on errorminimizing training criteria for discriminative training in automatic speech recognition," in Proc. Interspeech, Lisbon, Portugal, Sep. 2005, pp. 2133-2136. (Pubitemid 43908515)
7. D. Povey, "Discriminative training for large vocabulary speech recognition," Ph.D. dissertation, Cambridge, U.K., 2004.
8. Y. Normandin and S. Morgera, "An improved MMIE training algorithm for speaker-independent, small vocabulary, continuous speech recognition," in IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Toronto, ON, Canada, May 1991, pp. 537-540.
9. R. Schlüter, "Investigations on discriminative training criteria," Ph.D. dissertation, RWTH Aachen Univ., Aachen, Germany, Sep. 2000.
10. W. Macherey, R. Schlüter, and H. Ney, "Discriminative training with tied covariancematrices," in Proc. Interspeech, Jeju Island,Korea,Oct. 2004, pp. 681-684.
11. R. Hsiao, Y.-C. Tam, and T. Schultz, "Generalized Baum-Welch algorithm for discriminative training on large vocabulary continuous speech recognition systems," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Taipei, Taiwan, Apr. 2009, pp. 3769-3772.
12. P. Gopalakrishnan, D. Kanevsky, A. Nadas, and D. Nahamoo, "An inequality for rational functions with applications to some statistical estimation problems," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 107-113, Jan. 1991.
13. D. Kanevsky, "Extended Baum Welch transformations for general functions," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Montreal, QC, Canada, May 2004, pp. 821-824.
14. S. Axelrod, V. Goel, R. Gopinath, P. Olsen, and K. Visweswariah, "Discriminative estimation of subspace constrained Gaussian mixture models for speech recognition," IEEE Trans. Speech Audio Process., vol. 15, no. 1, pp. 172-189, Jan. 2007.
15. M. Riedmiller and H. Braun, "A direct adaptivemethod for faster backpropagation learning: The Rprop algorithm," in Proc. IEEE Int. Conf. Neural Netw. (ICNN), San Francisco, CA, USA, Mar.-Apr. 1993, pp. 586-591.
16. G. Heigold, "A log-linear discriminative modeling framework for speech recognition," Ph.D. dissertation, RWTH Aachen Univ., Aachen, Germany, Jun. 2010.
17. J. Nocedal and S. Wright, Numerical Optimization. NewYork,NY, USA: Springer, 1999.
18. A. D. Anastasiadis, G. D. Magoulas, and M. N. Vrahatis, "New globally convergent training scheme based on the resilient propagation algorithm," Neurocomputing, vol. 64, pp. 253-270, 2005. (Pubitemid 40425322)
19. A. Dempster, N. Laird, and D. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc., vol. 39, no. B, pp. 1-38, 1977.
20. J. Darroch and D. Ratcliff, "Generalized iterative scaling for log-linear models," Ann. Math. Statist., vol. 43, pp. 1470-1480, 1972.
21. S. Della Pietra, V. Della Pietra, and J. Lafferty, "Inducing features of random fields," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 4, pp. 380-393, Apr. 1997. (Pubitemid 127762893)
22. A. Gunawardana, "Maximum mutual information estimation of acoustic HMM emission densities, Center for Language and Speech Processing," Johns Hopkins Univ., Baltimore, MD, CLSP Research Note No. 40, 2001.
23. T. Jebara, "Discriminative, generative, and imitative learning," Ph.D. dissertation, Mass. Inst. of Technol., Cambridge, MA, USA, 2002.
24. M. Afify, "Extended Baum-Welch reestimation of Gaussian mixture models based on reverse Jensen inequality," in Interspeech, Lisbon, Portugal, Sep. 2005, pp. 1113-1116. (Pubitemid 43908261)
25. S. Riezler, "Probabilistic constraint logic programming," Ph.D. dissertation, Univ. Tübingen, Tübingen, Germany, 1998.
26. S. Riezler, J. Kuhn, D. Prescher, and M. Johnson, "Lexicalized stochastic modeling of constraint-based grammars using log-linear measures and EM training," in Proc. Annu. Meeting Assoc. Comput. Linguist. (ACL), Hong Kong, Oct. 2000, pp. 480-487.
27. G. Heigold, T. Deselaers, R. Schlüter, andH. Ney, "GIS-like estimation of log-linear models with hidden variables," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), LasVegas,NV, USA,Apr. 2008, pp. 4045-4048.
28. G. Heigold, S. Hahn, P. Lehnen, and H. Ney, "EM-style optimization of hidden conditional random fields for grapheme-to-phoneme conversion," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Prague, Czech Republic, May 2011, pp. 4920-4923.
29. G. Heigold, H. Ney, P. Lehnen, T. Gass, and R. Schlüter, "Equivalence of generative and log-linear models," IEEE Trans. Speech Audio Process., vol. 19, no. 5, pp. 1138-1148, Jul. 2011.
30. C. Bishop, Pattern Recognition and Machine Learning. NewYork, NY, USA: Springer, 2006.
31. C. Wu, "On the convergence properties of the EM algorithm," Ann. Statist., vol. 11, no. 1, pp. 95-103, 1983.
32. S. Wang, D. Schuurmans, and Y. Zhao, "The latent maximum entropy principle," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Lausanne, Switzerland, Jun.-Jul. 2002, p. 131. (Pubitemid 34964752)
33. A. Gunawardana, M. Mahajan, A. Acero, and J. Platt, "Hidden conditional random fields for phone classification," in Proc. Interspeech, Lisbon, Portugal, Sep. 2005, pp. 117-120.
34. G. Heigold, P. Dreuw, S. Hahn, R. Schlüter, and H. Ney, "Margin-based discriminative training for string recognition," IEEE J. Sel. Topics Signal Process., vol. 4, no. 6, pp. 917-925, Dec. 2010.
35. L. Armijo, "Minimization of functions having Lipschitz continuous first derivatives," Pacific J. Math., vol. 16, pp. 1-3, 1966.
36. E. Ising, "Beitrag zur Theorie des Ferromagnetismus," Z. Phys., vol. 31, pp. 253-258, 1925.
37. G. Heigold, T. Deselaers, R. Schlüter, and H. Ney, "Modified MMI/MPE: A direct evaluation of the margin in speech recognition," in Proc. Int. Conf. Mach. Learn. (ICML), Helsinki, Finland, Jul. 2008, pp. 384-391.
38. G. Heigold, R. Schlüter, and H. Ney, "Modified MPE/MMI in a transducer-based framework," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Taipei, Taiwan, Apr. 2009, pp. 3749-3752.
39. D. Keysers, T. Deselaers, C. Gollan, and H. Ney, "Deformation models for image recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 8, pp. 1422-1435, Aug. 2007. (Pubitemid 47040440)
40. L. Ramshaw and M. Marcus, "Text chunking using transformationbased learning," in Proc. 3rd Workshop Very Large Corpora, Cambridge, MA, USA, Jun. 1995, pp. 84-94.
41. T. Eisele, R. Haeb-Umbach, and D. Langmann, "A comparative study of linear feature transformation techniques for automatic speech recognition," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), Philadelphia, PA, USA, Oct. 1996, pp. 252-255.
42. Y. H. Abdel-Haleem, "Conditional random fields for continuous speech recognition," Ph.D. dissertation, Faculty of Eng., Univ. of Sheffield, Sheffield, U.K., 2006.
43. S.Wiesler and H. Ney, "A convergence analysis of log-linear training," in Advances in Neural Information Processing Systems (NIPS). Cambridge, MA, USA: MIT Press, Dec. 2011, pp. 657-665.
44. A. Senior, G. Heigold, M. Ranzato, and K. Yang, "An empirical study of learning rates in deep neural networks for speech recognition," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Vancouver, BC, Canada, Apr. 2013, vol. 1, pp. 6724-6728.