Discriminative training for automatic speech recognition: Modeling, criteria, optimization, implementation, and performance

IEEE Signal Processing Magazine

Volumn 29, Issue 6, 2012, Pages 58-69

  Heigold, Georg ^a   Ney, Hermann ^b   Schluter, Ralph ^b   Wiesler, Simon ^b  

^a GOOGLE INC   (United States)

^b RWTH AACHEN UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP NEURAL NETWORKS; MAXIMUM LIKELIHOOD;

ACOUSTIC MODEL TRAININGS; AUTOMATIC SPEECH RECOGNITION; DISCRIMINATIVE TRAINING; EFFICIENT IMPLEMENTATION; FUNDAMENTAL COMPONENT; PARAMETER OPTIMIZATION; STATE-OF-THE-ART SYSTEM; STATISTICAL MODELING;

SPEECH RECOGNITION;

EID: 85032751713     PISSN: 10535888     EISSN: None     Source Type: Journal    
DOI: 10.1109/MSP.2012.2197232     Document Type: Article

References (54)

1. T. Bayes, "An essay towards solving a problem in the doctrine of chances," Philos. Trans. R. Soc. Lond., vol. 53, pp. 370-418, 1763 (reprinted in Biometrika, vol. 45, no. 3/4, pp. 293-315, Dec. 1958).
2. H. Ney, U. Essen, and R. Kneser, "On structuring probabilistic dependencies in language modeling," Comput. Speech Lang., vol. 2, no. 8, pp. 1-38, 1994.
3. S. Davis and P. Mermelstein, "Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences," IEEE Trans. Acoustics Speech Signal Processing, vol. ASSP-28, no. 4, pp. 357-366, Aug. 1980. (Pubitemid 11464930)
4. H. Hermansky, "Perceptual linear predictive (PLP) analysis of speech," J. Acoust. Soc. Am., vol. 87, no. 4, pp. 1738-1752, June 1990. (Pubitemid 20256470)
5. H. Hermansky, D. Ellis, and S. Sharma, "Tandem connectionist feature extraction for conventional HMM systems," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Istanbul, Turkey, June 2000, pp. 1635-1638.
6. A. Dempster, N. Laird, and D. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Stat. Soc., vol. 39, no. B, pp. 1-38, 1977.
7. A. Nádas, "A decision theoretic formulation of a training problem in speech recognition and a comparison of training by unconditional versus conditional maximum likelihood," IEEE Trans. Acoust. Speech Signal Processing, vol. ASSP-31, pp. 814-817, Aug. 1983. (Pubitemid 14455162)
8. L . Bahl, P. Brown, P. de Souza, and R. Mercer, "Maximum mutual information estimation of hidden Markov model parameters for speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Tokyo, Japan, May 1986, pp. 49-52. (Pubitemid 17484870)
9. V . Valtchev, J. J. Odell, P. C. Woodland, and S. J. Young, "MMIE training of large vocabulary recognition systems," Speech Commun., vol. 22, no. 4, pp. 303-314, 1997. (Pubitemid 127433601)
10. P. C. Woodland and D. Povey, "Large scale discriminative training of hidden Markov models for speech recognition," Comput. Speech Lang., vol. 16, no. 1, pp. 25-48, 2002.
11. H . Ney, "On the relationship between classification error bounds and training criteria in statistical pattern recognition," in Proc. Iberian Conf. Pattern Recognition and Image Analysis, Puerto de Andratx, Spain, June 2003, pp. 636-645.
12. V . Goel and W. Byrne, "Minimum Bayes-risk automatic speech recognition," Comput. Speech Lang., vol. 14, no. 2, pp. 115-135, 2000.
13. R . Schlüter, W. Macherey, B. Müller, and H. Ney, "Comparison of discriminative training criteria and optimization methods for speech recognition," Speech Commun., vol. 34, no. 3, pp. 287-310, 2001. (Pubitemid 32284868)
14. B. Juang, W. Chou, and C. Lee, "Minimum classification error rate methods for speech recognition," IEEE Trans. Speech Audio Processing, vol. 5, no. 3, pp. 257-265, 1997. (Pubitemid 127745998)
15. W. Macherey, "Discriminative training and acoustic modeling for automatic speech recognition," Ph.D. dissertation, RWTH Aachen Univ., Aachen, Germany, 2010.
16. E. McDermott, T. Hazen, J. L. Roux, A. Nakamura, and S. Katagiri, "Discriminative training for large vocabulary speech recognition using minimum classification error," IEEE Trans. Audio Speech Lang. Processing, vol. 15, no. 1, pp. 203-223, 2007.
17. D. Povey, M. Gales, and P. Woodland, "Discriminative MAP for acoustic model adaptation," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Hong Kong, China, Apr. 2003, pp. 312-315.
18. M. Gibson, "Minimum Bayes risk acoustic model estimation and adaptation," Ph.D. dissertation, Univ. Sheffield, UK, 2008.
19. J. Zheng and A. Stolcke, "Improved discriminative training using phone lattices," in Proc. Interspeech, Lisbon, Portugal, Sept. 2005, pp. 2125-2128. (Pubitemid 43908513)
20. G. Heigold, "A log-linear discriminative modeling framework for speech recognition," Ph.D. dissertation, RWTH Aachen Univ., Aachen, Germany, June 2010.
21. S. Chen and R. Rosenfeld, "A Gaussian prior for smoothing maximum entropy models," Computer Science Dept., Carnegie Mellon Univ., Tech. Rep. CMUCS-99-108, 1999.
22. Y. Yin and H. Jiang, "A fast optimization method for large margin estimation of HMMs based on second order cone programming," in Proc. Interspeech, Antwerp, Belgium, Aug. 2007, pp. 34-37.
23. F. Sha and L. Saul, "Comparison of large margin training to other discriminative methods for phonetic recognition by hidden Markov models," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Honolulu, HI, Apr. 2007, pp. 313-316.
24. G. Heigold, T. Deselaers, R. Schlüter, and H. Ney, "Modified MMI/MPE: A direct evaluation of the margin in speech recognition," in Proc. Int. Conf. Machine Learning (ICML), Helsinki, Finland, July 2008, pp. 384-391.
25. G. H eigold, P. Dreuw, S. Hahn, R. Schlüter, and H. Ney, "Margin-based discriminative training for string recognition," IEEE J. Select. Topics Signal Processing- Stat. Learning Methods Speech Lang. Processing, vol. 4, no. 6, pp. 917-925, Dec. 2010.
26. D. Po vey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah, "Boosted MMI for model and feature-space discriminative training," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Las Vegas, NV, Apr. 2008, pp. 4057-4060.
27. Y. Al tun, I. Tsochantaridis, and T. Hofmann, "Hidden Markov support vector machines," in Proc. Int. Conf. Machine Learning (ICML), Washington, DC, Aug. 2003, pp. 3-10.
28. X. He , L. Deng, and W. Chou, "Discriminative learning in sequential pattern recognition-A unifying review for optimization-oriented speech recognition," IEEE Signal Processing Mag., vol. 25, no. 5, pp. 14-36, 2008.
29. E. Mc Dermott, S. Watanabe, and A. Nakamura, "Discriminative training based on an integrated view of MPE and MMI in margin and error space," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Dallas, TX, 2010, pp. 4894-4897.
30. H. Bo urlard and N. Morgan, Connectionist speech recognition. Norwell, MA: Kluwer, 1994.
31. G. Da hl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large vocabulary speech recognition," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 1, pp. 30-42, 2012.
32. H.-K. J. Kuo and Y. Gao, "Maximum entropy direct models for speech recognition," IEEE Trans. Audio Speech Lang. Processing, vol. 14, no. 3, pp. 873-881, 2006. (Pubitemid 46547649)
33. Y. Hi fny and S. Renals, "Speech recognition using augmented conditional random fields," IEEE Trans. Audio. Speech Lang. Processing, vol. 17, no. 2, pp. 354-365, 2009.
34. D. Yu , L. Deng, and A. Acero, "Using continuous features in the maximum entropy model," Pattern Recognit. Lett., vol. 30, no. 14, pp. 1295-1300, 2009.
35. S. Wi esler, R. Schlüter, and H. Ney, "A convergence analysis of log-linear training and its application to speech recognition," in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Hawaii, Dec. 2011, pp. 1-6.
36. J. Laf ferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data," in Proc. Int. Conf. Machine Learning (ICML), San Francisco, CA, June-July 2001, pp. 282-289.
37. J. Mor ris and E. Fosler-Lussier, "CRANDEM: conditional random fields for word recognition," in Proc. Interspeech, Brighton, England, Sept. 2009, pp. 3063-3066.
38. A. Gun awardana, M. Mahajan, A. Acero, and J. Platt, "Hidden conditional random fields for phone classification," in Proc. Interspeech, Lisbon, Portugal, Sept. 2005, pp. 117-120.
39. G. Zwe ig and P. Nguyen, "A segmental CRF approach to large vocabulary continuous speech recognition," in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Merano, Italy, Dec. 2009, pp. 152-157.
40. J.-T. Chien and C.-H. Chueh, "Joint acoustic and language modeling for speech recognition," Speech Commun., vol. 52, no. 3, pp. 223-235, Oct. 2010.
41. M. Leh r and I. Shafran, "Learning a discriminative weighted finite-state transducer for speech recognition," IEEE Trans. Audio Speech Lang Processing, vol. 19, no. 5, pp. 1360-1367, July 2011.
42. G. Hei gold, H. Ney, P. Lehnen, T. Gass, and R. Schlüter, "Equivalence of generative and log-linear models," IEEE Trans. Audio Speech Lang. Processing, vol. 19, no. 5, pp. 1138-1148, July 2011.
43. J. L. R oux and E. McDermott, "Optimization methods for discriminative training," in Proc. Interspeech, Lisbon, Portugal, Sept. 2005, pp. 3341-3344.
44. Y. Norm andin, "Hidden Markov models, maximum mutual information, and the speech recognition problem," Ph.D. dissertation, McGill Univ., Montreal, Canada, 1991.
45. R. Schlüter, "Investigations on discriminative training criteria," Ph.D. dissertation, RWTH Aachen University, Aachen, Germany, Sept. 2000.
46. M. Riedm iller and H. Braun, "A direct adaptive method for faster backpropagation learning: The Rprop algorithm," in Proc. IEEE Int. Conf. Neural Networks (ICNN), San Francisco, CA, Mar.-Apr. 1993, pp. 586-591.
47. C. Igel and M. Hüsken, "Empirical evaluation of the improved Rprop learning algorithms," Neurocomputing, vol. 50, pp. 105-123, Jan. 2003. (Pubitemid 36131033)
48. B. Hoffme ister, G. Heigold, D. Rybach, R. Schlüter, and H. Ney, "WFST enabled solutions to ASR problems: Beyond HMM decoding," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 2, pp. 551-564, 2011.
49. M. Mohri, "Weighted automata algorithms," in Handbook of Weighted Automata, M. Droste, W. Kuich, and H. Vogler, Eds. New York: Springer-Verlag, 2009, pp. 213-254.
50. S. Ortman ns, H. Ney, and X. Aubert, "A word graph algorithm for large vocabulary continuous speech recognition," Comput. Speech Lang., vol. 11, no. 1, pp. 43-72, Jan. 1997. (Pubitemid 127375893)
51. A. Ljolje , F. Pereira, and M. Riley, "Efficient general lattice generation and rescoring," in Proc. European Conf. Speech Communication and Technology (Eurospeech), Budapest, Hungary, Sept. 1999, pp. 1251-1254.
52. J. Eisner , "Expectation semirings: Flexible EM for finite-state transducers," in Proc. Int. Workshop on Finite-State Methods and Natural Language Processing (FSMNLP), Helsinki, Finland, Aug. 2001, pp. 1-5.
53. Z. Li and J. Eisner, "First- and second-order expectation semirings with applications to minimum-risk training on translation forests," in Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), Singapore, Aug. 2009, pp. 40-51.
54. A. Ng and M. Jordan, "On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes," in Proc. Advances in Neural Information Processing Systems (NIPS), Dec. 2002, pp. 841-848.