Discriminative training for large-vocabulary speech recognition using minimum classification error

IEEE Transactions on Audio, Speech and Language Processing

Volumn 15, Issue 1, 2007, Pages 203-223

  McDermott, Erik ^a   Hazen, Timothy J ^b   Le Roux, Jonathan Le ^c,d   Nakamura, Atsushi ^a   Katagiri, Shigeru ^a  

^a NTT Communication Science Laboratories   (Japan)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c TELECOM PARISTECH   (France)

^d UNIVERSITY OF TOKYO   (Japan)

Author keywords

Discriminative training; Pattern recognition; Speech recognition

Indexed keywords

CORPUS OF SPONTANEOUS JAPANESE; CRITERION FUNCTIONS; DISCRIMINATIVE TRAINING; JUPITER; MINIMUM CLASSIFICATION ERRORS; NAME RECOGNITION; PATTERN RECOGNITION PROBLEMS; PRACTICAL IMPLEMENTATIONS; RECOGNITION ACCURACIES; RECOGNITION PERFORMANCE; RELATIVE REDUCTIONS; SPEECH TRANSCRIPTIONS; SPEED-UP; WEATHER INFORMATIONS;

HIDDEN MARKOV MODELS; MAXIMUM LIKELIHOOD; OPTIMIZATION; SPEECH ANALYSIS; SPEECH TRANSMISSION; TELEPHONE; TELEPHONE SETS; TRANSCRIPTION;

SPEECH RECOGNITION;

EID: 34547522070     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2006.876778     Document Type: Article

References (75)

1. B. M. D. Azzopardi, S. Semnani, and R. Wiseman, "Improving accuracy of telephony-based, speaker-independent speech recognition," in Int. Conf. Spoken Language Processing, 1998, vol. 2, pp. 301-304.
2. J. Glass and T. Hazen, "Telephone-based conversational speech recognition in the JUPITER domain," in Int. Conf. Spoken Language Processing, 1998, vol. 4, pp. 1327-1330.
3. P. Brown, "The acoustic-modeling problem in automatic speech recognition," Ph.D. dissertation, Dept. Comput. Sci., Carnegie Mellon Univ., Pittsburgh, PA, 1987.
4. T. Applebaum and B. B. Hanson, "Enhancing the discrimination of speaker independent hidden Markov models with corrective training," in Proc. IEEE ICASSP, 1989, vol. 1, pp. 302-305.
5. H. Franco and A. Serralheiro, "Training HMMs using a minimum error approach," in Proc. IEEE ICASSP, 1990, vol. 1, pp. 357-360.
6. A. Ljolje, Y. Ephraim, and L. Rabiner, "Estimation of hidden Markov model parameters by minimizing empirical error rate," in Proc. IEEE ICASSP, 1990, vol. 2, pp. 709-712.
7. S. Katagiri, C.-H. Lee, and B.-H. Juang, "New discriminative training algorithms based on the generalized descent method," in Proc. IEEE Worshop Neural Networks for Signal Processing, 1991, pp. 299-308.
8. H. Gish, "A minimum classification error, maximum likelihood, neural network," in Proc. IEEE ICASSP, 1992, vol. 2, pp. 289-292.
9. J. Hampshire, "A differential theory of learning for efficient statistical pattern recognition," Ph.D. dissertation, Dept. Elect. Comput. Eng., Carnegie Mellon Univ., Pittsburgh, PA, 1991.
10. V. Valtchev, J. J. Odell, P. C. Woodland, and S. J. Young, "Lattice- based discriminative training for large vocabulary speech recognition," in Proc. Int. Conf. Spoken Language Processing, 1996, vol. 2, pp. 605-609.
11. P. Woodland and D. Povey, "Large scale discriminative training of hidden Markov models for speech recognition," Comput. Speech Lang., vol. 16, pp. 25-47, 2002.
12. L. Nguyen and B. Xiang, "Light supervision in acoustic model training," in Proc. IEEE ICASSP, 2004, vol. 1, pp. 185-188.
13. G. Evermann, H. Chan, M. Gales, B. Jia, D. Mrva, P. Woodland, and K. Yu, "Training LVCSR systems on thousands of hours of data," in Proc. IEEE ICASSP, 2005, vol. 1, pp. 209-212.
14. E. McDermott, A. Biem, S. Tenpaku, and S. Katagiri, "Discriminative training for large vocabulary telephone-based name recognition," in Proc. IEEE ICASSP, 2000, vol. 6, pp. 3739-3742.
15. E. McDermott and T. J. Hazen, "Minimum classification error training of landmark models for real-time continuous speech recognition," in Proc. IEEE ICASSP, 2004, vol. 1, pp. 937-940.
16. E. McDermott and S. Katagiri, "Minimum classification error for large scale speech recognition tasks using weighted finite state transducers," in Proc. IEEE ICASSP, 2005, vol. 1, pp. 113-116.
17. S. Katagiri, C.-H. Lee, and B.-H. Juang, "A generalized probabilistic descent method," in Proc. Acoustical Society of Japan, Ser. Fall Meeting ser. Fall Meeting, Sep. 1990, pp. 141-142.
18. B.-H. Juang and S. Katagiri, "Discriminative Learning for minimum error classification," IEEE Trans. Acoust., Speech, Signal Process., vol. 40, no. 12, pp. 3043-3054, Dec. 1992.
19. W. Chou, B.-H. Juang, and C.-H. Lee, "Segmental GPD training of HMM based speech recognizer," in Proc. IEEE ICASSP, Mar. 1992, vol. 1, pp. 473-476.
20. S. Katagiri, B.-H. Juang, and C.-H. Lee, "Pattern recognition using a family of design algorithms based upon the generalized probabilistic descent method," Proc. IEEE, vol. 86, no. 11, pp. 2345-2373, Nov. 1998.
21. M. Mohri, F. Pereira, and M. Riley, "Weighted finite state transducers in speech recognition," in Proc. Automatic Speech Recognition Workshop, 2000, pp. 97-106.
22. E. McDermott, "Discriminative training for speech recognition," Ph.D. dissertation, School Eng., Waseda Univ., Tokyo, Japan, Mar. 1997.
23. E. McDermott and S. Katagiri, "String-level MCE for continuous phoneme recognition," in Proc. Eurospeech, Rhodes, Greece, Sep. 1997, pp. 123-126.
24. -, "Prototype based discriminative training for various speech units," Comput. Speech Lang., vol. 8, pp. 351-368, 1994.
25. W. Chou, C.-H. Lee, and B.-H. Juang, "Minimum error rate training based on N-best string models," in Proc. IEEE ICASSP, 1993, vol. 2, pp. 652-655.
26. W. Chou, C.-H. Lee, B.-H. Juang, and F.-K. Soong, "A minimum speech error rate pattern recognition approach to speech recognition," Int. J. Pattern Recognition Artificial Intelligence, Special Issue on Speech Recognition for Different Languages, vol. 8, no. 1, pp. 5-31, 1994.
27. R. Schlueter, W. Macherey, B. Muller, and H. Ney, "Comparison of discriminative training criteria and optimization methods for speech recognition," Speech Communication, vol. 34, no. 3, pp. 287-310, 2001.
28. H. Jiang, O. Siohan, F.-K. Soong, and C.-H. Lee, "A dynamic in-search discriminative training approach for large vocabulary speech recognition," in Proc. ICASSP, 2002, vol. 1, pp. 113-116.
29. T. Kawahara, H. Nanjo, T. Shinozaki, and S. Furui, "Benchmark test for speech recognition using the corpus of spontaneous japanese," in Proc. Spontaneous Speech Processing Recognition Workshop, Tokyo, Japan, 2003, pp. 135-138.
30. A. Robinson, "Application of recurrent nets to phone probability estimation," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 298-305, Mar. 1994.
31. Y. Normandin, R. Lacouture, and R. Cardin, "MMIE training for large vocabulary continuous speech recognition," in Proc. Int. Conf. Spoken Language Processing, 1994, vol. 3, pp. 1367-1370.
32. B. Liu, H. Jiang, J.-L. Zhou, and R.-H.Wang, "Discriminative training based on the criterion of least phone competing tokens for large vocabulary speech recognition," in Proc. IEEE ICASSP, 2005, vol. 1, pp. 117-120.
33. E. McDermott and S. Katagiri, "Aderivation of minimum classification error from the theoretical classification risk using Parzen estimation," Comput. Speech Lang., vol. 18, pp. 107-122, 2004.
34. R. Schlueter and H. Ney, "Model-based MCE bound to the true Bayes' error," IEEE Signal Process. Lett., vol. 8, no. 5, pp. 131-133, May 2001.
35. A. Biem, S. Katagiri, E. McDermott, and B.-H. Juang, "An application of discriminative feature extraction to filter-bank based speech recognition," IEEE Trans. Speech Audio Process., vol. 9, no. 2, pp. 96-110, Mar. 2001.
36. R. Chengalvarayan and L. Deng, "HMM-based speech recognition using state-dependent, discriminatively derived transforms on Mel-warped DFT features," IEEE Trans. Speech Audio Process., vol. 5, no. 3, pp. 243-256, May 1997.
37. L. Deng, K. Wang, and W. Chou, "Guest editorial," IEEE Signal Process. Mag., vol. 22, no. 5, pp. 12-14, Sep. 2005.
38. A. Biem, "Minimum classification error training of hidden Markov models for handwriting recognition," in Proc. IEEE ICASSP, 2001, vol. 3, pp. 1529-1532.
39. C. Yen, S.-S. Kuo, and C.-H. Lee, "Minimum error rate training for PHMM-based text recognition," IEEE Trans. Image Process., vol. 8, no. 8, pp. 1120-1124, Aug. 1999.
40. S. Gao, W. Wu, C.-H. Lee, and T.-S. Chua, "A maximal figure-ofmerit learning approach to text categorization," in Proc. ACM SIGIR Conf. Research and Development in Information Retrieval, 2003, pp. 174-181.
41. F. J. Och, "Minimum error rate training in statistical machine translation," in ACL: Proc. Assoc. Comput. Ling., 2003, pp. 160-167.
42. Y. Normandin, "Hidden Markov models, maximum mutual information estimation, and the speech recognition problem," Ph.D. dissertation, Dept. Elect. Eng., McGill Univ., Montreal, QC, Canada, 1991.
43. P. S. Gopalakrishnan, D. Kanevsky, A. Nadas, D. Nahamoo, and M. Picheny, "Decoder selection based on cross-entropies," in Proc. IEEE ICASSP, 1988, vol. 1, pp. 20-23.
44. D. Povey and P. Woodland, "Minimum phone error and I-smoothing for improved discriminative training," in Proc. IEEE ICASSP, 2002, vol. 1, pp. 105-108.
45. D. Povey, "Discriminative training for large vocabulary speech recognition," Ph.D. dissertation, Univ. Cambridge, Cambridge, U.K., 2004.
46. T. Hori, C. Hori, and Y. Minami, "Fast on-the-fly composition for weighted finite-state transducers in 1.8 million-word vocabulary continuous speech recognition," in Proc. Int. Conf. Spoken Language Processing, 2004, vol. 1, pp. 289-292.
47. W. Macherey, L. Haferkamp, R. Schlueter, and H. Ney, "Investigations on error minimizing training criteria for discriminative training in automatic speech recognition," in Proc. Eurospeech, 2005, pp. 2133-2136.
48. R. Schlueter,W.Macherey, S. Kanthak, H. Ney, and L.Welling, "Comparison of optimization methods for discriminative training criteria," in Proc. Eurospeech, 1997, vol. 1, pp. 15-18.
49. Q. Li and B.-H. Juang, "A new algorithm for fast discriminative training," in Proc. IEEE ICASSP, 2002, vol. 1, pp. 97-100.
50. X. He and W. Chou, "Minimum classification error linear regression for acoustic model adaptation of continuous density HMMs," in Proc. IEEE ICASSP, 2003, vol. 1, pp. 556-559.
51. S. Amari, "A theory of adaptive pattern classifiers," IEEE Trans. Electron. Comput., vol. EC-16, no. 3, pp. 299-307, Mar. 1967.
52. L. Bottou, "Une approche théorique de l'apprentissage connectionniste: Application à la reconnaissane de la parole," Ph.D. dissertation, Univ. de Paris Sud, Paris, France, 1991.
53. S. E. Fahlman, An empirical study of learning speed in back-propagation networks Canergie Mellon Univ., Pittsburgh, PA, 1988, Tech. Rep..
54. S. Kapadia, V.Valtchev, and S. J.Young, "MMItraining for continuous phoneme recognition on the TIMIT database," in Proc. ICASSP, 1993, vol. 2, pp. 491-494.
55. R. Battiti, "First- and second- order methods for learning: Between steepest descent and newton's method," Neural Comput., vol. 4, pp. 141-166, 1992.
56. J. Leroux and E. McDermott, "Optimization methods for discriminative training," in Proc. Eurospeech, 2005, pp. 3341-3344.
57. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. IEEE Intl. Conf. Neural Networks, San Francisco, CA, 1993, pp. 586-591 [Online]. Available: http://www.citeseer.ist.psu.edu/riedmiller93direct.html
58. D. Kanevsky, "A generalization of the Baum algorithm to functions on nonlinear manifolds," in Proc. IEEE ICASSP, 1995, pp. 473-476.
59. K.-F. Lee and H.-W. Hon, "Speaker-independent phone recognition using hidden Markov models," IEEE Trans. Acoustics, Speech, Signal Process., vol. 37, no. 11, pp. 1641-1648, Nov. 1989.
60. P. Clarkson and P. Moreno, "On the use of support vector machines for phonetic classification," in Proc. IEEE ICASSP, 1999, vol. 2, pp. 585-588.
61. A. Halberstadt and J. Glass, "Heterogeneous measurements and multiple classifiers for speech recognition," in Proc. ICSLP, Sydney, Australia, Dec. 1998, pp. 995-998.
62. A. Gunawardana, M. Mahajan, A. Acero, and J. Platt, "Hidden conditional random fields for phone classification," in Proc. Eurospeech, 2005, pp. 1117-1120.
63. J. Odell, "The use of context in large vocabulary speech recognition," Ph.D. dissertation, Univ. Cambridge, Cambridge, U.K., 1995.
64. S. Young and P.Woodland, "State clustering in hidden Markov modelbased continuous speech recognition," Comput. Speech Lang., vol. 8, pp. 369-383, 1994.
65. L.Wang and P.Woodland, "Discriminative adaptive training using the MPE criterion," in Proc. ASRU, 2003, pp. 279-284.
66. H. Hermansky and N. Morgan, "RASTA processing of speech," IEEE Trans. Speech Audio Process., vol. 2, no. 4, pp. 578-589, Mar. 1994.
67. V. Zue et al., "JUPITER: A telephone-based conversational interface for weather information," IEEE Trans. Speech Audio Process., vol. 8, no. 1, pp. 85-96, Jan. 2000.
68. J. Glass, "A probabilistic framework for segment-based speech recognition," Comput. Speech Lang., vol. 17, no. 2-3, pp. 137-152, Apr.-Jul. 2003.
69. T. Hazen, S. Seneff, and J. Polifroni, "Recognition confidence scoring for use in speech understanding systems," Comput. Speech Lang., vol. 16, no. 1, pp. 49-67, Jan. 2002.
70. I. Bazzi and J. Glass, "A multi-class approach for modelling out-of-vocabulary words," in Proc. ICSLP, Denver, CO, Sep. 2002, pp. 1613-1616.
71. C. Meyer and G. Rose, "Improved noise robustness by corrective and rival training," in Proc. ICASSP, 2001, vol. 1, pp. 293-296.
72. D. Willett, "Error-weighted discriminative training for HMM parameter estimation," in Proc. Int. Conf. Spoken Language Processing, 2004, pp. 1661-1664.
73. J. E. Dennis and R. B. Schnabel, "Numerical methods for unconstrained optimization and nonlinear equations," in Classics in Applied Mathematics. Philadelphia, PA: SIAM, 1996.
74. L. E. Scales, Introduction to Non-Linear Optimization. New York: Macmillan, 1985.
75. D. Rainton and S. Sagayama, "Minimum error classification training of HMMs-Implementation details and experimental results," J. Acoust. Soc. Jpn., vol. 13, no. 6, pp. 379-387, 1992.