Pattern recognition using a family of design algorithms based upon the generalized probabilistic descent method

Proceedings of the IEEE

Volumn 86, Issue 11, 1998, Pages 2345-2372

  Katagiri, Shigeru ^a,c,d,e,f,g   Juang, Biing Hwang ^b   Lee, Chin Hui ^b  

^a ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

^b LUCENT TECHNOLOGIES   (United States)

^c IEEE   (Japan)

^d ACOUSTICAL SOCIETY OF AMERICA   (United States)

^e Inst Electronics Info Commun E   (Japan)

^f Institute of Electronics   (Japan)

^g Japanese Society for Artificial Intelligence   (Japan)

Author keywords

Bayes decision theory; Discriminant function approach; Discriminative feature extraction; Discriminative training; Generalized probabilistic descent method; Minimum classification error learning; Pattern recognition; Speech recognition

Indexed keywords

DECISION THEORY; ERROR ANALYSIS; FUNCTIONS; LEARNING ALGORITHMS; PROBABILITY;

BAYES DECISION THEORY; DISCRIMINANT FUNCTION METHOD; GENERALIZED PROBABILISTIC DESCENT METHOD (GPD); MINIMUM CLASSIFICATION ERROR LEARNING;

PATTERN RECOGNITION;

EID: 0032203256     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/5.726793     Document Type: Article

References (109)

1. S. Amari, "A theory of adaptive pattern classifiers," IEEE Trans. Electron. Comput., vol. EC-16, pp. 299-307, Mar. 1967.
2. _, Information Theory II - Geometrical Theory of Information, (in Japanese). Tokyo: Kyoritsu, 1968.
3. _, Neural Network Model and Connectionism (in Japanese). Tokyo: Tokyo Univ. Press, 1989.
4. A. Ando and K. Ozeki, "A clustering algorithm to minimize recognition error function," IEICE Trans. A, vol. J74-A, no. 3, pp. 360-367, Mar. 1991.
5. H. Asou, Neural Network Information Processing-Introduction to Connectionism, or Toward Soft Symbolic Representation (in Japanese). Tokyo: Sangyo Tosho, 1988.
6. H. Asou and N. Otsu, "Nonlinear data analysis and multilayer perceptrons," in Proc. IEEE IJCNN, vol. 2, 1989, pp. 411-415.
7. L. Bahl, P. Brown, P. de Souza, and R. Mercer, "Maximum mutual information estimation of hidden Markov model parameters for speech recognition," in Proc. IEEE ICASSP'86, vol. 1, pp. 49-52.
8. _, "A new algorithm for the estimation of hidden Markov model parameters," in Proc. IEEE. ICASSP'98, vol. 1, pp. 493-496.
9. L. E. Baum and T. Petrie, "Statistical inference for probabilistic functions of finite state Markov chains," Ann. Math. Stat., vol. 37, pp. 1554-1563, 1966.
10. L. E. Baum and G. R. Sell, "Growth functions for transformations on manifolds," Pacific J. Math., vol. 27, no. 2, pp. 211-227, 1968.
11. Y. Bengio, R. De Mori, G. Flammia, and R. Kompe, "Global optimization of a neural network-hidden Markov model hybrid," IEEE Trans. Neural Networks, vol. 3, no. 2, pp. 252-259, Mar. 1992.
12. A. Biem and S. Katagiri, "Feature extraction based on minimum classification error/generalized probabilistic descent method," in Proc. IEEE ICASSP'93, vol. 2, pp. 275-278.
13. A. Biem, S. Katagiri, and B.-H. Juang, "Discriminative feature extraction for speech recognition," in Proc. 1993 IEEE Workshop Neural Networks for Signal Processing III, pp. 392-401.
14. A. Biem, E. McDermott, and S. Katagiri, "Discriminative feature extraction to filter bank design," in Proc. 1996 IEEE Workshop Neural Network for Signal Processing VI, pp. 273-282.
15. A. Biem, S. Katagiri, and B.-H. Juang, "Pattern recognition using discriminative feature extraction," IEEE Trans. Signal Processing, vol. 45, pp. 500-504, Feb. 1997.
16. H. Bourlard and C. Wellekens, "Links between Markov models and multilayer perceptrons," IEEE Trans. Pattern Anal. Machine Intell., vol. 12, pp. 1167-1178, Dec. 1990.
17. H. Bourlard and N. Morgan, Connectionist Speech Recognition - A Hybrid Approach. Norwell, MA: Kluwer, 1994.
18. J. Bridle, "Alpha-nets: A recurrent 'neural' network architecture with a hidden Markov model interpretation," Speech Commun., vol. 9, pp. 83-92, 1990.
19. P.-C. Chang and B.-H. Juang, "Discriminative template training for dynamic programming speech recognition," in Proc. IEEE ICASSP'92, vol. 1, pp. 493-496.
20. _, "Discriminative training of dynamic programming based speech recognizers," IEEE Trans. Speech Audio Processing, vol. 1, pp. 135-143, Feb. 1993.
21. J.-K. Chen and F. K. Soong, "An N-best candidates-based discriminative training for speech recognition applications," IEEE Trans. Speech Audio Processing, vol. 2, pp. 206-216, Jan. 1994.
22. R.-Chengalvarayan and L. Deng, "Use of generalized dynamic feature parameters for speech recognition," IEEE Trans. Speech Audio Processing, vol. 5, pp. 232-242, Mar. 1997.
23. _, "HMM-based speech recognition using state-dependent, discriminatively derived transforms on Mel-warped DFT features," IEEE Trans. Speech Audio Processing, vol. 5, pp. 243-256, Mar. 1997.
24. W. Chou, B.-H. Juang, and C.-H. Lee, "Segmental GPD training of HMM based speech recognition," in Proc. IEEE ICASSP'92, vol. 1, pp. 473-476.
25. _, "Minimum error rate training based on N-best string models," in Proc. IEEE ICASSP'93, vol. 2, pp. 652-655.
26. W. Chou, C.-H. Lee, and B.-H. Juang, "Minimum error rate training of inter-word context dependent acoustic model units in speech recognition," in Proc. ICSLP'94, pp. 439-442.
27. W. Chou, M. G. Rahim, and E. Buhrke, "Signal conditioned minimum error rate training," in Proc. EUROSPEECH'95, pp. 495-498.
28. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Roy. Statist. Soc., vol. 39, no. 1, pp. 1-38, 1977.
29. R. Duda, and P. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
30. S. Euler, "Integrated optimization of feature transformation for speech recoanition," in Proc. EUROSPEECH'95, pp. 109-112.
31. K. R. Farrell, R. J. Mammone, and K. T. Assaleh, "Speaker recognition using neural networks and conventional classifiers," IEEE Trans. Speech Audio Processing, vol. 2, no. 1, pt. 2, pp. 194-205, Jan. 1994.
32. K. Fukunaga, Introduction to Statistical Pattern Recognition. New York: Academic, 1972.
33. K. Funahashi, "On the approximate realization of continuous mappings by neural networks," Neural Networks, vol. 2, no. 3, pp. 183-191, 1989.
34. Y.-Q. Gao, T.-Y. Huang, and D.-W. Chen, "HMM-based warping in neural networks," in Proc. IEEE ICASSP'90, vol. 1, pp. 501-504.
35. S. Geman and D. Geman, "Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images," IEEE Trans. Parallel Anal. Machine Intell., vol. PAMI-6, pp. 721-741, June 1984.
36. H. Gish, "A probabilistic approach to the understanding and training of neural network classifiers," Proc. IEEE ICASSP'90, vol. 3, pp. 1361-1364.
37. P. Haffner, M. Franzini, and A. Waibel, "Integrating time alignment and neural networks for high performance continuous speech recognition," in Proc. IEEE ICASSP'91, pp. 105-108.
38. P. Haffner, "A new probabilistic framework for connectionist time alignment," in Proc. ICSLP'94, pp. 1559-1562.
39. J. Hampshire II and A. Waibel, "A novel objective function for improved phoneme recognition using time-delay neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 216-228, Mar. 1990.
40. E. Hartman, J. Keeler, and J. Kowalski, "Layered neural networks with Gaussian hidden units as universal approximations," Neural Computation, vol. 2, pp. 210-215, 1990.
41. J. Hernando, J. Ayarte, and E. Monte, "Optimization of speech parameter weighting for CDHMM word recognition," in Proc. EUROSPEECH'95, pp. 105-108.
42. J. Hopfield and D. Tank, "Neural computation of decisions in optimization problems," in Biological Cybernetics, vol. 52. New York: Springer-Verlag, pp. 141-152, 1985.
43. D. Howell, "The multi-layer perceptron as a discriminating post processor for hidden Markov networks," in Proc. 7th FASE Symp., 1988, pp. 1389-1396.
44. X. Huang, Y. Ariki, and M. Jack, Hidden Markov Models For Speech Recognition. Edinburg, U.K.: Edinburg Univ. Press, 1990.
45. X. Huang, M. Belin, F. Alleva, and M. Hwang, "Unified stochastic engine (USE) for speech recognition," in Proc. IEEE ICASSP'93, vol. 2, pp. 636-639.
46. T. Iijinia, Pattern Recognition Theory, (in Japanese). Tokyo: Morikita, 1989.
47. H. Iwamida, S. Katagiri, E. McDermott, and Y. Tohkura, "A hybrid speech recognition system using HMM's with an LVQ-trained codebook," ASJ, J. Acoustical Soc. Japan (E), vol. 11, no. 5, pp. 277-286, Sept. 1990.
48. H. Iwamida, S. Katagiri, and E. McDermott, "Re-evaluation of LVQ-HMM hybrid algorithm," ASJ, J. Acoustical Soc. Japan (E), vol. 14, no. 4, pp. 267-274, July 1993.
49. F. Jelinek, "Self-organized language modeling for speech recognition," IBM T. J. Watson Research Center, Yorktown Heights, NY, Tech. Rep., 1985.
50. B.-H. Juang and L. Rabiner, "The segmental K-means algorithm for estimating parameters of hidden Markov models," IEEE Trans. Audio Speech Signal Processing, vol. 38, pp. 1639-1641, Sept. 1990.
51. B.-H. Juang and S. Katagiri, "Discriminative training," ASJ, J. Acoustical Soc. Japan (E), vol. 13, no. 6, pp. 333-339, Nov. 1992.
52. _, "Discriminative learning for minimum error classification," IEEE Trans. Signal Processing., vol. 40, pp. 3043-3054, Dec. 1992.
53. B.-H. Juang, W. Chou, and C.-H. Lee, "Minimum classification error rate methods for speech recognition," IEEE Trans. Speech Audio Processing., vol. 5, pp. 257-265, Mar. 1997.
54. S. Katagiri, C.-H. Lee, and B.-H. Juang, "A generalized probabilistic descent method," in Proc. ASJ Autumn Conf., 1990, pp. 141-142.
55. _, "Discriminative multi-layer feed-forward networks," in Proc. IEEE Workshop Neural Networks for Signal Processing, 1991, pp. 11-20.
56. _, "New discriminative training algorithms based on the generalized probabilistic descent method," in Proc.- IEEE Workshop Neural Networks for Signal Processing, 1991, pp. 299-308.
57. S. Katagiri and C.-H. Lee, "A new hybrid algorithm for speech recognition based on HMM segmentation and learning vector quantization," IEEE Trans. Speech Audio Processing, vol. 1, pp. 421-430, Apr. 1993.
58. S. Katagiri, B.-H. Juang, and A. Biem, "Discriminative feature extraction," in Artificial Neural Networks for Speech and Vision, R. Mammone, Ed. London, U.K.: Chapman and Hall, 1994, pp. 278-293.
59. S. Katagiri, "A unified approach to pattern recognition," in Proc. ISANN'94, pp. 561-570.
60. D. Kimber, M. Bush, and G. Tajchman, "Speaker-independent vowel classification using hidden Markov models and LVQ2," in Proc. IEEE ICASSP'90, vol. 1, pp. 497-500.
61. T. Kohonen, 'The self-organizing map," Proc. IEEE, vol. 78, pp. 1464-1480, Sept. 1990.
62. T. Komori and S. Katagiri, "GPD training of dynamic programming-based speech recognizers," ASJ, J. Acoustical Soc. Japan (E), vol. 13, no. 6, pp. 341-349, Nov. 1992.
63. _, "A minimum error approach to spotting-based pattern recognition," IEICE Trans. Inform. Syst., vol. E78-D, no. 8, pp. 1032-1043, Aug. 1995.
64. _, "A novel spotting-based approach to continuous speech recognition: Minimum error classification of keyword-sequences," ASJ, J. Acoustical Soc. Japan (E), vol. 16, no. 3, pp. 147-157, May 1995.
65. C.-H. Lee, F. K. Soong, and K. K. Paliwal, Eds., Automatic Speech and Speaker Recognition. Norwell, MA: Kluwer, 1996.
66. C.-S. Liu, C.-H. Lee, W. Chou, B.-H. Juang, and A; Rosenberg, "A study on minimum error discriminative training for speaker recognition," J. Acoustical Soc. Amer., vol. 97, no. 1, pp. 637-648, Jan. 1995.
67. S. Makino, M. Endo, T. Sone, and K. Kido, "Recognition of phonemes in continuous speech using a modified LVQ2 method," ASJ, J. Acoustical Soc. Japan (E), vol. 13, no. 6, pp. 351-360, Nov. 1992.
68. T. Matsui and S. Furui, "A study of speaker adaptation based on minimum classification error training," in Proc. EUROSPEECH'95, pp. 81-84.
69. E. McDermott, "LVQ3 for phoneme recognition," in Proc. ASJ Spring Conf., 1990, pp. 151-152.
70. E. McDermott and S. Katagiri, "LVQ-based shift-tolerant phoneme recognition," IEEE Trans. Signal Processing, vol. 39, pp. 1398-1411, June 1991.
71. _, "Prototype-based MCE/GPD training for word spotting and connected word recognition," in Proc. IEEE ICASSP'93, vol. 2, pp. 291-294.
72. _, "Prototype-based MCE/GPD training for various speech units," Comput. Speech Language, vol. 8, pp. 351-368, 1994.
73. _, "String-level MCE for continuous phoneme recognition," in Proc. EUROSPEECH'97, vol. 1, pp. 123-126.
74. M. Miyatake, H. Sawai, Y. Minami, and K. Shikano, "Integrated training for spotting Japanese phonemes using large phonemic. time-delay neural networks," in Proc. IEEE ICASSP'90, vol. 1, pp. 449-452.
75. S. Mizuta and K. Nakajima, "A discriminative training method for continuous mixture density HMM's and its implementation to recosnize noisy speech," ASJ, J. Acoustical Soc. Japan (E), vol. 13, no. 6, pp. 389-393, Nov. 1992.
76. H. Mizutani, "Discriminative learning for minimum error classification with reject options," (in Japanese), IEICE, Tokyo, Tech. Rep. PRMU97-245, 1998.
77. N. Morgan and H. A. Bourlard, "Neural networks for statistical recognition of continuous speech," Proc. IEEE, vol. 83, pp. 742-770, May 1995.
78. L. Niles and H. Silverman, "Combining hidden Markov model and neural network classifier," in Proc. IEEE ICASSP'90, vol. 1, pp. 417-420.
79. N. Nilsson, The Mathematical Foundations of Learning Machines. San Mateo, CA: Morgan Kaufmann, 1990.
80. Y. Normandin, R.- Cardin, and R. De Mon, "High-performance connected digit recognition using maximum mutual information estimation," IEEE Trans. Speech Audio Processing, vol. 2, pp. 299-311, Feb. 1994.
81. E. Oja, Subspace Methods of Pattern Recognition. Letchworth: Research Studies Press, 1983.
82. K. K. Paliwal, M. Bacchiani, and Y. Sagisaka, "Minimum classification error training algorithm for feature extractor and pattern classifier in speech recognition," in Proc. EUROSPEECH'95, pp. 541-544.
83. Y.-H. Pao, Adaptive Pattern Recognition and Neural Networks. Reading, MA: Addison-Wesley, 1989.
84. J. Park and I. Sandberg, "Universal approximation using radialbasis-function networks," Neural Computation, vol. 3, pp. 246-257, 1991.
85. L. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993.
86. M. G. Rahim and C.-H. Lee, "Simultaneous ANN feature and HMM recognizer design using string-based minimum classification error (MCE) training," in Proc. ICSLP'96, pp. 1824-1827.
87. D. Rainton and S. Sagayama, "Minimum error classification training of HMM's-Implementation details and experimental results," ASJ, J. Acoustical Soc. Japan (E), vol. 13, no. 6, pp. 379-387, Nov. 1992.
88. A. J. Robinson, "An application of recurrent nets to phone probability estimation," IEEE Trans. Neural Networks, vol. 5, pp. 298-305, Mar. 1994.
89. R. C. Rose, B.-H. Juang, and C.-H. Lee, "A training procedure for verifying string hypothesis in continuous speech recognition," in Proc. ICASSP'95 pp. 281-284.
90. R. C. Rose, "Word spotting from continuous speech utterances," in Automatic Speech and Speaker Recognition: Advanced Topics, C.-H. Lee, F. K. Soong, and K. K. Paliwal, Eds. Norwell, MA: Kluwer, 1996, pp. 303-329.
91. D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, Parallel Distributed Processing. Cambridge, MA: MIT, 1986.
92. H. Sakoe, R. Isotani, K. Yoshida, K. Iso, and T. Watanabe, "Speaker-independent word recognition using dynamic programming neural networks," in Proc. IEEE ICASSP'89, vol. 1, pp. 29-32.
93. A. Sato and K. Yamada, "A learning method for definite canonicalization based on minimum classification error," (in Japanese), Tokyo, IEICE, Tech. Rep. PRMU97-244, 1998.
94. A. Setlur and T. Jacobs, "Results of a Speaker verification service trial using HMM models," in Proc. EUROSPEECH'95, pp. 639-642.
95. H. Sorenson and D. Alspach, "Recursive Baysian estimation using Gaussian sums," Automatica, vol. 7, pp. 465-479, 1971.
96. K.-Y. Su, T.-H. Chiang, and Y.-C. Lin, "A unified framework to incorporate speech and language information in spoken language processing," in Proc. ICASSP'92, vol. 1, pp. 185-188.
97. K.-Y. Su and C.-H. Lee, "Speech recognition using weighted HMM and subspace projection approaches," in IEEE Trans. Speech Audio Processing, vol. 2, pp. 69-79, Jan. 1994.
98. M. Sugiyama and K. Kurinami, "Minimal classification error optimization for a speaker mapping neural network," in Proc. IEEE Workshop Neural Networks for Signal Processing II, 1992, pp. 233-242.
99. K. Unnikrishnan, J. Hopfield, and D. Tank, "Connected-digit speaker-dependent speech recognition using a neural network with time-delayed connections," IEEE Trans. Signal Processing, vol. 39, pp. 698-713, Mar. 1991.
100. V. Valtchev, J. J. Odell, P. C. Woodland, and S. J. Young, "Lattice-based discriminative training for large vocabulary speech recognition," in Proc. IEEE ICASSP'96, pp. 605-608.
101. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, "Phoneme recognition using time-delay neural networks," IEEE Trans. Audio Speech Signal Processing, vol. 37, pp. 328-339, Mar. 1989.
102. S. Watanabe, P. F. Lambert, C. A. Kulikowski, J. L. Buxton, and R. Walker, "Evaluation and selection of variables in pattern recognition," in Computer and Information Sciences II, J. T. Tou, Ed. New York: Academic, 1967, pp. 91-122.
103. H. Watanabe, T. Yamaguchi, and S. Katagiri, "A novel approach to pattern recognition based on discriminative metric design," in Proc. IEEE Workshop Neural Networks for Signal Processing V, 1995, pp. 48-57.
104. H. Watanabe and S. Katagiri, "Discriminative subspace method for minimum error pattern recognition," in Proc. IEEE Workshop Neural Networks for Signal Processing V, 1995, pp. 77-86.
105. H. Watanabe, T. Yamaguchi, and S. Katagiri, "Discriminative metric design for robust pattern recognition," IEEE Trans. Signal Processing, vol. 45, pp. 2655-2662, Nov. 1997.
106. H. Watanabe and S. Katagiri, "Subspace method for minimum error pattern recognition," IEICE Trans. Inform. Syst., vol. E80D, no. 12, pp. 1195-1204, Dec. 1997.
107. H. Watanabe, Y. Matsumoto, and S. Katagiri, "Minimum detection error training for acoustic sisnal monitoring," in Proc. IEEE ICASSP'98, vol. 2, pp. 1193-1196.
108. S. Young, "A review of large-vocabulary continuous-speech recognition," IEEE Signal Processing Mag., vol. 13, pp. 45-57, May 1996.
109. G. Yu, W. Russel, R. Schwanz, and J. Makhoul, "Discriminant continuous speech recognition," in Proc. IEEE ICASSP'90, vol. 2, pp. 685-688.