Optimization algorithms and applications for speech and language processing

IEEE Transactions on Audio, Speech and Language Processing

Volumn 21, Issue 11, 2013, Pages 2231-2243

  Wright, Stephen J ^a   Kanevsky, Dimitri ^b   Deng, Li ^c   He, Xiaodong ^c   Heigold, Georg ^d   Li, Haizhou ^e  

^a University of Wisconsin   (United States)

^b IBM RESEARCH   (United States)

^c MICROSOFT RESEARCH   (United States)

^d GOOGLE INC   (United States)

^e INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

Author keywords

Natural language processing; Optimization methods; Speech processing

Indexed keywords

AUTOMATIC SPEECH RECOGNITION; COMPUTATIONAL PROBLEM; NATURAL LANGUAGE PROCESSING; OPTIMIZATION ALGORITHMS; OPTIMIZATION FORMULATIONS; OPTIMIZATION METHOD; OPTIMIZATION PROCEDURES; OPTIMIZATION TECHNIQUES;

NATURAL LANGUAGE PROCESSING SYSTEMS; OPTIMIZATION; SPEECH PROCESSING;

ALGORITHMS;

EID: 84887037596     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2013.2283777     Document Type: Article

References (97)

1. L. E. Baum and T. Petrie, "Statistical inference for probabilistic functions of finite state Markov chains," Ann. Math. Statist., vol. 37, no. 6, pp. 1554-1563, 1966.
2. C. Leggetter and P.Woodland, "Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models," Comput., Speech, Lang., vol. 9, pp. 171-185, 1995.
3. R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. New York, NY, USA: Wiley, 2001.
4. P. C. Woodland and D. Povey, "Large scale discriminative training of hidden markov models for speech recognition," Comput. Speech, Lang., pp. 25-47, 2002.
5. D. Povey, "Discriminative training for large vocabulary speech recognition," Ph.D. dissertation, University of Cambridge, Cambridge,U.K., 2003.
6. L. Deng and D. O'Shaughnessy, Speech Processing-A Dynamic and Optimization-Oriented Approach. New York, NY, USA: Marcel Dekker, 2003.
7. W.Macherey and H. Ney, "A comparative study on maximum entropy and discriminative training for acoustic modeling in automatic speech recognition," in Proc. Eurospeech, 2003, pp. 493-496.
8. A. Gunawardana, M. Mahajan, A. Acero, and J. C. Platt, "Hidden conditional random fields for phone classification," in Interspeech, Sep. 2005.
9. S.Axelrod, V. Goel, R.A.Gopinath, P. A. Olsen, andK.Visweswariah, "Discriminative estimation of subspace constrained gaussian mixture models for speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 172-189, Jan. 2007.
10. X. He, L. Deng, and W. Chou, "Discriminative learning in sequential pattern recognition-a unifying review for optimization-oriented speech recognition," IEEE Signal Process. Mag., vol. 25, no. 5, pp. 14-36, Sep. 2008.
11. T. Sainath, B. Ramabhadran, D. Nahamoo, D. Kanevsky, and A. Sethy, "Exemplar-based sparse representation features for speech recognition," in Proc. Interspeech, 2010.
12. T. Sainath, B. Kingsbury, H. Soltau, and B. Ramabhadran, "Optimization techniques to improve training speech of deep belief networks for large speech tasks," IEEE Trans. Audio, Speech, Lang. Process., Spec. Iss. Large-Scale Optimization for Audio, Speech, Lang. Process., vol. 21, no. 11, Nov. 2013.
13. G. Heigold, S. Hahn, P. Lehnen, and H. Ney, "EM-style optimization of hidden conditional random fields for grapheme-to-phoneme conversion," in Proc. ICASSP, 2011, pp. 4920-4923.
14. V. Hautamäki, K. A. Lee, T. Kinnunen, B.Ma, and H. Li, "Optimizing the performance of spoken language recognition with discriminative training," IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 8, pp. 1622-1631, Aug. 2013.
15. T. P. Minka, "Algorithms for maximum-likelihood logistic regression," Carnegie Mellon Univ., Tech. Rep. 738, 2001.
16. W. Campbell, J. Campbell, D. Reynolds, E. Singer, and P. Torres-Carrasquillo, "Support vector machines for speaker and language recognition," Computer Speech Lang., pp. 210-229, Apr. 2006. (Pubitemid 41787537)
17. G. Saon and J.-T. Chien, "Bayesian sensing hidden Markov models," IEEE Trans. Audio, Speech Lang. Process., vol. 20, no. 1, pp. 43-54, Jan. 2012.
18. C. H. Lee and Q. Huo, "On adaptive decision rules and decision parameter adaptation for automatic speech recognition," Proc. IEEE, vol. 88, no. 8, pp. 1241-1269, Aug. 2000.
19. S.Watanabe and A. Nakamura, "Bayesian approaches to acousticmodeling: A Review," APSIPA Trans. Signal Inf. Process., vol. 1, 2012.
20. L. Xiao and L.Deng, "A geometric perspective of large-margin training of Gaussian models," IEEE Signal Process. Mag., vol. 27, no. 6, pp. 118-123, Nov. 2010.
21. T.-H. Chang, Z.-Q. Luo, L. Deng, and C.-Y. Chi, "A convex optimization method for joint mean and variance parameter estimation of large-margin CDHMM," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2008, pp. 4053-4056.
22. X. He and L. Deng, "Speech-centric information processing: An optimization-oriented approach," Proc. IEEE, vol. 101, no. 5, pp. 1116-1135, May 2013.
23. J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. ed. New York, NY, USA: Springer, 2006.
24. S. P. Boyd and L. Vandenberghe, Convex optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
25. B. T. Polyak, Introduction to Optimization. Ann Arbor, MI, USA: Optimization Software, 1987.
26. C. Liu, Y. Hu, L.-R. Dai, and H. Jiang, "Trust region-based optimization for maximum mutual information estimation of hmms in speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 8, pp. 2474-2485, Nov. 2011.
27. M. Gibson and T. Hain, "Error approximation and minimum phone error acoustic model estimation," IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no. 6, pp. 1269-1279, Aug. 2010.
28. S. Chen and R. Rosenfeld, A Gaussian prior for smoothing maximum entropymodels Comput. Sci.Dept., CarnegieMellonUniv., Tech. Rep. CMUCS-99-108, 1999.
29. D. Povey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah, "BoostedMMI for model and feature-space discriminative training," in Proc. IEEE Int. Conf. Acoustic, Speech, Signal Process., Las Vegas, NV, USA, Apr. 2008, pp. 4057-4060.
30. G. Heigold, "A log-linear discriminative modeling framework for speech recognition," Ph.D. dissertation, RWTH Aachen Univ., Aachen, Germany, 2010.
31. Y. Bar-Hillel, "The present status of automatic translation of languages," Adv. Comput., pp. 158-163, 1960.
32. P. Brown, S. Pietra, V. Pietra, and R.Mercer, "The mathematics of statistical machine translation: Parameter estimation," Comput. Linguist., vol. 19, no. 2, pp. 263-311, 1993.
33. P. Koehn, F. Och, and D.Marcu, "Statistical phrase-based translation," in Proc. HLT-NAACL, 2003.
34. F. Och, "Minimum error rate training in statistical machine translation," in Proc. ACL, 2003.
35. X. He, L. Deng, and W. Chou, "Speech recognition, machine translation, and speech translation-a unified discriminative learning paradigm," IEEE Signal Process. Mag., vol. 28, no. 5, pp. 126-133, Sep. 2011.
36. D. Xiong, M. Zhang, and H. Li, "A maximum-entropy segmentation model for statistical machine translation," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 8, pp. 2494-2505, Nov. 2011.
37. I. D. El-Kahlout and K. Oflazer, "Exploiting morphology and local word reordering in english-to-turkish phrase-based statistical machine translation," IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no. 6, pp. 1313-1322, Aug. 2010.
38. X. He and L. Deng, "Maximum expected bleu training of phrase and lexicon translation models," in Proc. ACL, Assoc. Comput. Linguist., 2012.
39. K. Papineni, S. Roukos, T. Ward, and W. Zhu, "Bleu: a method for evaluation of machine translation," inProc. 40th Annu. Meeting Assoc. Comput. Linguist., 2002, pp. 311-318.
40. T. Kinnunen and H. Li, "An overview of text-independent speaker recognition: from features to supervectors," Speech Commun., vol. 52, no. 1, pp. 12-40, 2010.
41. D. Reynolds, T. Quatieri, and R. Dunn, "Speaker verification using adapted gaussian mixture models," Digital Signal Process., vol. 10, no. 1, pp. 19-41, Jan. 2000. (Pubitemid 30592166)
42. P. Kenny, P. Ouellet, N. Dehak, V. Gupta, and P. Dumouchel, "A study of inter-speaker variability in speaker verification," IEEE Trans. Audio, Speech, Lang. Process., vol. 16, no. 5, pp. 980-988, Jul. 2008.
43. K. A. Lee, C. H. You, H. Li, T. Kinnunen, and K. C. Sim, "Using discrete probabilities with bhattacharyya measure for svm-based speaker verification," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 4, pp. 861-870, May 2011.
44. N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, "Front-end factor analysis for speaker verification," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 4, pp. 788-798, May 2011.
45. H. Li, B. Ma, and K. A. Lee, "Spoken language recognition: From fundamentals to practice," Proc. IEEE, vol. 101, no. 5, pp. 1136-1159, May 2013.
46. H. Li, B. Ma, and C.-H. Lee, "Vector-based spoken language classification," in Springer Handbook of Speech Processing, J. Benesty, M. Sondhi, and A. Huang, Eds. New York, NY, USA: Springer, 2007.
47. H. Li, B. Ma, and C.-H. Lee, "A vector space modeling approach to spoken language identification," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 271-284, Jan. 2007.
48. N. Brümmer and J. Preez, "Application-independent evaluation of speaker detection," Comput. Speech Lang., vol. 20, no. 2, pp. 230-275, 2006. (Pubitemid 41787538)
49. D. A. van Leeuwen and N. Brümmer, "An introduction to application independent evaluation of speaker recognition systems," in Speaker Classification, Lecture Notes in Computer Science/Artificial Intelligence, R. Müller, Ed. New York, NY, USA: Springer, 2007, vol. 4343.
50. H. Li and B.Ma, "TechWare: Speaker and spoken language recognition resources," IEEE Signal Process. Mag., vol. 27, no. 6, pp. 139-142, Nov. 2010.
51. A. F. Martin and A. N. Le, "NIST 2007 language recognition evaluation," in Proc. Odyssey: Speaker Lang. Recogn. Workshop, 2008, p. 016.
52. A. F. Martin and A. N. Le, "The current state of language recognition: NIST 2005 evaluation results," in Proc. Odyssey: Speaker Lang. Recogn. Workshop, 2006, pp. 1-6.
53. A. F. Martin and J. S. Garofolo, "NIST speech processing evaluations: Lvcsr, speaker recognition, language recognition," in Proc. IEEE Workshop Signal Process. Applicat. Public Security Forensics, 2007, pp. 1-7.
54. A. F. Martin and C. Greenberg, "NIST 2009 language recognition evaluation," in Proc. Odyssey: Speaker Lang. Recogn.Workshop, 2010, pp. 165-171.
55. D. Zhu, H. Li, B. Ma, and C. H. Lee, "Optimizing the performance of spoken language recognition with discriminative training," IEEE Trans. Audio, Speech, Lang. Process., vol. 16, no. 8, pp. 1642-1653, Nov. 2008.
56. B.-H. Juang, W. Chou, and C.-H. Lee, "Minimum classification error rate methods for speech recognition," IEEE Trans. Speech Audio Process., vol. 5, no. 3, pp. 257-265, May 1997. (Pubitemid 127745998)
57. N. Brümmer, "Focal bilinear: Tools for detector fusion and calibration, with use of side-information," [Online]. Available: https://sites.google. com/site/nikobrummer/focalbilinear.
58. N. Brümmer, "Measuring, refining and calibrating speaker and language information extracted from speech," Ph.D. dissertation, Stellenbosch Univ., Stellenbosch, South Africa, 2010.
59. N. Brümmer and D. Leeuwen, "On calibration of language recognition scores," in Proc. IEEE Odyssey: Speaker Lang. Recogn. Workshop, 2006, pp. 1-8.
60. P. S. Gopalakrishnan, D. Kanevsky, D. Nahamoo, and A. Nadas, "An inequality for rational functions with applications to some statistical estimation problems," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 107-113, Jan. 1991.
61. Y. Normandin, "An improvedMMIE training algorithmfor speaker-independent, small vocabulary, continuous speech recognition," in Proc. ICASSP, 1991, pp. 537-540.
62. D. Povey, "Discriminative training for large vocabulary speech recognition," Ph.D. dissertation, Univ. of Cambridge, Cambridge, U.K., 2003.
63. D. Kanevsky, "Extended Baum transformations for general functions, II," Human Language Technol., IBM, Tech. Rep. RC23645(W0506-120), 2005.
64. T. Jebara, "On reversing Jensen's inequality," in Proc. NIPS, 2002.
65. S. Axelrod, V. Goel, P. Gopinath, R. Olsen, and K. Visweswariah, "Discriminative estimation of subspace constrained Gaussian mixture models for speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 172-189, Jan. 2007.
66. D. Kanevsky, T. Sainath, B. Ramabhadran, and D. Nahamoo, "Generalization of extended Baum-Welch parameter estimation for discriminative training and decoding," in Proc. Interspeech, 2008.
67. D. Kanevsky, D. Nahamoo, T. N. Sainath, B. Ramabhadran, and P. A. Olsen, "A-Functions: A generalization of extended Baum-Welch transformations to convex optimization," in Proc. ICASSP, 2011, pp. 5164-5167.
68. R. Schlüter, W. Macherey, B. Müller, and H. Ney, "Comparison of discriminative training criteria and optimization methods for speech recognition," Speech Commun., pp. 287-310, 2001. (Pubitemid 32284868)
69. C. Liu, P. Liu, H. Jiang, F. Soong, and R. Wang, "Constrained Line Search Optimization for Discriminative Training in Speech Recognition," in Proc. ICASSP, 2007, pp. 329-332.
70. DR. Hsiao and T. Schultz, "Generalized Baum-Welch algorithm and its application to new extended Baum-Welch algorithm," in Proc. Interspeech, 2011.
71. M. Afify, "Extended Baum-Welch reestimation of Gaussian mixture models based on reverse Jensen inequality," in Proc. Interspeech, Lisbon, Portugal, Sep. 2005.
72. T. Jebara and A. Pentland, "On reversing Jensen's inequality," Adv. Neural Inf. Process. Syst., Dec. 2000.
73. C. Bishop, Pattern Recognition and Machine Learning. NewYork, NY, USA: Springer, 2006.
74. C. F. J. Wu, "On the convergence properties of the EM algorithm," Ann. Statist., vol. 11, no. 1, pp. 95-103, 1983.
75. L. Rabiner, "Tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
76. Y. Normandin, R. Cardin, and R. Demori, "High-performance connected digit recognition using maximum mutual information estimation," IEEE Trans. Speech Audio Process., vol. 2, no. 2, pp. 299-311, Apr. 1994.
77. D. Povey and P. C.Woodland, "Minimum phone error and i-smoothing for improved discriminative training," in Proc. IEEE Int. Conf. Acoustic, Speech, Signal Process., 2002, pp. 105-108.
78. E. McDermott, T. J. Hazen, J. Le Roux, A. Nakamura, and S. Katagiri, "Discriminative training for large vocabulary speech recognition usingminimumclassification error," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 203-223, Jan. 2007.
79. G. Heigold, H. Ney, P. Lehnen, T. Gass, and R. Schlüter, "Equivalence of generative and log-linearmodels," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 5, pp. 1138-1148, Jul. 2011.
80. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc. Ser. B., vol. 39, 1977.
81. J. Darroch and D. Ratcliff, "Generalized iterative scaling for log-linear models," Ann. Math. Statist., vol. 43, pp. 1470-1480, 1972.
82. S. A. Della Pietra, V. J. Della Pietra, and J. Lafferty, "Inducing features of random fields," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 4, pp. 380-393, Apr.. 1997. (Pubitemid 127762893)
83. G. Heigold, T. Deselaers, R. Schlüter, andH. Ney, "GIS-like estimation of log-linear models with hidden variables," in Proc. ICASSP, 2008, pp. 4045-4048.
84. M. Riedmiller and H. Braun, "A direct adaptivemethod for faster backpropagation learning: The Rprop algorithm," in IEEE International Conference on Neural Networks (ICNN), 1993.
85. L. Deng and X. Li, "Machine learning paradigms for speech recognition: An overview," IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 5, pp. 1060-1089, May 2013.
86. B. Kingsbury, T. Sainath, and H. Soltau, "Scalable minimum Bayes risk training of deep neural network acoustic models using distributed hessian-free optimization," in Proc. Interspeech, 2012.
87. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. W. Mao, M.-A. Ranzato, A.-W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng, "Large scale distributed deep networks," NIPS, 2012.
88. G. Hinton, L. Deng, D. Yu, G. Dahl, A.Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
89. G. Dahl,D.Yu, L.Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012.
90. A. Mohamed, G. Dahl, and G. Hinton, "Acoustic modeling using deep belief networks," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 14-22, Jan. 2012.
91. N. Morgan, "Deep and wide: Multiple layers in automatic speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 7-13, Jan. 2012.
92. M. Siniscalchi, L. Deng, D. Yu, and C.-H. Lee, "Exploiting deep neural networks for detection-based speech recognition," Neurocomputing, pp. 148-157, 2013.
93. L. Deng and D. Yu, "Deep convex network: A scalable architecture for speech pattern classification," in Proc. Interspeech, 2011.
94. L. Deng, D. Yu, and J. Platt, "Scalable stacking and learning for building deep architectures," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2012, pp. 2133-2136.
95. B. Hutchinson, L. Deng, and D. Yu, "Tensor deep stacking networks," IEEE Trans. Pattern Anal.Mach. Intell., vol. 35, no. 8, pp. 1944-1957, Aug. 2013.
96. I. Suskever, J. Martens, and G. E. Hinton, "Generating text with recurrent neural networks," in Proc. 28th Int. Conf. Mach. Learn., 2011.
97. L. Deng, G. E. Hinton, and B. Kingsbury, "New types of deep neural network learning for speech recognition and related applications: An overview," in Proc. ICASSP, 2013.