Speech-centric information processing: An optimization-oriented approach

Proceedings of the IEEE

Volumn 101, Issue 5, 2013, Pages 1116-1135

  He, Xiaodong ^a   Deng, Li ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

Joint optimization; speech recognition; speech centric information processing (SCIP); spoken language translation (SLT); spoken language understanding (SLU); voice search

Indexed keywords

INFORMATION RETRIEVAL; SEARCH ENGINES; SPEECH; SYSTEMS ANALYSIS; TRANSLATION (LANGUAGES);

AUTOMATIC SPEECH RECOGNITION; HUMAN LANGUAGE TECHNOLOGIES; INFORMATION PROCESSING SYSTEMS; JOINT OPTIMIZATION; SPOKEN DOCUMENT RETRIEVAL; SPOKEN LANGUAGE TRANSLATION; SPOKEN LANGUAGE UNDERSTANDING; VOICE SEARCHES;

SPEECH RECOGNITION;

EID: 84876669905     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2012.2236631     Document Type: Article

References (106)

1. M. Bacchiani, F. Beaufays, J. Schalkwyk, M. Schuster, and B. Strope, "Deploying GOOG-411: Early lessons in data, measurement, and testing," in Proc. Int. Conf. Acoust. Speech Signal Process., 2008, pp. 5260-5263.
2. A. Bagnell and D. Bradley, "Differentiable sparse coding," Proc. Neural Inf. Process. Syst., pp. 113-120, 2008.
3. J. Baker, L. Deng, J. Glass, S. Khudanpur, C.-H. Lee, N. Morgan, and D. O'Shaughnessy, "Research developments and directions in speech recognition and understandingVPart 1," IEEE Signal Process. Mag., vol. 26, no. 3, pp. 75-80, May 2009.
4. J. Baker, L. Deng, S. Khudanpur, C. Lee, J. Glass, N. Morgan, and D. O'Shaughnessy, "Updated MINDS report on speech recognition and understandingVPart 2," IEEE Signal Process. Mag., vol. 26, no. 4, pp. 78-85, Jul. 2009.
5. R. Battiti, "First-and second-order methods for learning: Between steepest descent and Newton's method," Neural Comput., vol. 4, pp. 141-166, 1992.
6. L. Baum and G. Sell, "Growth transformations for functions on manifolds," Pacific J. Math., vol. 27, no. 2, pp. 211-227, 1968.
7. L. Baum and J. Eagon, "An inequality with applications to statistical prediction for functions of Markov processes and to a model of ecology," Bull. Amer. Math. Soc., vol. 73, pp. 360-363, 1967.
8. N. Bertoldi, R. Zens, M. Federico, and W. Shen, "Efficient speech translation through confusion network decoding," IEEE Trans. Audio Speech Lang. Process., vol. 16, no. 8, pp. 1696-1705, Nov. 2008.
9. C. Bishop, Pattern Recognition and Machine Learning. New York: Springer-Verlag, 2006.
10. D. Bohus, S. G. Puerto, D. Huggins-Daines, V. Keri, G. Krishna, R. Kumar, A. Raux, and S. Tomkoohus, "ConQuest: An open-source dialog system for conferences," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguist., Human Lang. Technol., 2007, pp. 9-12.
11. D. Bolanos, G. Zweig, and P. Nguyen, "Multi-scale personalization for voice search applications," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguist., Human Lang. Technol., 2009, pp. 101-104.
12. R. Brent, Algorithms for Minimization Without Derivatives. Englewood Cliffs, NJ: Prentice-Hall, 1973.
13. P. Brown, S. Pietra, V. Pietra, and R. Mercer, "The mathematics of statistical machine translation: Parameter estimation," Comput. Linguist., vol. 19, no. 2, pp. 263-311, 1993.
14. F. Casacuberta, M. Federico, H. Ney, and E. Vidal, "Recent efforts in spoken language translation," IEEE Signal Process. Mag., vol. 25, no. 3, pp. 80-88, May 2008.
15. C. Chelba, T. Hazen, and M. Saraclar, "Retrieval and browsing of spoken content," IEEE Signal Process. Mag., vol. 25, no. 3, pp. 39-49, May 2008.
16. S. Axelrod, V. Goel, R. Gopinath, P. Olsen, and K. Visweswariah, "Discriminative estimation of subspace constrained Gaussian mixture models for speech recognition," IEEE Trans. Audio Speech Lang. Process., vol. 15, no. 1, pp. 172-189, Jan. 2007.
17. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large vocabulary speech recognition," IEEE Trans. Audio Speech Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012.
18. R. De Mori, F. Bechet, D. Hakkani-Tur, M. McTear, G. Riccardi, and G. Tur, "Spoken language understanding," IEEE Signal Process. Mag., vol. 25, no. 3, pp. 50-58, May 2008.
19. L. Deng and D. O'Shaughnessy, Speech ProcessingVA Dynamic and Optimization-Oriented Approach. New York: Marcel Dekker, 2003.
20. L. Deng and X. Huang, "Challenges in adopting speech recognition," Commun. ACM, vol. 47, no. 1, pp. 11-13, Jan. 2004.
21. L. Deng, K. Wang, A. Acero, H. Hon, J. Droppo, C. Boulis, Y. Wang, D. Jacoby, M. Mahajan, C. Chelba, and X. D. Huang, "Distributed speech processing in MiPad's multimodal user interface," IEEE Trans. Audio Speech Process., vol. 10, no. 8, pp. 605-619, Nov. 2002.
22. L. Deng, "Front-end, back-end, and hybrid techniques to noise-robust speech recognition," in Robust Speech Recognition of Uncertain Data, D. Kolossa and R. Haeb-Umbach, Eds. New York: Springer-Verlag, 2011, pp. 67-99.
23. J. E. Dennis and R. B. Schnabel, "Numerical methods for unconstrained optimization and nonlinear equations SIAM's Classics in Applied Mathematics. Philadelphia, PA: SIAM, 1996.
24. J. Droppo and A. Acero, "Joint discriminative front end and back end training for improved speech recognition accuracy," in Proc. Int. Conf. Acoust. Speech Signal Process., 2006, DOI: 10.1109/ICASSP.2006.1660012.
25. Y. Est̀eve, C. Raymond, F. Bechet, and R. DeMori, "Conceptual decoding for spoken dialog systems," in Proc. Eurospeech Conf., Geneva, Switzerland, Sep. 1-4, 2003.
26. S. E. Fahlman, "An empirical study of learning speed in back-propagation networks," Canergie Mellon Univ., Pittsburgh, PA, Tech. Rep., 1988.
27. J. Feng, S. Bangalore, and M. Gilbert, "Role of natural language understanding in voice local search," in Proc. Interspeech 2009, Brighton, U.K., Sep. 6-10, 2009.
28. J. Feng, "Query parsing in mobile voice search," in Proc. World Wide Web 2010, Raleigh, NC, USA, Apr. 26-30, 2010.
29. J. Feng, M. Johnston, and S. Bangalore, "Speech and multimodal interaction in mobile search," IEEE Signal Process. Mag., vol. 28, no. 4, pp. 40-49, Jul. 2011.
30. A. Franz and B. Milch, "Searching the Web by voice," in Proc. Conf. Comput. Linguist., 2002, pp. 1213-1217.
31. V. Goel, S. Kumar, and W. Byrne, "Segmental minimum Bayes-risk decoding for automatic speech recognition," IEEE Trans. Speech Audio Process., vol. 12, no. 3, pp. 234-249, May 2004.
32. P. Gopalakrishnan, D. Kanevsky, A. Nadas, and D. Nahamoo, "An inequality for rational functions with applications to some statistical estimation problems," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 107-113, Jan. 1991.
33. A. Gunawardana and W. Byrne, "Discriminative speaker adaptation with conditional maximum likelihood linear regression," in Proc. Eurospeech 2001, Aalborg, Denmark, Sep. 3-7, 2001.
34. S. Hahn, M. Dinarelli, C. Raymond, F. Lef̀evre, P. Lehnen, R. De Mori, H. Ney, and G. Riccardi, "Comparing stochastic approaches to spoken language understanding in multiple languages," IEEE Trans. Audio Speech Lang. Process., vol. 19, no. 6, pp. 1569-1583, Aug. 2011.
35. D. Harman, Meeting of the MINDS: Future Directions for Human Language Technology. [Online]. Available: http://www-nlpir.nist. gov/MINDS/FINAL/exec. summary.pdf.
36. K. Hashimoto, J. Yamagishi, W. Byrne, S. King, and K. Tokuda, "Impacts of machine translation and speech synthesis on speech-to-speech translation," Speech Commun., vol. 54, no. 7, pp. 857-866, Sep. 2012.
37. X. He and L. Deng, "Speech recognition, machine translation, and speech translationVA unified discriminative learning paradigm," IEEE Signal Process. Mag., vol. 28, no. 5, pp. 126-133, Sep. 2011.
38. X. He and L. Deng, "Maximum expected BLEU training of phrase and lexicon translation models," in Proc. Annu. Meeting Assoc. Comput. Linguist., Jul. 2012, vol. 1, pp. 292-301.
39. X. He, L. Deng, and A. Acero, "Why word error rate is not a good metric for speech recognizer training for the speech translation task?" in Proc. Int. Conf. Acoust. Speech Signal Process., 2011, pp. 5632-5635.
40. X. He, A. Axelrod, L. Deng, A. Acero, M. Hwang, A. Nguyen, A. Wang, and X. Huang, "The MSR system for IWSLT 2011 evaluation," in Proc. IWSLT, San Francisco, CA, USA, Dec. 8-9, 2011.
41. X. He, L. Deng, and W. Chou, "Discriminative learning in sequential pattern recognition," IEEE Signal Process. Mag., vol. 25, no. 5, pp. 14-36, Sep. 2008.
42. G. Heigold, P. Dreuw, S. Hahn, R. Schlüter, and H. Ney, "Margin-based discriminative training for string recognition," IEEE J. Sel. Top. Signal Process., vol. 4, no. 6, pp. 917-925, Dec. 2010.
43. G. Heigold, H. Ney, P. Lehnen, T. Gass, and R. Schluter, "Equivalence of generative and log-linear models," IEEE Trans. Audio Speech Lang. Process., vol. 19, no. 5, pp. 1138-1148, Jul. 2011.
44. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
45. R. Hsiao and T. Schultz, "Generalized Baum-Welch algorithm and its implication to a new extended Baum-Welch algorithm," in Proc. Interspeech 2011, Florence, Italy, Aug. 27-31, 2011.
46. X. Huang and L. Deng, "An overview of modern speech recognition," in Handbook of Natural Language Processing, 2nd ed. London, U.K.: Chapman & Hall/CRC Press, 2010, pp. 339-366.
47. K. Jarvelin and J. Kekalainen, "IR evaluation methods for retrieving highly relevant documents," in Proc. Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2000, pp. 41-48.
48. Y. Ju and T. Paek, "A voice search approach to replying to SMS messages in automobiles," in Proc. Interspeech 2009, Brighton, U.K., Sep. 6-10, 2009.
49. D. Jurafsky and J. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 2008.
50. D. Kanevsky, D. Nahamoo, T. Sainath, B. Ramabhadran, and P. Olsen, "A-functions: A generalization of extended Baum-Welch transformations to convex optimization," in Proc. Int. Conf. Acoust. Speech Signal Process., 2011, pp. 5164-5167.
51. D. Kanevsky, D. Nahamoo, T. Sainath, and B. Ramabhadran, "Convergence of line search A-function methods," in Proc. Interspeech 2011, Florence, Italy, Aug. 27-31, 2011.
52. P. Koehn, F. Och, and D. Marcu, "Statistical phrase-based translation," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguist., Human Lang. Technol., 2003, vol. 1, pp. 48-54.
53. D. Kolossa and R. Haeb-Umbach, Eds., Robust Speech Recognition of Uncertain Data. New York: Springer-Verlag, 2011.
54. S. Krauwer, D. Arnold, W. Kasper, M. Rayner, and H. Somers, "Spoken language translation," in Proc. ACL Spoken Lang. Transl. Workshop, Madrid, Spain, 1997, pp. 1-5.
55. J. Le Roux and E. McDermott, "Optimization for discriminative training," in Proc. Interspeech 2005, Lisbon, Portugal, Sep. 4-8, 2005.
56. M. Lehr and I. Shafran, "Discriminatively estimated joint acoustic, duration and language model for speech recognition," in Proc. Int. Conf. Acoust. Speech Signal Process., 2010, pp. 5542-5545.
57. F. Lef̀evre, F. Mairesse, and S. Young, "Cross-lingual spoken language understanding from unaligned data using discriminative classification models and machine translation," in Proc. Interspeech 2010, Makuhari, Japan, Sep. 26-30, 2010.
58. L.-S. Lee and B. Chen, "Spoken document understanding and organization," IEEE Signal Process. Mag., vol. 22, no. 5, pp. 42-60, Sep. 2005.
59. M. Levit, S. Chang, B. Buntschuh, and N. Kibre, "End-to-end speech recognition accuracy metric for voice search tasks," in Proc. Int. Conf. Acoust. Speech Signal Process., 2012, pp. 5141-5144.
60. X. Li, Y. Wang, and G. Tur, "Multi-task learning for spoken language understanding with shared slots," in Proc. Interspeech 2011, Florence, Italy, Aug. 27-31, 2011.
61. S. Mann, A. Berton, and U. Ehrlich, "How to access audio files of large databases using in-car speech dialogue systems," in Proc. Interspeech Conf., Antwerp, Belgium, 2007, pp. 138-141.
62. E. Matusov, S. Kanthak, and H. Ney, "Integrating speech recognition and machine translation: Where do we stand?" in Proc. Int. Conf. Acoust. Speech Signal Process., 2006, DOI: 10.1109/ICASSP.2006.1661501.
63. E. McDermott and T. Hazen, "Minimum classification error training of landmark models for real-time continuous speech recognition," in Proc. Int. Conf. Acoust. Speech Signal Process., 2006, vol. 1, pp. 937-940.
64. E. McDermott, T. Hazen, J. Le Roux, A. Nakamura, and S. Katagiri, "Discriminative training for large vocabulary speech recognition using minimum classification error," IEEE Trans. Speech Audio Process., vol. 15, no. 1, pp. 203-223, Jan. 2007.
65. S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J. Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto, "The ATR multilingual speech-to-speech translation system," IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 2, pp. 365-376, Mar. 2006.
66. H. Ney, "Speech translation: Coupling of recognition and translation," in Proc. Int. Conf. Acoust. Speech Signal Process., 1999, vol. 1, pp. 517-520.
67. F. Och, "Minimum error rate training in statistical machine translation," in Proc. Annu. Meeting Assoc. Comput. Linguist., 2003, pp. 160-167.
68. J. Olive, C. Christianson, and J. McCary, Eds., Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. New York: Springer-Verlag, 2011.
69. M. Ostendorf, B. Favre, R. Grishman, D. Hakkani-Tur, M. Harper, D. Hillard, J. Hirschberg, J. Heng, J. Kahn, L. Yang, S. Maskey, E. Matusov, H. Ney, A. Rosenberg, E. Shriberg, W. Wen, and C. Woofers, "Speech segmentation and spoken document processing," IEEE Signal Process. Mag., vol. 25, no. 3, pp. 59-69, May 2008.
70. K. Papineni, S. Roukos, T. Ward, and W. Zhu, "BLEU: A method for automatic evaluation of machine translation," in Proc. Annu. Meeting Assoc. Comput. Linguist., 2002, pp. 311-318.
71. M. Paul, M. Federico, and S. Stücker, "Overview of the IWSLT 2010 evaluation campaign," in Proc. IWSLT, Paris, France, Dec. 2-3, 2010.
72. D. Povey, "Discriminative training for large vocabulary speech recognition," Ph.D. dissertation, Cambridge Univ. Eng. Dept., Cambridge Univ., Cambridge, U.K., 2004.
73. A. Reddy and R. Rose, "Integration of statistical models for dictation of document translations in a machine-aided human translation task," IEEE Trans. Audio Speech Lang. Process., vol. 18, no. 8, pp. 2015-2027, Nov. 2010.
74. G. Riccardi and A. L. Gorin, "Stochastic language models for speech recognition and understanding," in Proc. ICSLP, Sydney, Australia, Nov. 30-Dec. 4, 1998.
75. M. Riedmiller and H. Braun, "A direct adaptive method for faster back propagation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Netw., San Francisco, CA, 1993, pp. 586-591.
76. Y. Normandin, "Hidden Markov models, maximum mutual information estimation, and the speech recognition problem," Ph.D. dissertation, Electr. Comput. Eng. Dept., McGill Univ., Montreal, QC, Canada, 1991.
77. M. Seltzer, Y. Ju, I. Tashev, Y. Wang, and D. Yu, "In-car media search," IEEE Signal Process. Mag., vol. 28, no. 4, pp. 50-60, Jul. 2011.
78. P. Simard, Y. Steinkraus, and J. Platt, "Best practice for convolutional neural networks applied to visual document analysis," in Proc. Int. Conf. Document Anal. Recognit., 2003, pp. 958-962.
79. Y. Song, Y.-Y. Wang, Y. Ju, M. Seltzer, I. Tashev, and A. Acero, "Voice search of structured media data," in Proc. Int. Conf. Acoust. Speech Signal Process., 2009, pp. 3941-3944.
80. G. Tur and L. Deng, "Intent determination and spoken utterance classification," in Spoken Language Understanding: Systems for Extracting Semantic Information from Speech, Tur and De Mori, Eds. New York: Wiley, 2011, pp. 81-104.
81. G. Tur, L. Deng, D. Hakkani-Tür, and X. He, "Towards deeper understanding: Deep convex networks for semantic utterance classification," in Proc. Int. Conf. Acoust. Speech Signal Process., Kyoto, Japan, Mar. 2012, pp. 5045-5048.
82. E. Vidal, "Finite-state speech-to-speech translation," in Proc. Int. Conf. Acoust. Speech Signal Process., Munich, Germany, 1997, pp. 111-114.
83. A. Waibel and C. Fugen, "Spoken language translation," IEEE Signal Process. Mag., vol. 25, no. 3, pp. 70-79, May 2008.
84. Y. Wang, D. Yu, Y. Ju, and A. Acero, "An introduction to voice search," IEEE Signal Process. Mag., vol. 25, no. 3, pp. 28-38, May 2008.
85. Y. Wang, L. Deng, and A. Acero, "Semantic frame based spoken language understanding," in Spoken Language Understanding: Systems for Extracting Semantic Information from Speech, Tur and De Mori, Eds. New York: Wiley, 2011, pp. 35-80.
86. Y. Wang, A. Acero, and C. Chelba, "Is word error rate a good indicator for spoken language understanding accuracy," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., 2003, pp. 577-582.
87. Y. Wang, L. Deng, and A. Acero, "Spoken language understanding," IEEE Signal Process. Mag., vol. 22, no. 5, pp. 16-31, Sep. 2005.
88. P. Xu, P. Fung, and R. Chan, "Phrase-level transduction model with reordering for spoken to written language transformation," in Proc. Int. Conf. Acoust. Speech Signal Process., 2012, pp. 4965-4968.
89. S. Yaman, L. Deng, D. Yu, Y. Wang, and A. Acero, "An integrative and discriminative technique for spoken utterance classification," IEEE Trans. Audio Speech Lang. Process., vol. 16, no. 6, pp. 1207-1214, Aug. 2008.
90. S. Young, "Cognitive user interfaces," IEEE Signal Process. Mag., vol. 27, no. 3, pp. 128-140, May 2010.
91. S. Young, M. Gasic, B. Thomson, and J. Williams, "POMDP-based statistical spoken dialog systems: A review," Proc. IEEE, 2013, DOI: 10.1109/JPROC.2012.2225812.
92. D. Yu and L. Deng, "Deep learning and its applications to signal and information processing," IEEE Signal Process. Mag., vol. 28, no. 1, pp. 145-154, Jan. 2011.
93. D. Yu, Y.-C. Ju, Y.-Y. Wang, G. Zweig, and A. Acero, "Automated directory assistance system: From theory to practice," in Proc. Interspeech Conf., Antwerp, Belgium, 2007, pp. 2709-2712.
94. Y. Zhang, L. Deng, X. He, and A. Acero, "A novel decision function and the associated decision-feedback learning for speech translation," in Proc. Int. Conf. Acoust. Speech Signal Process., 2011, pp. 5608-5611.
95. B. Zhou, "Statistical translation for speech: A perspective on structures and learning," Proc. IEEE, 2013.
96. B. Zhou, L. Besacier, and Y. Gao, "On efficient coupling of ASR and SMT for speech translation," in Proc. Int. Conf. Acoust. Speech Signal Process., 2007, vol. IV, pp. 101-104.
97. G. Zweig, P. Nguyen, Y.-C. Ju, Y.-Y. Wang, D. Yu, and A. Acero, "The voice rate dialog system for consumer ratings," in Proc. Interspeech Conf., 2007, pp. 2713-2716.
98. Tur and De Mori, Eds., Spoken Language Understanding: Systems for Extracting Semantic Information From Speech. New York: Wiley, 2011.
99. D. Chiang, K. Knight, and W. Wang, "11, 001 new features for statistical machine translation," in Proc. of Conf. North Amer. Chapter Assoc. Comput. Linguist., Human Lang. Technol., 2009, pp. 218-226.
100. P. Liang, A. Bouchard-Cote, D. Klein, and B. Taskar, "An end-to-end discriminative approach to machine translation," in Proc. Conf. Comput. Linguist./Assoc. Comput. Linguist., 2006, pp. 761-768.
101. B. Jabaian, L. Besacier, and F. Lefevre, "Comparison and combination of lightly supervised approaches for language portability of a spoken language understanding system," IEEE Trans. Audio Speech Lang. Process., vol. 21, no. 3, 2013.
102. L. Deng, A. Acero, M. Plumpe, and X. D. Huang, "Large-vocabulary speech recognition under adverse acoustic environments," Proc. Int. Conf. Spoken Lang., pp. 806-809.
103. L. Deng, G. Tur, X. He, and D. Hakkani-Tur, "Use of kernel deep convex networks and end-to-end learning for spoken language understanding," Proc. IEEE Workshop Spoken Lang. Technol., Dec. 2012.
104. B. Jabaian, L. Besacier, and F. Lefevre, "Investigating multiple approaches for SLU portability to a new language," in Proc. Interspeech, 2010.
105. B. Jabaian, L. Besacier, and F. Lefevre, "Combination of stochastic understanding and machine translation systems for language portability of dialogue systems," in Proc. ICASSP, 2011.
106. N. Camelin, C. Raymond, F. Bechet, and R. De Mori, "On the use of machine translation for spoken language understanding portability," in Proc. ICASSP, 2010.