Ensemble deep learning for speech recognition

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1915-1919

  Deng, Li ^a   Platt, John C ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

Deep learning; Ensemble learning; Log linear system combination; Speech recognition; Stacking

Indexed keywords

CONVEX OPTIMIZATION; LEARNING SYSTEMS; LINEAR SYSTEMS; OPTIMIZATION; RECURRENT NEURAL NETWORKS; SPEECH; SPEECH COMMUNICATION;

CONVEX OPTIMIZATION PROBLEMS; DEEP LEARNING; ENSEMBLE LEARNING; HIERARCHICAL FEATURES; POSTERIOR PROBABILITY; RECOGNITION ACCURACY; STACKING; SYSTEM COMBINATION;

SPEECH RECOGNITION;

EID: 84910048046     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (40)

1. Baker, J., Deng, L., Glass, J., Khudanpur, S., Lee, C.-H., Morgan, N., and O'Shaughnessy, D. "Research developments and directions in speech recognition and understanding, " IEEE Sig. Proc. Mag., vol. 26, no. 3, May 2009, pp. 75-80.
2. Bengio, Y., Courville, A., and Vincent, P. "Representation learning: A review and new perspectives, " IEEE Trans. PAMI, vol. 38, pp. 1798-1828, 2013.
3. Bengio, Y., Boulanger, N., and Pascanu, R. "Advances in optimizing recurrent networks, " Proc. ICASSP, 2013.
4. Breiman, L. "Stacked regression, " Machine Learning, Vol. 24, pp. 49-64, 1996.
5. Bourlard, H. and Morgan, N., Connectionist Speech Recognition: A Hybrid Approach, Kluwer, 1993.
6. Chen, J. and Deng, L. "A primal-dual method for training recurrent neural networks constrained by the echo-state property, " Proc. Int. Conf. Learning Representations, April, 2014.
7. Dahl, G., Yu, D., Deng, L., and Acero, A. "Contextdependent, pre-trained deep neural networks for large vocabulary speech recognition, " IEEE Trans. Audio, Speech, & Language Proc., Vol. 20, pp. 30-42, 2012.
8. Deng, L. and Chen, J. "Sequence classification using the high-level features extracted from deep neural networks, " Proc. ICASSP, 2014.
9. Deng, L., Abdel-Hamid, O., and Yu, D. "A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion, " Proc. ICASSP, 2013.
10. Deng, L., Li, J., Huang, K., Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X. He, J. Williams, Y. Gong, and A. Acero. "Recent advances in deep learning for speech research at Microsoft, " Proc. ICASSP, 2013.
11. Deng, L., Hinton, G., and Kingsbury, B. "New types of deep neural network learning for speech recognition and related applications: An overview, " Proc. ICASSP, 2013.
12. Deng, L., Yu, D., and Hinton, G. "Deep Learning for Speech Recognition and Related Applications" NIPS Workshop, 2009.
13. Deng, L., Yu, D., and Platt, J. "Scalable stacking and learning for building deep architectures, " Proc. ICASSP, 2012.
14. Graves, A., Mohamed, A., and Hinton, G. "Speech recognition with deep recurrent neural networks, " Proc. ICASSP, 2013.
15. Graves, A., Jaitly, N., and Mohamed, A. "Hybrid speech recognition with deep bidirectional LSTM, " Proc. ASRU, 2013.
16. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B., "Deep Neural Networks for Acoustic Modeling in Speech Recognition, " IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
17. Hinton, G. and Salakhutdinov, R. "Reducing the dimensionality of data with neural networks, " Science, vol. 313. no. 5786, pp. 504 - 507, July 2006.
18. Jaitly, N., Nguyen, P., and Vanhoucke, V. "Application of pre-trained deep neural networks to large vocabulary speech recognition, " Proc. Interspeech, 2012.
19. Kingsbury, B., Sainath, T., and Soltau, H. "Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization, " Proc. Interspeech, 2012.
20. Maas, A., Le, Q., O'Neil, T., Vinyals, O., Nguyen, P., and Ng, P. "Recurrent neural networks for noise reduction in robust ASR, " Proc. Interspeech, 2012.
21. Martens, J. and Sutskever, I. "Learning recurrent neural networks with Hessian-free optimization, " Proc. ICML, 2011.
22. Mikolov, T., Deoras, A., Povey, D., Burget, L., and Cernocky, J. "Strategies for training large scale neural network language models, " Proc. ASRU, 2011.
23. Mikolov, T., Karafiat, M., Burget, L., Cernocky, J., and Khudanpur, S. "Recurrent neural network based language model, " Proc. ICASSP, 2010, 1045-1048.
24. Mohamed, A., Dahl, G. and Hinton, G. "Acoustic modeling using deep belief networks", IEEE Trans. Audio, Speech, and Language Proc. Vol. 20., January 2012.
25. Mohamed, A., Yu, D., and Deng, L. "Investigation of fullsequence training of deep belief networks for speech recognition, " Proc. Interspeech, 2010.
26. Morgan, N. "Deep and wide: Multiple layers in automatic speech recognition, " IEEE Trans. Audio, Speech, and Language Processing, Vol. 20 (1), January 2012.
27. Pascanu, R., Mikolov, T., and Bengio, Y. "On the difficulty of training recurrent neural networks, " Proc. ICML, 2013.
28. Robinson, A. "An application of recurrent nets to phone probability estimation, " IEEE Trans. Neural Networks, Vol. 5, pp. 298-305, 1994.
29. Sainath, T., Kingsbury, B., Soltau, H., and Ramabhadran, B. "Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech Tasks, " IEEE Trans. Audio, Speech, and Language Processing, vol.21, no.11, pp.2267-2276, Nov. 2013.
30. Sainath, T., Mohamed, A., Kingsbury, B., and Ramabhadran, B. "Convolutional neural networks for LVCSR, " Proc. ICASSP, 2013.
31. Sainath, T., Kingsbury, Mohamed, A., Dahl, G., Saon, G., Soltau, H., Beran, T., Aravkin, A., and B. Ramabhadran. "Improvements to deep convolutional neural networks for LVCSR, " Proc. ASRU, 2013.
32. Sainath, T., Kingsbury, B., Ramabhadran, B., Novak, P., and Mohamed, A. "Making deep belief networks effective for large vocabulary continuous speech recognition, " Proc. ASRU, 2011.
33. Seide, F., Li, G., and Yu, D. "Conversational speech transcription using context-dependent deep neural networks, " Proc. Interspeech, 2011.
34. Sutskever, I., Martens J., and Hinton, G. "Generating text with recurrent neural networks, " Proc. ICML, 2011.
35. Triefenbach, F., Jalalvand, A., Demuynck, K., Martens, J.- P. "Acoustic modeling with hierarchical reservoirs, " IEEE Trans. Audio, Speech, and Language Processing, vol.21, no.11, pp. 2439-2450, Nov. 2013.
36. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., and Lang, K. "Phoneme recognition using time-delay neural networks, " IEEE Trans. Acoust. Speech, and Signal Proc., vol. 37, pp. 328-339, 1989.
37. Wolpert, D. "Stacked generalization, " Neural Networks, vol. 5, no. 2, pp. 241-259, 1992.
38. Yao, K., Zweig, G., Hwang, M., Shi, Y., and Yu, D. "Recurrent Neural Networks for Language Understanding, " Proc. Interspeech, 2013.
39. Yu, D., Deng, L., and Seide, F. "The deep tensor neural network with applications to large vocabulary speech recognition, " IEEE Trans. Audio, Speech, and Language Processing, vol. 21, no. 2, pp. 388-396, 2013.
40. Yu, D., Deng, L., and Dahl, G.E., "Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition, " NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010.