Investigation of maxout networks for speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2014, Pages 7649-7653

  Swietojanski, Pawel ^a   Li, Jinyu ^b   Huang, Jui Ting ^b  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b MICROSOFT   (United States)

Author keywords

deep neural networks; low resource speech recognition; maxout networks; multitask learning

Indexed keywords

KNOWLEDGE MANAGEMENT; SIGNAL PROCESSING;

ACOUSTIC MODELLING; DEEP NEURAL NETWORKS; DIFFERENT LAYERS; KNOWLEDGE TRANSFER; LARGE VOCABULARY SPEECH RECOGNITION; LOW-RESOURCE SPEECH RECOGNITION; MULTITASK LEARNING; WORD ERROR RATE;

SPEECH RECOGNITION;

EID: 84905270524     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2014.6855088     Document Type: Conference Paper

References (33)

1. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach, Kluwer Academic Publishers, 1994.
2. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco, "Connectionist probability estimators in HMM speech recognition, " IEEE Transactions on Speech and Audio Processing, vol. 2, no. 1, pp. 161-174, 1994.
3. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A-R Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, 2012.
4. A. Mohamed, G. E. Dahl, and G. Hinton, "Acoustic modeling using deep belief networks, " IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 1, pp. 14-22, 2012.
5. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in In Proc. IEEE ASRU, 2011, pp. 24-29.
6. B. Kingsbury, T. N. Sainath, and H. Soltau, "Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization, " in Proc. Interspeech, 2012.
7. N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition, " in Proc. Interspeech, 2012.
8. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, " IEEE Transactions on Audio, Speech and Language Processing, vol. 20, no. 1, pp. 30-42, 2012.
9. M. Seltzer, D. Yu, and Y. Wang, "An investigation of deep neural networks for noise robust speech recognition, " in In Proc. ICASSP, 2013.
10. J-T Huang, J. Li, D. Yu, L. Deng, and Y. Gong, "Crosslanguage knowledge transfer using multilingual deep neural network with shared hidden layers, " in Proc. ICASSP, 2013.
11. P. Swietojanski, A. Ghoshal, and S. Renals, "Unsupervised cross-lingual knowledge transfer in DNN-based LVCSR, " in In Proc. IEEE SLT, Miami, 2012.
12. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, "Multilingual acoustic models using distributed deep neural networks, " in In Proc. ICASSP, 2013.
13. A. Ghoshal, P. Swietojanski, and S. Renals, "Multilingual training of deep neural networks, " in In Proc. ICASSP, 2013.
14. Y. Miao and F. Metze, "Improving low-resource CD-DNNHMM using dropout and multilingual DNN training, " in Interspeech, 2013.
15. P. Swietojanski, A. Ghoshal, and S. Renals, "Hybrid acoustic models for distant and multichannel large vocabulary speech recognition, " in Proc. IEEE ASRU, Dec. 2013.
16. V. Nair and G. Hinton, "Rectified linear units improve restricted Boltzmann machines, " in In Proc. ICML, 2010, pp. 131-136.
17. X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse rectifier neural networks, " Journal of Machine Learning Research, vol. 15, pp. 315323, 2011.
18. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult, " Neural Networks, IEEE Transactions on, vol. 5, no. 2, pp. 157-166, 1994.
19. L. Toth, "Phone recognition with deep sparse rectifier neural networks, " in In Proc. ICASSP, 2013.
20. G. E. Dahl, T. N. Sainath, and G. E. Hinton, "Improving deep neural networks for LVCSR using rectified linear units and dropout, " in Proc. ICASSP, 2013.
21. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors, " CoRR, vol. abs/1207. 0580, 2012.
22. M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean, and G. E. Hinton, "On rectified linear units for speech processing, " in In Proc. ICASSP, 2013.
23. A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models, " in Proc. ICML, 2013.
24. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, "Maxout networks, " arXiv:1302. 4389, 2013.
25. M. Cai, Y. Shi, and J. Liu, "Deep maxout neural networks for speech recognition, " in Proc. ASRU, Dec 2013, pp. 291-296.
26. Y. Miao, F. Metze, and S. Rawat, "Deep maxout networks for low-resource speech recognition, " in Proc. ASRU, 2013.
27. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradientbased learning applied to document recognition, " Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
28. Y. Bengio, "Deep learning of representations: Looking forward, " CoRR, vol. abs/1305. 0445, 2013.
29. G. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets, " Neural Computation, vol. 18, pp. 1527-1554, 2006.
30. T. Mikolov, Statistical Language Models based on Neural Networks, Ph. D. Thesis, Brno University of Technology, 2012.
31. J. Li, D. Yu, J. Huang, and Y. Gong, "Improving wideband speech recognition using mixed-bandwidth training data in CD-DNN-HMM, " in In Proc. IEEE SLT, 2012, pp. 131-136.
32. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " in In Proc. ICASSP, 2009, pp. 3761-3764.
33. J. Snoek, H. Larochelle, and R. P. Adams, "Practical Bayesian optimization of machine learning algorithms, " in Neural Information Processing Systems, 2012.