Improving deep neural network acoustic models using generalized maxout networks

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn , Issue , 2014, Pages 215-219

  Zhang, Xiaohui ^a   Trmal, Jan ^a   Povey, Daniel ^a   Khudanpur, Sanjeev ^a  

^a Johns Hopkins University   (United States)

Author keywords

Acoustic Modeling; Deep Learning; Maxout Networks; Speech Recognition

Indexed keywords

CONTINUOUS SPEECH RECOGNITION; CONTROL NONLINEARITIES; SPEECH RECOGNITION;

ACOUSTIC MODEL; DEEP LEARNING; DEEP NEURAL NETWORKS; LARGE VOCABULARY CONTINUOUS SPEECH RECOGNITION; LINEAR UNITS;

SIGNAL PROCESSING;

EID: 84905239342     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2014.6853589     Document Type: Conference Paper

References (29)

1. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82-97, 2012.
2. George E Dahl, Dong Yu, Li Deng, and Alex Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 1, pp. 30-42, 2012.
3. Frank Seide, Gang Li, and Dong Yu, "Conversational speech transcription using context-dependent deep neural networks.," in INTERSPEECH, 2011, pp. 437-440.
4. MD Zeiler, M Ranzato, R Monga, M Mao, K Yang, QV Le, P Nguyen, A Senior, V Vanhoucke, J Dean, et al., "On rectified linear units for speech processing," in Proc. ICASSP, 2013.
5. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng, "Rectifier nonlinearities improve neural network acoustic models," in ICML Workshop on Deep Learning for Audio, Speech, and Language Processing (WDLASL 2013), 2013.
6. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio, "Maxout networks," arXiv preprint arXiv:1302.4389, 2013.
7. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors," arXiv preprint arXiv:1207.0580, 2012.
8. Y. Miao, S. Rawat, and F. Metze, "Deep maxout networks for low resource speech recognition," in Proc. ASRU, 2013.
9. M. Cai, Y. Shi, and J. Liu, "Deep maxout neural networks for speech recognition," in Proc. ASRU, 2013.
10. Koray Kavukcuoglu, M Ranzato, Rob Fergus, and Yann Le-Cun, "Learning invariant features through topographic filter maps," in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009, pp. 1605-1612.
11. Y-Lan Boureau, Jean Ponce, and Yann LeCun, "A theoretical analysis of feature pooling in visual recognition," in Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010, pp. 111-118.
12. Pierre Sermanet, Soumith Chintala, and Yann LeCun, "Convolutional neural networks applied to house numbers digit classification," in Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 2012, pp. 3288-3291.
13. Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, and Yoshua Bengio, "Learned-norm pooling for deep neural networks," arXiv preprint arXiv:1311.1780, 2013.
14. D. Povey, A. Ghoshal, et al., "The Kaldi Speech Recognition Toolkit," in Proc. ASRU, 2011.
15. Karel Veselỳ, Arnab Ghoshal, Lukás Burget, and Daniel Povey, "Sequence-discriminative training of deep neural networks," in Interspeech, 2013.
16. D. Povey and D. Kanevsky and B. Kingsbury and B. Ramabhadran and G. Saon and K. Visweswariah, "Boosted MMI for Feature and Model Space Discriminative Training," in ICASSP, 2008.
17. Gibson M. and Hain T., "Hypothesis Spaces For Minimum Bayes Risk Training In Large Vocabulary Speech Recognition," in Interspeech, 2006.
18. Daniel Povey and Brian Kingsbury, "Evaluation of proposed modifications to MPE for large scale discriminative training," in ICASSP, 2007.
19. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle, "Greedy layer-wise training of deep networks," Advances in neural information processing systems (NIPS), vol. 19, pp. 153, 2007.
20. Feng Niu, Benjamin Recht, Christopher Ré, and Stephen J Wright, "Hogwild!: A lock-free approach to parallelizing stochastic gradient descent," arXiv preprint arXiv:1106.5730, 2011.
21. Jeffrey Dean and Greg S. Corrado and Rajat Monga and Kai Chen and Matthieu Devin and Quoc V. Le and Mark Z. Mao and Marc'Aurelio Ranzato and Andrew Senior and Paul Tucker and Ke Yang and Andrew Y. Ng, "Large Scale Distributed Deep Networks," in Neural Information Processing Systems (NIPS), 2012.
22. Noboru Murata and Shun-ichi Amari, "Statistical analysis of learning dynamics," Signal Processing, vol. 74, no. 1, pp. 3-28, 1999.
23. Elad Hazan, Alexander Rakhlin, and Peter L Bartlett, "Adaptive online gradient descent," in Advances in Neural Information Processing Systems (NIPS), 2007, pp. 65-72.
24. Andrew Senior, Georg Heigold, Marc'Aurelio Ranzato, and Ke Yang, "An empirical study of learning rates in deep neural networks for speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2013.
25. Xavier Glorot, Antoine Bordes, and Yoshua Bengio, "Deep sparse rectifier networks," in Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, 2011, vol. 15, pp. 315-323.
26. D. Povey, L. Burget, et al., "The Subspace Gaussian Mixture Model-A Structured Model for Speech Recognition," Computer Speech & Language, vol. 25, no. 2, pp. 404-439, April 2011.
27. M. J. F. Gales and P. C.Woodland, "Mean and Variance Adaptation Within the MLLR Framework," Computer Speech and Language, vol. 10, pp. 249-264, 1996.
28. Daniel PWEllis and Byunk Suk Lee, "Noise robust pitch tracking by subband autocorrelation classification," in 13th Annual Conference of the International Speech Communication Association, 2012.
29. Kornel Laskowski, Mattias Heldner, and Jens Edlund, "The fundamental frequency variation spectrum," in FONETIK, 2008.