Improving language-universal feature extraction with deep maxout and convolutional neural networks

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 800-804

  Miao, Yajie ^a   Metze, Florian ^a  

^a Language Technologies Institute School of Computer Science Carnegie Mellon University ^*  (United States)

Author keywords

Deep convolutional networks; Deep maxout networks; Language universal feature extraction

Indexed keywords

COMPLEX NETWORKS; CONVOLUTION; EXTRACTION; FEATURE EXTRACTION; NEURAL NETWORKS; SPEECH COMMUNICATION;

AUTOMATED SPEECH RECOGNITION; CONVOLUTIONAL NETWORKS; CONVOLUTIONAL NEURAL NETWORK; DEEP NEURAL NETWORKS; LINEAR CLASSIFIERS; SIGMOID NONLINEARITY; UNIVERSAL FEATURE EXTRACTORS; WORD ERROR RATE REDUCTIONS;

SPEECH RECOGNITION;

EID: 84910028405     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. G. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for large vocabulary speech recognition, " IEEE Transactions on Audio, Speech and Language Processing, vol. 20(1), pp. 30-42, 2012.
2. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. ASRU, pp. 24-29, 2011.
3. D. Yu, M. L. Seltzer, J. Li, J. Huang, and F. Seide, "Feature learning in deep neural networks - studies on speech recognition tasks, " in International Conference on Learning Representations 2013.
4. J. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, "Crosslanguage knowledge transfer using multilingual deep neural network with shared hidden layers, " in Proc. ICASSP, pp. 7304-7308, 2013.
5. Y. Miao, F. Metze, and S. Rawat, "Deep maxout networks for low-resource speech recognition, " in Proc. ASRU, 2013.
6. Y. Miao, H. Zhang, and F. Metze, "Distributed learning of multilingual DNN feature extractors using GPUs, " to appear in Proc. Interspeech, 2014.
7. H. Lee, C. Ekanadham, and A. Y. Ng, "Sparse deep belief net model for visual area V2, " in Proc. NIPS, 2008.
8. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, " in Proc. ICML, pp. 609-616, 2009.
9. J. Ngiam, P. Koh, Z. Chen, S. Bhaskar, and A. Y. Ng, "Sparse filtering, " in Proc. NIPS, 2013.
10. G. Sivaram, S.K. Nemala, M. Elhilali, T.D. Tran, and H. Hermansky, "Sparse coding for speech recognition, " in Proc. ICASSP, pp. 4346-4349, 2010.
11. H. Lee, Y. Largman, P. Pham, and A. Y. Ng, "Unsupervised feature learning for audio classification using convolutional deep belief networks, " in Proc. NIPS, 2009.
12. G. Sivaram, and H. Hermansky, "Multilayer perceptron with sparse hidden outputs for phoneme recognition, " in Proc. ICASSP, pp. 5336-5339, 2011.
13. O. Vinyals, and L. Deng, "Are sparse representations rich enough for acoustic modeling?, " in Proc. Interspeech, 2012.
14. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, "Maxout networks, " in Proc. ICML, 2013.
15. X. Zhang, J. Trmal, D. Povey, and S. Khudanpur, "Improving deep neural network acoustic models using generalized maxout networks, " in Proc. ICASSP, 2014.
16. O. Abdel-Hamid, A. Mohamed, H. Jiang, and G. Penn, "Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition, " in Proc. ICASSP, pp. 4277-4280, 2012.
17. T. N. Sainath, A. Mohamed, B. Kingsbury, and B. Ramabhadran, "Deep convolutional neural networks for LVCSR, " in Proc. ICASSP, pp. 8614-8618, 2013.
18. O. Abdel-Hamid, L. Deng, and D. Yu, "Exploring convolutional neural network structures and optimization techniques for speech recognition, " in Proc. Interspeech, pp. 3366-3370, 2013.
19. T. N. Sainath, B. Kingsbury, A. Mohamed, G. Dahl, G. Saon, H. Soltau, T. Beran, A. Aravkin, and B. Ramabhadran, "Improvements to deep convolutional neural networks for LVCSR, " in Proc. ASRU, 2013.
20. J. Gehring, W. Lee, K. Kilgour, I. Lane, Y. Miao, and A. Waibel, "Modular combination of deep neural networks for acoustic modeling, " in Proc. Interspeech, pp. 94-98, 2013.
21. J. Gehring, Y. Miao, F. Metze, and A. Waibel, "Extracting deep bottleneck features using stacked autoencoders, " in Proc. ICASSP, pp. 3377-3381, 2013.
22. J. Chiu, and A. Rudnicky, "Using conversational word bursts in spoken term detection, " in Proc. Interspeech, 2013.
23. J. Chiu, Y. Wang, J. Trmal, D. Povey, G. Chen, and A. Rudnicky, "Combination of FST and CN search in spoken term detection, " to appear in Proc. Interspeech, 2014.
24. Y. Miao, and F. Metze, "Improving low-resource CDDNN- HMM using dropout and multilingual DNN training, " in Proc. Interspeech, pp. 2237-2241, 2013.
25. Y. Miao, F. Metze, and A. Waibel, "Subspace mixture model for low-resource speech recognition in crosslingual settings, " in Proc. ICASSP, pp. 7339-7342, 2013.
26. X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse rectifier neural networks, " in Proc. AISTATS, 2011.
27. G. Dahl, T. N. Sainath, and G. E. Hinton, "Improving deep neural networks for LVCSR using rectified linear units and dropout, " in Proc. ICASSP, pp. 8609-8613, 2013.
28. L. Toth, "Phone recognition with deep sparse rectifier neural networks, " in Proc. ICASSP, pp. 6985-6989, 2013.
29. A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models, " in ICML Workshop on Deep Learning for Audio, Speech, and Language Processing (WDLASL), 2013.
30. Y. Miao, "Kaldi+PDNN: Building DNN-based ASR systems with Kaldi and PDNN, " arXiv:1401.6984, 2014.
31. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol, "Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion, " Journal of Machine Learning Research, vol. 11, 2010.
32. G. E. Hinton, "A practical guide to training restricted Boltzmann machines, " UTML TR., 2010.
33. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors, " arXiv:1207.0580, 2012.
34. Y. Miao, H. Zhang, and F. Metze, "Towards speaker adaptive training of deep neural network acoustic models, " to appear in Proc. Interspeech, 2014.
35. U. Nallasamy, M. Fuhs, M. Woszczyna, F. Metze, and T. Schultz, "Neighbour selection and adaptation for rapid speaker-dependent ASR, " in Proc. ASRU, pp. 60-65, 2013.