Structured output layer with auxiliary targets for context-dependent acoustic modelling

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2015-January, Issue , 2015, Pages 3605-3609

  Swietojanski, Pawel ^a   Bell, Peter ^a   Renals, Steve ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Adaptation; Deep neural networks; Multitask learning; Structured output layer

Indexed keywords

SPEECH RECOGNITION; TREES (MATHEMATICS);

ADAPTATION; CONSTRAINED CONDITIONS; CONTEXT INDEPENDENT; DEEP NEURAL NETWORKS; LARGE VOCABULARY SPEECH RECOGNITION; MULTITASK LEARNING; OUTPUT LAYER; UNSUPERVISED SPEAKER ADAPTATION;

SPEECH COMMUNICATION;

EID: 84959095902     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (42)

1. S. J. Young and P. C. Woodland, "State clustering in hidden Markov model-based continuous speech recognition, " Computer Speech & Language, vol. 8, no. 4, pp. 369-383, 1994.
2. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, 1994.
3. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco, "Connectionist probability estimators in HMM speech recognition, " IEEE Transactions on Speech and Audio Processing, vol. 2, no. 1, pp. 161-174, 1994.
4. G. Hinton, L. Deng, D. Yu, G. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, 2012.
5. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pretrained deep neural networks for large-vocabulary speech recognition, " IEEE Transactions on Audio, Speech and Language Processing, vol. 20, no. 1, pp. 30-42, 2012.
6. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in Proc. Interspeech, 2011.
7. N. Morgan and H. Bourlard, "Factoring networks by a statistical method, " Neural Computation, vol. 4, no. 6, pp. 835-838, 1992.
8. H. Bourlard, N. Morgan, C. Wooters, and S. Renals, "CDNN: A context dependent neural network for continuous speech recognition, " in Proc. ICASSP, vol. 2, 1992, pp. 349-352.
9. H. Franco, M. Cohen, N. Morgan, D. Rumelhart, and V. Abrash, "Context-dependent connectionist probability estimation in a hybrid HMM-neural net speech recognition system, " Computer Speech and Language, vol. 8, pp. 211-222, 1994.
10. G. Wang and K. C. Sim, "Regression-based context-dependent modeling of deep neural networks for speech recognition, " Audio, Speech, and Language Processing, IEEE/ACM Transactions on, vol. 22, no. 11, pp. 1660-1669, Nov 2014.
11. M. Bacchiani and D. Rybach, "Context dependent state tying for speech recognition using deep neural network acoustic models, " in Proc. ICASSP, 2014, pp. 230-234.
12. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription, " in Proc. ASRU, 2011.
13. C. Zhang and P. Woodland, "Context independent discriminative pre-training, " unpublished work.
14. P. Swietojanski, A. Ghoshal, and S. Renals, "Unsupervised crosslingual knowledge transfer in DNN-based LVCSR, " in Proc. IEEE SLT, December 2012, pp. 246-251.
15. Y. Miao and F. Metze, "Improving low-resource cd-dnn-hmm using dropout and multilingual dnn training. " in Proc. Interspeech. ISCA, 2013, pp. 2237-2241.
16. R. Caruana, "Multitask learning, " Machine learning, vol. 28, pp. 41-75, 1997.
17. S. Parveen and P. Green, "Multitask learning in connectionist robust ASR using recurrent neural networks, " in Proc. Interspeech, 2003.
18. J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, "Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers, " in Proc. ICASSP, 2013.
19. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, "Multilingual acoustic models using distributed deep neural networks, " in In Proc. ICASSP, 2013.
20. A. Ghoshal, P. Swietojanski, and S. Renals, "Multilingual training of deep neural networks, " in In Proc. ICASSP, 2013.
21. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks, " in Advances in Neural Information Processing Systems 19, 2007, pp. 153-160.
22. P. Bell and S. Renals, "Regularization of context-dependent deep neural networks with context-independent multi-task training, " in Proc. ICASSP, 2015.
23. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, "Curriculum learning, " in Proc. ICML, 2009.
24. D. Chen, B. Mak, C.-C. Leung, and S. Sivadas, "Joint acoustic modelling of triphones and trigraphemes by multi-task learning deep neural networks for low-resource speech recognition, " in Proc. ICASSP, 2014.
25. M. Seltzer and J. Droppo, "Multi-task learning in deep neural networks for improved phoneme recognition, " in Proc. ICASSP, 2013.
26. M. Minsky and S. Papert, Perceptrons. MIT Press, 1969.
27. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error-propagation, " in Parallel Distributed Processing. MIT Press, 1986, vol. 1, pp. 318-362.
28. M. á. Carreira-Perpinãn and W. Wang, "Distributed optimization of deeply nested systems, " CoRR, vol. abs/1212. 5921, 2012. [Online]. Available: http: //arxiv. org/abs/1212. 5921
29. M. Cettolo, C. Girardi, and M. Federico, "Wit3: Web inventory of transcribed and translated talks, " in Proc EAMT, 2012, pp. 261-268.
30. J. J. Godfrey, E. C. Holliman, and J. McDaniel, "SWITCHBOARD: Telephone speech corpus for research and development, " in Proc. ICASSP. IEEE, 1992, pp. 517-520.
31. E. Hasler, P. Bell, A. Ghoshal, B. Haddow, P. Koehn, F. McInnes, S. Renals, and P. Swietojanski, "The UEDIN systems for the IWSLT 2012 evaluation, " in Proc. IWSLT, 2012.
32. P. Swietojanski, A. Ghoshal, and S. Renals, "Revisiting hybrid and GMM-HMM system combination techniques, " in Proc. ICASSP, 2013.
33. P. Bell, P. Swietojanski, J. Driesen, M. Sinclair, F. McInnes, and S. Renals, "The UEDIN system for the IWSLT 2014 evaluation, " in Proc. IWSLT, 2014.
34. H. Hermansky, "Perceptual linear predictive (PLP) analysis of speech, " The Journal of the Acoustical Society of America, vol. 87, no. 4, pp. 1738-1752, Apr. 1990.
35. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets. " in Proc. ICASSP, 2013, pp. 6655-6659.
36. F. GrÉzl, M. Karafiát, S. Kontar, and J. Cernokcý, "Probabilistic and bottleneck features for LVCSR of meetings, " in Proc. ICASSP, 2007.
37. K. Vesely, A. Ghoshal, L. Burget, and D. Povey, "Sequencediscriminative training of deep neural networks, " in Proc. Interspeech, Lyon, France, August 2013.
38. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlícek, Y. Qian, P. Schwarz, J. Silovský, G. Stemmer, and K. Veselý, "The Kaldi speech recognition toolkit, " in Proc. IEEE ASRU, December 2011.
39. R. Price, K. Iso, and K. Shinoda, "Speaker adaptation of deep neural networks using a hierarchy of output layers, " in Proc. IEEE SLT, 2014.
40. P. Swietojanski and S. Renals, "Learning hidden unit contributions for unsupervised speaker adaptation of neural network acoustic models, " in Proc. IEEE SLT, 2014.
41. O. Abdel-Hamid and H. Jiang, "Rapid and effective speaker adaptation of convolutional neural network based models for speech recognition. " in Proc. Interspeech. ISCA, pp. 1248-1252.
42. I. Goodfellow, D. Warde-Farley, P. Lamblin, V. Dumoulin, M. Mirza, R. Pascanu, J. Bergstra, F. Bastien, and Y. Bengio, "Pylearn2: A machine learning research library, " arXiv: 1308. 4214, 2013.