Regularization of context-dependent deep neural networks with context-independent multi-task training

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 4290-4294

  Bell, Peter ^a   Renals, Steve ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

deep neural networks; multitask learning; regularization

Indexed keywords

AUDIO SIGNAL PROCESSING; SPEECH COMMUNICATION; SPEECH RECOGNITION;

AUTOMATIC SPEECH RECOGNITION; BASELINE SYSTEMS; CONTEXT DEPENDENT; CONTEXT INDEPENDENT; LARGE VOCABULARY; LECTURE RECOGNITION; MULTITASK LEARNING; REGULARIZATION;

DEEP NEURAL NETWORKS;

EID: 84946094751     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7178780     Document Type: Conference Paper

References (30)

1. S. J. Young and P. C. Woodland, "State clustering in hidden Markov model-based continuous speech recognition," Computer Speech &Language, vol. 8, no. 4, pp. 369-383, 1994
2. M. M. Hochberg, S. J. Renals, A. J. Robinson, and G. D. Cook, "Recent improvements to the ABBOT large vocabulary CSR system," in Proc. IEEE ICASSP, 1995, pp. 69-72
3. A. J. Robinson, G. D. Cook, D. P. W. Ellis, E. Fosler-Lussier, S. J. Renals, and D. A. G. Williams, "Connectionist speech recognition of broadcast news," Speech Communication, vol. 37,no.I-2,pp. 27-45,2002
4. H Bourlard, N Morgan, C Wooters, and S Renals, "CONN: a context dependent neural network for continuous speech recognition," in Proc. IEEE ICASSP, 1992, vol. 2, pp. 349-352
5. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach, Kluwer Academic Publishers, 1994
6. D. Kershaw, T. Robinson, and S. Renals, "The 1995 ABBOT LVCSR system for multiple unknown microphones," in Proc. ICSLP, 1996, pp. 1325-1328
7. H. Hermanksy, D.P.W. Ellis, and S. Sharma, "Tandem connectionist feature extraction for conventional HMM systems," in Proc. ICASSP, 2000, pp. 1635-1630
8. F. Grezl, M Karafiat, S. Kontar, and J. Cernokcy, "Probabilistic and bottleneck features for LVCSR of meetings," in Proc. ICASSP, 2007
9. G.E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," IEEE Transactions on Audio, Speech and Language Processing, vol. 20, no. 1, pp. 30-42,2012
10. F. Seide, G. Li, X. Chen, and D. Yu, "Feature engineering in context-dependent deep neural networks for conversational speech transcription," in Proc. ASRU, 2011
11. G Hinton, L. Deng, D. Yu, G.E. Dahl, A-R Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T.N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, 2012
12. Michiel Bacchiani and David Rybach, "Context dependent state tying for speech recognition using deep neural network acoustic models," in Proc. ICASSP, 2014, pp. 230-234
13. C. Zhang and P. C. Woodland, "Standalone training of contextdependent deep neural network acoustic models," in Proc. ICASSP, Florence, Italy, 2014
14. D. Povey and P.c. Woodland, "Minimum phone error and I-smoothing for improved discriminative training," in Proc. ICASSP. IEEE, 2002, vol. I, pp. 105-108
15. R. Caruana, "Multitask learning," Machine learning, vol. 28, pp. 41-75, 1997
16. S. Parveen and P. Green, "Multi task learning in connectionist robust ASR using recurrent neural networks," in Proc. Interspeech, 2003
17. D. Chen, B. Mak, c.-C. Leung, and S. Sivadas, "Joint acoustic modelling of triphones and trigraphemes by multi-task learning deep neural networks for low-resource speech recognition," in Proc. ICASSP, 2014
18. Z. Ttiske, R. Schli.iter, and H. Ney, "Multi-lingual hierarchical MRASTA features for ASR," in Proc. Interspeech, 2013
19. P. Bell, J. Driesen, and S. Renals, "Cross-lingual adaptation with multi-task adaptive networks," in Proc. Interspeech, 2014
20. M. Selzer and J. Droppo, "Multi-task learning in deep neural networks for improved phoneme recognition," in Proc. ICASSP, 2013
21. G. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Computation, vol. 18, pp. 1527-1554,2006
22. Y. Bengio, J. Louradour, R. Coliobert, and J. Weston, "Curriculum learning," in Proc. ICML, 2009
23. C. Zhang and P.c. Woodland, "Context independent discriminative pre-training," Unpublished work
24. D. Yu, K. Yao, H. Su, G. Li, and F. Seide, "KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition," in Proc ICASSP, 2013
25. J Bergstra, O Breuleux, F Bastien, P Lamblin, R Pascanu, G Desjardins, J Turian, O Warde-Farley, and Y Bengio, "Theano: a CPU and GPU math expression compiler," in Proc. SciPy, June 2010
26. P. Bell, H. Yamamoto, P. Swietojanski, Y. Wu, F. Mcinnes, C. Hori, and S. Renals, "A lecture transcription system combining neural network acoustic and language models," in Proc. Interspeech, Aug. 2013
27. M.J.F. Gales, "Maximum likelihood linear transforms for HMM-based speech recognition," Computer Speech and Language, vol. 12, no. 75-98, 1998
28. P. Bell, P. Swietojanski, and S. Renals, "Multi-level adaptive networks in tandem and hybrid ASR systems," in Proc. ICASSP, 2013
29. M. Cettolo, J. Niehues, S. Stliker, L. Bentivogli, and M. Frederico, "Report on the 10th IWSLT evaluation campaign," in Proc. International Workshop on Spoken Language Translation, Heidelberg, Germany, December 2013
30. M. Cettolo, C. Girardi, and M Federico, "Wit3: Web inventory of transcribed and translated talks," in Proceedings of the lt1-h Conference of the European Association for Machine Translation (EAMT), Trento, Italy, May 2012, pp. 261-268