Dynamic layer normalization for adaptive neural acoustic modeling in speech recognition

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2017-August, Issue , 2017, Pages 2411-2415

  Kim, Taesup ^a   Song, Inchul ^b   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b Samsung Advanced Institute of Technology (SAIT)   (South Korea)

Author keywords

Adaptive acoustic model; Dynamic layer normalization; Speech recognition

Indexed keywords

DEEP NEURAL NETWORKS; SPEECH; SPEECH COMMUNICATION;

ACOUSTIC MODEL; ACOUSTIC VARIABILITY; ADAPTATION PARAMETERS; BENCHMARK DATASETS; CHANNEL NOISE; LARGE VOCABULARY; MODEL SIZE; TRAINING SPEED;

SPEECH RECOGNITION;

EID: 85039151782     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: 10.21437/Interspeech.2017-556     Document Type: Conference Paper

References (29)

1. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, Vol. 29, no. 6, pp. 82-97, 2012.
2. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., Vol. 9, no. 8, pp. 1735-1780, Nov. 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735
3. A. Graves, S. Fernandez, and J. Schmidhuber, Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition, 2005, pp. 799-804.
4. H. Sak, A. W. Senior, and F. Beaufays, "Long short-term memory recurrent neural network architectures for large scale acoustic modeling." in Interspeech, 2014, pp. 338-342.
5. G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, "Speaker adaptation of neural network acoustic models using i-vectors." in ASRU, 2013, pp. 55-59.
6. V. Gupta, P. Kenny, P. Ouellet, and T. Stafylakis, "I-vector-based speaker adaptation of deep neural networks for french broadcast audio transcription," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 6334-6338.
7. T. Tan, Y. Qian, D. Yu, S. Kundu, L. Lu, K. C. Sim, X. Xiao, and Y Zhang, "Speaker-aware training of lstm-rnns for acoustic modelling," in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on. IEEE, 2016, pp. 5280-5284.
8. N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, "Front-end factor analysis for speaker verification," IEEE Trans. Audio, Speech & Language Processing, Vol. 19, no. 4, pp. 788-798, 2011. [Online]. Available: http://dx.doi.org/10.1109/TASL.2010.2064307
9. H. Liao, "Speaker adaptation of context dependent deep neural networks," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7947-7951.
10. P. Swietojanski and S. Renals, "Learning hidden unit contributions for unsupervised speaker adaptation of neural network acoustic models," in Spoken Language Technology Workshop (SLT), 2014 IEEE. IEEE, 2014, pp. 171-176.
11. P. Swietojanski, J. Li, and S. Renals, "Learning hidden unit contributions for unsupervised acoustic model adaptation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 24, no. 8, pp. 1450-1463, 2016.
12. Y Miao, H. Zhang, and F. Metze, "Speaker adaptive training of deep neural network acoustic models using i-vectors," IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), Vol. 23, no. 11, pp. 1938-1949, 2015.
13. L. J. Ba, R. Kiros, and G. E. Hinton, "Layer normalization," CoRR, Vol. abs/1607.06450, 2016. [Online]. Available: http://arxiv.org/abs/1607.06450
14. D. Ulyanov, V. Lebedev, A. Vedaldi, and V. S. Lempitsky, "Texture networks: Feed-forward synthesis of textures and stylized images," CoRR, Vol. abs/1603.03417, 2016. [Online]. Available: http://arxiv.org/abs/1603.03417
15. J. Johnson, A. Alahi, and F. Li, "Perceptual losses for real-time style transfer and super-resolution," CoRR, Vol. abs/1603.08155, 2016. [Online]. Available: http://arxiv.org/abs/1603.08155
16. D. Ulyanov, A. Vedaldi, and V. S. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," CoRR, Vol. abs/1607.08022, 2016. [Online]. Available: http://arxiv.org/abs/1607.08022
17. V. Dumoulin, J. Shlens, and M. Kudlur, "A learned representation for artistic style," CoRR, Vol. abs/1610.07629, 2016. [Online]. Available: http://arxiv.org/abs/1610.07629
18. D. B. Paul and J. M. Baker, "The design for the wall street journal-based csr corpus," in Proceedings of the workshop on Speech and Natural Language. Association for Computational Linguistics, 1992, pp. 357-362.
19. A. Rousseau, P. Deléglise, and Y Estève, "Enhancing the tedlium corpus with selected data for language modeling and more ted talks." in LREC, 2014, pp. 3935-3939.
20. S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift." in ICML, ser. JMLR Workshop and Conference Proceedings, F. R. Bach and D. M. Blei, Eds., Vol. 37. JMLR.org, 2015, pp. 448-456.
21. D. Ha, A. Dai, and Q. V. Le, "Hypernetworks," arXiv preprint arXiv:1609.09106, 2016.
22. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural networks," IEEE Transactions on Signal Processing, Vol. 45, no. 11, pp. 2673-2681, Nov 1997.
23. A. Graves, N. Jaitly, and A.-r. Mohamed, "Hybrid speech recognition with deep bidirectional lstm," in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 273-278.
24. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," CoRR, Vol. abs/1412.6980, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980
25. M. Henaff, A. Szlam, and Y LeCun, "Orthogonal rnns and long-memory tasks," arXiv preprint arXiv:1602.06662, 2016.
26. Theano Development Team, "Theano: A Python framework for fast computation of mathematical expressions," arXiv eprints, Vol. abs/1605.02688, May 2016. [Online]. Available: http://arxiv.org/abs/1605.02688
27. S. Dieleman, J. Schlüter, C. Raffel, E. Olson, S. K. Sønderby, D. Nouri, D. Maturana, M. Thoma, E. Battenberg, J. Kelly, J. D. Fauw, M. Heilman, D. M. de Almeida, B. McFee, H. Weideman, G. Takâcs, P. de Rivaz, J. Crall, G. Sanders, K. Rasul, C. Liu, G. French, and J. Degrave, "Lasagne: First release." Aug. 2015. [Online]. Available: http://dx.doi.org/10.5281/zenodo.27878
28. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y Qian, P. Schwarz et al, "The kaldi speech recognition toolkit," in IEEE 2011 workshop on automatic speech recognition and understanding, no. EPFLCONF-192584. IEEE Signal Processing Society, 2011.
29. L. van der Maaten and G. E. Hinton, "Visualizing high-dimensional data using t-sne," Journal of Machine Learning Research, Vol. 9, pp. 2579-2605, 2008.