On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2016-May, Issue , 2016, Pages 5970-5974

  Prabhavalkar, Rohit ^a   Alsharif, Ouais ^a   Bruguier, Antoine ^a   McGraw, Lan ^a  

^a GOOGLE INC   (United States)

Author keywords

embedded speech recognition; LSTM; model compression; RNN; SVD

Indexed keywords

EID: 84973402069     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2016.7472823     Document Type: Conference Paper

References (28)

1. F. Seide, G. Li, and D. Yu, "Conversational speech transcription using context-dependent deep neural networks, " in Proc. of Interspeech, 2011, pp. 437-440
2. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012
3. H. Sak, A. Senior, and F. Beaufays, "Long short-term memory recurrent neural network architectures for large scale acoustic modeling, " in Proc. of Interspeech, 2014, pp. 338-342
4. T. N. Sainath, O. Vinyals, A. Senior, and H. Sak, "Convolutional, long short-term memory, fully connected deep neural networks, " in Proc. of ICASSP, 2015, pp. 4580-4584
5. L. Deng and D. Yu, "Deep learning: methods and applications, " Foundations and Trends in Signal Processing, vol. 7, no. 3-4, pp. 197-387, 2014
6. J. Schalkwyk, D. Beeferman, F. Beaufays, B. Byrne, C. Chelba, M. Cohen, M. Kamvar, and B. Strope, ""YourWord is my Command": google search by voice: A case study, " in Advances in Speech Recognition, pp. 61-90. Springer US, 2010
7. X. Lei, A. Senior, A. Gruenstein, and J. Sorensen, "Accurate and compact large vocabulary speech recognition on mobile devices, " in Proc. of Interspeech, 2013, pp. 662-665
8. J. Xue, J. Li, and Y. Gong, "Restructuring of deep neural network acoustic models with singular value decomposition, " in Proc. of Interspeech, 2013, pp. 2365-2369
9. J. Xue, J. Li, D. Yu, M. Seltzer, and Y. Gong, "Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network, " in Proc. of ICASSP, 2014, pp. 6359-6363
10. A. Graves, M. Liwicki, S. Fernández, R. Bertolami, H. Bunke, and J. Schmidhuber, "A novel connectionist system for unconstrained handwriting recognition, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, pp. 855-868, 2009
11. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks, " in Proc. of NIPS, 2014, pp. 3104-3112
12. M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas, "Predicting parameters in deep learning, " in Proc. of NIPS, 2013, pp. 2148-2156
13. C. Bucilua, R. Caruana, and A. Niculescu-Mizil, "Model compression, " in Proc. of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006, pp. 535-541
14. L. J. Ba and R. Caruana, "Do deep nets really need to be deep, " in Proc. of NIPS, 2014, pp. 2654-2662
15. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network, " arXiv preprint arXiv: 1503. 02531, 2015
16. W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, "Compressing neural networks with the hashing trick, " in Proc. of ICML, 2015, pp. 2285-2294
17. Y. LeCun, J. S. Denker, and S. A. Solla, "Optimal brain damage, " in Proc. of NIPS, 1989, pp. 598-605
18. F. Grézl and P. Fousek, "Optimizing bottle-neck features for LVCSR, " in Proc. of ICASSP, March 2008, pp. 4729-4732
19. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets, " in Proc. of ICASSP, 2013, pp. 6655-6659
20. Y. Wang, J. Li, and Y. Gong, "Small-footprint highperformance deep neural network-based speech recognition using split-VQ, " in Proc. of ICASSP, 2015, pp. 4984-4988
21. P. Nakkiran, R. Alvarez, R. Prabhavalkar, and C. Parada, "Compressing deep neural networks using a rankconstrained topology, " in Proc. of Interspeech, 2015, pp. 1473-1477
22. H. Sak, A. Senior, K. Rao, O. Irsoy, A. Graves, F. Beaufays, and J. Schalkwyk, "Learning acoustic frame labeling for speech recognition with recurrent neural networks, " in Proc. of ICASSP, 2015, pp. 4280-4284
23. H. Sak, A. Senior, K. Rao, and F. Beaufays, "Fast and accurate recurrent neural network acoustic models for speech recognition, " in Proc. of Interspeech, 2015, pp. 1468-1472
24. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks, " in Proc. of ICML, 2006, pp. 369-376
25. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng, "Large scale distributed deep networks, " in Proc. of NIPS, 2012, pp. 1223-1231
26. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " in Proc. of ICASSP, 2009, pp. 3761-3764
27. H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao, "Sequence discriminative distributed training of long short-term memory recurrent neural networks, " in Proc. of Interspeech, 2014, pp. 1209-1213
28. I. McGraw, R. Prabhavalkar, R. Alvarez, M. Gonzalez Arenas, K. Rao, D. Rybach, O. Alsharif, H. Sak, A. Gruenstein, F. Beaufays, and C. Parada, "Personalized speech recognition on mobile devices, " in Proc. of ICASSP, 2016.