On the efficient representation and execution of deep acoustic models

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 08-12-September-2016, Issue , 2016, Pages 2746-2750

  Alvarez, Raziel ^a   Prabhavalkar, Rohit ^a   Bakhtin, Anton ^a  

^a GOOGLE INC   (United States)

Author keywords

Acoustic modeling; Compression; Deep neural networks; Embedded speech recognition; Quantization

Indexed keywords

COMPACTION; DIGITAL ARITHMETIC; INTEGER PROGRAMMING; SPEECH COMMUNICATION; SPEECH PROCESSING;

ACOUSTIC MODEL; COMPUTATIONALLY EFFICIENT; DEEP NEURAL NETWORKS; INTEGER ARITHMETIC; LARGE VOCABULARY SPEECH RECOGNITION; LONG SHORT TERM MEMORY; QUANTIZATION; QUANTIZATION SCHEMES;

SPEECH RECOGNITION;

EID: 84994259689     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: 10.21437/Interspeech.2016-128     Document Type: Conference Paper

References (27)

1. X. Lei, A. Senior, A. Gruenstein, and J. Sorensen, "Accurate and compact large vocabulary speech recognition on mobile devices." in INTERSPEECH. ISCA, 2013, pp. 662-665.
2. I. McGraw, R. Prabhavalkar, R. Alvarez, M. G. Arenas, K. Rao, D. Rybach, O. Alsharif, H. Sak, A. Gruenstein, F. Beaufays, and C. Parada, "Personalized speech recognition on mobile devices," in Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016.
3. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in CVPR 2015, 2015. [Online]. Available: http://arxiv.org/abs/1409.4842
4. O. Alsharif, T. Ouyang, F. Beaufays, S. Zhai, T. Breuel, and J. Schalkwyk, "Long short term memory neural network for keyboard gesture decoding," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, 2015, pp. 2076-2080. [Online]. Available: http://dx.doi.org/10.1109/ICASSP.2015.7178336
5. E. Ozturk, J. Guilford, V. Gopal, and W. Feghali, "New instructions supporting large integer arithmetic on intel architecture processors," Intel Corporation, Tech. Rep., aug 2012.
6. Cortex-A9 NEON Media Processing Engine: Technical reference manual, ARM, jun 2012.
7. H. Zen, Y. Agiomyrgiannakis, N. Egberts, F. Henderson, and P. Szczepaniak, "Optimizing lstm-rnn based statistical parametric speech synthesizers for production deployment," in Proc. Interspeech (to appear), 2016.
8. Y. Xie and M. A. Jabri, "Analysis of the effects of quantization in multilayer neural networks using a statistical model," IEEE Transactions on Neural Networks, vol. 3, no. 2, pp. 334-338, Mar 1992.
9. G. Dündar and K. Rose, "The effects of quantization on multilayer neural networks," IEEE Transactions on Neural Networks, vol. 6, no. 6, pp. 1446-1451, Nov 1995.
10. V. Vanhoucke, A. Senior, and M. Mao, "Improving the speed of neural networks on cpus," in Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 2011.
11. S. Han, H. Mao, and W. J. Dally, "Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding," CoRR, vol. abs/1510.00149, 2015. [Online]. Available: http://arxiv.org/abs/1510.00149
12. K. Hwang and W. Sung, "Fixed-point feedforward deep neural network design using weights-1, 0, and +1," in Signal Processing Systems (SiPS), 2014 IEEE Workshop on, Oct 2014, pp. 1-6.
13. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, "1-bit stochastic gradient descent and application to dataparallel distributed training of speech dnns," in Interspeech 2014, September 2014. [Online]. Available: http://research.microsoft.com/apps/pubs/default.aspx?id=230137
14. M. Kim and P. Smaragdis, "Bitwise neural networks," CoRR, vol. abs/1601.06071, 2016. [Online]. Available: http://arxiv.org/abs/1601.06071
15. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, "Deep learning with limited numerical precision," CoRR, vol. abs/1502.02551, 2015. [Online]. Available: http://arxiv.org/abs/1502.02551
16. M. Courbariaux, Y. Bengio, and J. David, "Binaryconnect: Training deep neural networks with binary weights during propagations," CoRR, vol. abs/1511.00363, 2015. [Online]. Available: http://arxiv.org/abs/1511.00363
17. J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, "Quantized convolutional neural networks for mobile devices," CoRR, vol. abs/1512.06473, 2015. [Online]. Available: http://arxiv.org/abs/1512.06473
18. W. Sung, S. Shin, and K. Hwang, "Resiliency of deep neural networks under quantization," CoRR, vol. abs/1511.06488, 2015. [Online]. Available: http://arxiv.org/abs/1511.06488
19. H. Sak, A.W. Senior, and F. Beaufays, "Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition," CoRR, vol. abs/1402.1128, 2014. [Online]. Available: http://arxiv.org/abs/1402.1128
20. A. C. Bovik, Handbook of Image and Video Processing (Communications, Networking and Multimedia). Orlando, FL, USA: Academic Press, Inc., 2005.
21. K. Y. Tan and Y.-H. Leung, "Roundoff errors in fixed-point digital filters," in 2nd International Conference on Information and Systems Sciences, Y. Zhang and J. Liu, Eds. Institute for Scientific Computing and Information, dec 2008, pp. 611-619.
22. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. Springer US, 1992.
23. R. Prabhavalkar, O. Alsharif, A. Bruguier, and I. McGraw, "On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition," in ICASSP. IEEE, 2016.
24. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, "Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks," in ICML, 2006, pp. 369-376. [Online]. Available: http://doi.acm.org/10.1145/1143844.1143891
25. B. Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling," in ICASSP, 2009, pp. 3761-3764.
26. H. Sak, A. Senior, K. Rao, and F. Beaufays, "Fast and accurate recurrent neural network acoustic models for speech recognition," in INTERSPEECH. ISCA, 2015, pp. 1468-1472.
27. A. Senior, H. Sak, F. de Chaumont Quitry, T. N. Sainath, and K. Rao, "Acoustic modelling with cd-ctc-smbr lstm rnns," in ASRU, 2015.