Structured transforms for small-footprint deep learning

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 3088-3096

  Sindhwani, Vikas ^a   Sainath, Tara N ^a   Kumar, Sanjiv ^a  

^a GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION SCIENCE; MOBILE DEVICES;

DISPLACEMENT RANKS; KEYWORD SPOTTING; PARAMETER SHARING; SMALL FOOTPRINTS; STATE OF THE ART; STATISTICAL MODELING; STRUCTURED PARAMETERS; UNIFIED FRAMEWORK;

SPEECH RECOGNITION;

EID: 84965177696     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Supplementary material: Structured transforms for small footprint deep learning. 2015. http://vikas.sindhwani.org/st-supplementary.pdf.
2. G. Chen, C. Parada, and G. Heigold. Small-footprint keyword spotting using deep neural networks. In ICASSP, 2014.
3. W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen. Compressing neural networks with the hashing trick. In ICML, 2015.
4. Y. Cheng, F. X. Xu, R. S. Feris, S. Kumar, A. Choudhary, and S.-F. Chang. Fast neural networks with circulant projections. In arXiv:1502.03436, 2015.
5. D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and Schmidhuber. High-performance neural networks for visual object classification. In arXiv:1102.0183, 2011.
6. M. D. Collins and P. Kohli. Memory-bounded deep convolutional neural networks. In ICASSP, 2013.
7. M. Courbariaux, J.-P. David, and Y. Bengio. Low-precision storage for deep learning. In ICLR, 2015.
8. J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large-scale distributed deep networks. In NIPS, 2012.
9. M. Denil, B. Shakibi, L. Dinh, and N. de Freitas. Predicting parameters in deep learning. In NIPS, 2013.
10. R. M. Gray. Toeplitz and circulant matrices: A review. Foundations and Trends in Communications and Information Theory, 2005.
11. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS workshop, 2014.
12. T. Kailath and J. Chun. Generalized displacement structure for block toeplitz, toeplitz block and toeplitz-derived matrices. SIAM J. Matrix Anal. Appl., 15, 1994.
13. T. Kailath, S. Y. Kung, and M. Morf. Displacement ranks of matrices and linear equations. Journal of Mathematical Analysis and Applications, pages 395-407, 1979.
14. T. Kailath and A. H. Sayed. Displacement structure: Theory and applications. SIAM Review, 37, 1995.
15. H. Larochelle, D. Erhan, A. C. Courville, J. Bergstra, and Y. Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In ICML, 2007.
16. Q. Le, T. Sarlos, and A. Smola. Fastfood-approximating kernel expansions in loglinear time. In ICML, 2013.
17. E. Liberty and S. W. Zucker. The mailman algorithm: a note on matrix vector multiplication. In Information Processing Letters, 2009.
18. V. Pan. Structured Matrices and Polynomials: Unified Superfast Algorithms. Springer, 2001.
19. V. Pan. Inversion of displacement operators. SIAM Journal of Matrix Analysis and Applications, pages 660-677, 2003.
20. A. Rahimi and B. Recht. Random features for large-scale kernel machines. In NIPS, 2007.
21. M. V. Rakhuba and I. V. Oseledets. Fast multidimensional convolution in low-rank tensor formats via cross approximation. SIAM J. Sci. Comput., 37, 2015.
22. T. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, 2013.
23. T. Sainath and C. Parada. Convolutional neural networks for small-footprint keyword spotting. In Proc. Interspeech, 2015.
24. V. Vanhoucke, A. Senior, and M. Z. Mao. Improving the speed of neural networks on cpus. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
25. Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang. Deep fried convnets. In arXiv:1412.7149, 2015.