Compression of deep convolutional neural networks for fast and low power mobile applications

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Kim, Yong Deok ^a   Park, Eunhyeok ^b   Yoo, Sungjoo ^b   Choi, Taelim ^a   Yang, Lu ^a   Shin, Dongjun ^a  

^a SAMSUNG Electronics   (South Korea)

^b Seoul National University   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

CONVOLUTION; ENERGY UTILIZATION; FACTORIZATION; LOW POWER ELECTRONICS; NEURAL NETWORKS; SMARTPHONES;

COMPLEX TASK; CONVOLUTIONAL NEURAL NETWORK; KEY OPERATIONS; LOSS OF ACCURACY; MOBILE APPLICATIONS; NETWORK COMPRESSION; TUCKER DECOMPOSITIONS; VARIATIONAL BAYESIAN;

DEEP NEURAL NETWORKS;

EID: 85083951289     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (38)

1. Bader, Brett W., Kolda, Tamara G., et al. Matlab tensor toolbox version 2.6. Available online, February 2015. URL http://www.sandia.gov/tgkolda/TensorToolbox/.
2. Carroll, J Douglas and Chang, Jih-Jie. Analysis of individual differences in multidimensional scaling via an n-way generalization of eckart-young decomposition. Psychometrika, 35(3):283–319, 1970.
3. Chellapilla, Kumar, Puri, Sidd, and Simard, Patrice. High performance convolutional neural networks for document processing. In Tenth International Workshop on Frontiers in Handwriting Recognition. Suvisoft, 2006.
4. Chen, Wenlin, Wilson, James T, Tyree, Stephen, Weinberger, Kilian Q, and Chen, Yixin. Compressing neural networks with the hashing trick. arXiv preprint arXiv:1504.04788, 2015.
5. Cheng, Yu, Yu, Felix X, Feris, Rogerio S, Kumar, Sanjiv, Choudhary, Alok, and Chang, Shih-Fu. Fast neural networks with circulant projections. arXiv preprint arXiv:1502.03436, 2015.
6. De Lathauwer, Lieven, De Moor, Bart, and Vandewalle, Joos. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4):1253–1278, 2000.
7. De Silva, Vin and Lim, Lek-Heng. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications, 30(3):1084–1127, 2008.
8. Denil, Misha, Shakibi, Babak, Dinh, Laurent, de Freitas, Nando, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pp. 2148–2156, 2013.
9. Denton, Emily L, Zaremba, Wojciech, Bruna, Joan, LeCun, Yann, and Fergus, Rob. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pp. 1269–1277, 2014.
10. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics, pp. 249–256, 2010.
11. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
12. Han, Song, Mao, Huizi, and Dally, William J. A deep neural network compression pipeline: Pruning, quantization, huffman encoding. arXiv preprint arXiv:1510.00149, 2015a.
13. Han, Song, Pool, Jeff, Tran, John, and Dally, William J. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.02626, 2015b.
14. Harshman, Richard A and Lundy, Margaret E. Parafac: Parallel factor analysis. Computational Statistics & Data Analysis, 18(1):39–72, 1994.
15. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision, 2015.
16. Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Rus-lan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
17. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
18. Jaderberg, M., Vedaldi, A., and Zisserman, A. Speeding up convolutional neural networks with low rank expansions. In British Machine Vision Conference, 2014.
19. Kim, Y.-D. and Choi, S. Nonnegative Tucker decomposition. In Proceedings of the IEEE CVPR-2007 Workshop on Component Analysis Methods, Minneapolis, Minnesota, 2007.
20. Kolda, Tamara G and Bader, Brett W. Tensor decompositions and applications. SIAM review, 51(3): 455–500, 2009.
21. Lebedev, Vadim, Ganin, Yaroslav, Rakhuba, Maksim, Oseledets, Ivan, and Lempitsky, Victor. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In International Conference on Learning Representations, 2015.
22. Lin, M., Chen, Q., and Yan, S. Network in network. In International Conference on Learning Representations, 2014.
23. MacKay, David JC. Bayesian interpolation. Neural computation, 4(3):415–447, 1992.
24. Mathieu, Michael, Henaff, Mikael, and LeCun, Yann. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851, 2013.
25. Mørup, Morten and Hansen, Lars Kai. Automatic relevance determination for multi-way models. Journal of Chemometrics, 23(7-8):352–363, 2009.
26. Nakajima, Shinichi. Variational Bayesian matrix factorization version 1.02, 2015. URL https://sites.google.com/site/shinnkj23/downloads.
27. Nakajima, Shinichi, Tomioka, Ryota, Sugiyama, Masashi, and Babacan, S Derin. Perfect dimensionality recovery by variational bayesian pca. In Advances in Neural Information Processing Systems, pp. 971–979, 2012.
28. Nakajima, Shinichi, Sugiyama, Masashi, Babacan, S Derin, and Tomioka, Ryota. Global analytic solution of fully-observed variational bayesian matrix factorization. The Journal of Machine Learning Research, 14(1):1–37, 2013.
29. Novikov, Alexander, Podoprikhin, Dmitry, Osokin, Anton, and Vetrov, Dmitry. Tensorizing neural networks. arXiv preprint arXiv:1509.06569, 2015.
30. Shashua, Amnon and Hazan, Tamir. Non-negative tensor factorization with applications to statistics and computer vision. In Proceedings of the 22nd international conference on Machine learning, pp. 792–799. ACM, 2005.
31. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
32. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
33. Tipping, Michael E. Sparse bayesian learning and the relevance vector machine. The journal of machine learning research, 1:211–244, 2001.
34. Tucker, Ledyard R. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279–311, 1966.
35. Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, volume 1, 2011.
36. Ye, Jieping. Generalized low rank approximations of matrices. Machine Learning, 61(1-3):167–191, 2005.
37. Zhang, Xiangyu, Zou, Jianhua, He, Kaiming, and Sun, Jian. Accelerating very deep convolutional networks for classification and detection. arXiv preprint arXiv:1505.06798, 2015a.
38. Zhang, Xiangyu, Zou, Jianhua, Ming, Xiang, He, Kaiming, and Sun, Jian. Efficient and accurate approximations of nonlinear convolutional networks. 2015b.