Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Han, Song ^a   Mao, Huizi ^b   Dally, William J ^a,c  

^a STANFORD UNIVERSITY   (United States)

^b TSINGHUA UNIVERSITY   (China)

^c NVIDIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BANDWIDTH COMPRESSION; CACHE MEMORY; CODES (SYMBOLS); DYNAMIC RANDOM ACCESS STORAGE; EMBEDDED SYSTEMS; ENERGY EFFICIENCY; NETWORK CODING; NEURAL NETWORKS; STATIC RANDOM ACCESS STORAGE;

COMPLEX NEURAL NETWORKS; COMPRESSION METHODS; DRAM MEMORY; HARDWARE RESOURCES; HUFFMAN CODING; LOSS OF ACCURACY; MOBILE APPLICATIONS; STORAGE REQUIREMENTS;

DEEP NEURAL NETWORKS;

EID: 85083950579     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Anwar, Sajid, Hwang, Kyuyeon, and Sung, Wonyong. Fixed point optimization of deep convolutional neural networks for object recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 1131–1135. IEEE, 2015.
2. Arora, Sanjeev, Bhaskara, Aditya, Ge, Rong, and Ma, Tengyu. Provable bounds for learning some deep representations. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, pp. 584–592, 2014.
3. BVLC. Caffe model zoo. URL http://caffe.berkeleyvision.org/model_zoo.
4. Chen, Wenlin, Wilson, James T., Tyree, Stephen, Weinberger, Kilian Q., and Chen, Yixin. Compressing neural networks with the hashing trick. arXiv preprint arXiv:1504.04788, 2015.
5. Collins, Maxwell D and Kohli, Pushmeet. Memory bounded deep convolutional networks. arXiv preprint arXiv:1412.1442, 2014.
6. Denil, Misha, Shakibi, Babak, Dinh, Laurent, de Freitas, Nando, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pp. 2148–2156, 2013.
7. Denton, Emily L, Zaremba, Wojciech, Bruna, Joan, LeCun, Yann, and Fergus, Rob. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pp. 1269–1277, 2014.
8. Girshick, Ross. Fast r-cnn. arXiv preprint arXiv:1504.08083, 2015.
9. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
10. Han, Song, Pool, Jeff, Tran, John, and Dally, William J. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, 2015.
11. Han, Song, Liu, Xingyu, Mao, Huizi, Pu, Jing, Pedram, Ardavan, Horowitz, Mark A, and Dally, William J. EIE: Efficient inference engine on compressed deep neural network. arXiv preprint arXiv:1602.01528, 2016.
12. Hanson, Stephen José and Pratt, Lorien Y. Comparing biases for minimal network construction with back-propagation. In Advances in neural information processing systems, pp. 177–185, 1989.
13. Hassibi, Babak, Stork, David G, et al. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, pp. 164–164, 1993.
14. Hwang, Kyuyeon and Sung, Wonyong. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pp. 1–6. IEEE, 2014.
15. Jia, Yangqing, Shelhamer, Evan, Donahue, Jeff, Karayev, Sergey, Long, Jonathan, Girshick, Ross, Guadarrama, Sergio, and Darrell, Trevor. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
16. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS, pp. 1097–1105, 2012.
17. LeCun, Yann, Denker, John S, Solla, Sara A, Howard, Richard E, and Jackel, Lawrence D. Optimal brain damage. In NIPs, volume 89, 1989.
18. LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
19. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. arXiv:1312.4400, 2013.
20. NVIDIA. Technical brief: NVIDIA jetson TK1 development kit bringing GPU-accelerated computing to embedded systems, a. URL http://www.nvidia.com.
21. NVIDIA. Whitepaper: GPU-based deep learning inference: A performance and power analysis, b. URL http://www.nvidia.com/object/white-papers.html.
22. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
23. Ström, Nikko. Phoneme probability estimation with dynamic sparsely connected artificial neural networks. The Free Speech Journal, 1(5):1–41, 1997.
24. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.
25. Van Leeuwen, Jan. On the construction of huffman trees. In ICALP, pp. 382–410, 1976.
26. Van Nguyen, Hien, Zhou, Kevin, and Vemulapalli, Raviteja. Cross-domain synthesis of medical images using efficient location-sensitive deep network. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, pp. 677–684. Springer, 2015.
27. Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011.
28. Yang, Zichao, Moczulski, Marcin, Denil, Misha, de Freitas, Nando, Smola, Alex, Song, Le, and Wang, Ziyu. Deep fried convnets. arXiv preprint arXiv:1412.7149, 2014.