MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn , Issue , 2018, Pages 1586-1595

  Gordon, Ariel ^a   Eban, Elad ^a   Nachum, Ofir ^a   Chen, Bo ^a   Wu, Hao ^a   Yang, Tien Ju ^a,b   Choi, Edward ^a,c  

^a GOOGLE INC   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIGITAL ARITHMETIC; ITERATIVE METHODS; NETWORK ARCHITECTURE;

CONSTRAINED STRUCTURE; FLOATING POINT OPERATIONS; LARGE NETWORKS; MULTIPLICATIVE FACTORS; NEURAL NETWORK STRUCTURES; NOVEL STRUCTURES; REGULARIZER; RESOURCE CONSTRAINT;

COMPUTER VISION;

EID: 85058146752     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2018.00171     Document Type: Conference Paper

References (35)

1. J. M. Alvarez and M. Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2270-2278, 2016.
2. T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015.
3. M. D. Collins and P. Kohli. Memory bounded deep convolutional networks. arXiv preprint arXiv:1412.1442, 2014.
4. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.
5. J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield. Large automatic learning, rule extraction, and generalization. Complex systems, 1(5):877-922, 1987.
6. J. Feng and T. Darrell. Learning the structure of deep convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2749-2757, 2015.
7. J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, LA, 2017.
8. S. Han, J. Pool, J. Tran, andW. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135-1143, 2015.
9. B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pages 164-171, 1993.
10. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11. S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al. Cnn architectures for large-scale audio classification. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 131-135. IEEE, 2017.
12. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
13. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko,W.Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
14. Z. Huang and N. Wang. Data-driven sparse structure selection for deep neural networks. arXiv preprint arXiv:1707.01213, 2017.
15. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448-456, 2015.
16. Y.-H. Kim, B. Reddy, S. Yun, and C. Seo. Nemo: Neuroevolution with multiobjective optimization of deep neural network for speed and accuracy. 1, 3
17. Y. Le Cun, J. S. Denker, and S. A. Solla. Optimal brain damage. 2
18. V. Lebedev and V. Lempitsky. Fast convnets using groupwise brain damage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2554-2564, 2016.
19. B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 806-814, 2015.
20. Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. arXiv preprint arXiv:1708.06519, 2017.
21. P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient transfer learning. arXiv preprint arXiv:1611.06440, 2016.
22. K. Murray and D. Chiang. Auto-sizing neural networks: With applications to n-gram language models. arXiv preprint arXiv:1508.05051, 2015.
23. A. Y. Ng. Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, page 78. ACM, 2004.
24. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, Q. Le, and A. Kurakin. Large-scale evolution of image classifiers. arXiv preprint arXiv:1703.01041, 2017.
25. C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968, 1, 2017.
26. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818-2826, 2016.
27. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.
28. T. Veniat and L. Denoyer. Learning time-efficient deep architectures with budgeted super networks. arXiv preprint arXiv:1706.00046, 2017.
29. W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, pages 2074-2082, 2016.
30. P. M. Williams. Bayesian regularization and pruning using a laplace prior. Neural computation, 7(1):117-143, 1995.
31. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-67, 2006.
32. H. Zhou, J. M. Alvarez, and F. Porikli. Less is more: Towards compact cnns. In European Conference on Computer Vision, pages 662-677. Springer, 2016.
33. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.
34. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.
35. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012, 2017.