Learning Transferable Architectures for Scalable Image Recognition

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn , Issue , 2018, Pages 8697-8710

  Zoph, Barret ^a   Vasudevan, Vijay ^a   Shlens, Jonathon ^a   Le, Quoc V ^a  

^a GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CELLS; COMPUTER VISION; CONVOLUTION; CYTOLOGY; IMAGE RECOGNITION; LARGE DATASET; NETWORK ARCHITECTURE; OBJECT DETECTION;

ARCHITECTURAL BUILDINGS; ARCHITECTURE ENGINEERING; CLASSIFICATION MODELS; COMPUTATIONAL COSTS; COMPUTATIONAL DEMANDS; COMPUTER VISION PROBLEMS; MODEL ARCHITECTURE; REGULARIZATION TECHNIQUE;

IMAGE CLASSIFICATION;

EID: 85062864819     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2018.00907     Document Type: Conference Paper

References (71)

1. M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3981-3989, 2016. 2
2. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. ArXiv preprint arXiv: 1607. 06450, 2016. 12
3. B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. In In-ternational Conference on Learning Representations, 2016. 2
4. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Neural Information Processing Systems, 2011. 2
5. J. Bergstra and Y. Bengio. Random search for hyperparameter optimization. Journal of Machine Learning Research, 2012. 2, 8
6. J. Bergstra, D. Yamins, and D. D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. International Conference on Machine Learning, 2013. 2
7. J. Chen, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous sgd. In International Conference on Learning Representations Workshop Track, 2016. 12
8. Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng. Dual path networks. ArXiv preprint arXiv: 1707. 01083, 2017. 5, 7
9. F. Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 2, 7
10. D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In International Conference on Learning Representations, 2016. 11
11. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009. 1, 12
12. T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv preprint arXiv: 1708. 04552, 2017. 5, 6
13. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning, volume 32, pages 647-655, 2014. 6
14. Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. ArXiv preprint arXiv: 1611. 02779, 2016. 2
15. C. Finn, P. Abbeel, and S. Levine. Model-agnostic metalearning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017. 2
16. D. Floreano, P. Dürr, and C. Mattiussi. Neuroevolution: From architectures to learning. Evolutionary Intelligence, 2008. 2
17. K. Fukushima. A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, page 93202, 1980. 1
18. X. Gastaldi. Shake-shake regularization of 3-branch residual networks. In International Conference on Learning Representations Workshop Track, 2017. 6, 12
19. D. Ha, A. Dai, and Q. V. Le. Hypernetworks. In International Conference on Learning Representations, 2017. 2
20. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. 1, 2, 3, 4
21. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016. 11
22. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 1997. 2, 11
23. S. Hochreiter, A. Younger, and P. Conwell. Learning to learn using gradient descent. Artificial Neural Networks, pages 87-94, 2001. 2
24. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv preprint arXiv: 1704. 04861, 2017. 2, 5, 7, 8, 11
25. J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. ArXiv preprint arXiv: 1709. 01507, 2017. 5, 7
26. G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. 6
27. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. 11
28. J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. 6, 7, 8, 14
29. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations, 2015. 2, 5, 7, 8
30. R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In International Conference on Learning Representations, 2015. 2
31. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 4, 11
32. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing System, 2012. 1, 3
33. G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. ArXiv preprint arXiv: 1605. 07648, 2016. 4, 11
34. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 1998. 1
35. K. Li and J. Malik. Learning to optimize neural nets. ArXiv preprint arXiv: 1703. 00441, 2017. 2
36. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 7
37. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. ArXiv preprint arXiv: 1708. 02002, 2017. 7, 8
38. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Com-mon objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014. 7
39. I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. 12
40. H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, and F. Hutter. Towards automatically-tuned neural networks. In Proceedings of the 2016 Workshop on Automatic Machine Learning, pages 58-65, 2016. 2
41. T. Miconi. Neural networks with differentiable structure. ArXiv preprint arXiv: 1606. 06216, 2016. 2
42. R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, A. Navruzyan, N. Duffy, and B. Hodjat. Evolving deep neural networks. ArXiv preprint arXiv: 1703. 00548, 2017. 2
43. R. Negrinho and G. Gordon. DeepArchitect: Automatically designing and training deep architectures. ArXiv preprint arXiv: 1704. 08792, 2017. 2
44. N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox. A highthroughput screening approach to discovering good forms of biologically inspired visual representation. PLoS Computational Biology, 5 (11): E1000579, 2009. 2
45. S. Ravi and H. Larochelle. Optimization as a model for fewshot learning. In International Conference on Learning Representations, 2017. 2
46. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, Q. Le, and A. Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, 2017. 2
47. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91-99, 2015. 2, 6, 7
48. S. Saxena and J. Verbeek. Convolutional neural fabrics. In Advances in Neural Information Processing Systems, 2016. 2
49. T. Schaul and J. Schmidhuber. Metalearning. Scholarpedia, 2010. 2
50. F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815-823, 2015. 8
51. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. ArXiv preprint arXiv: 1707. 06347, 2017. 4, 11
52. A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Top-down modulation for object detection. ArXiv preprint arXiv: 1612. 06851, 2016. 7, 8
53. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. 1, 2, 3, 4
54. J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Neural Information Processing Systems, 2012. 2
55. J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Ali, R. P. Adams, et al. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning, 2015. 2
56. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 (1): 1929-1958, 2014. 11
57. K. O. Stanley, D. B. D'Ambrosio, and J. Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial Life, 2009. 2
58. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi. Inceptionv4, Inception-Resnet and the impact of residual connections on learning. In International Conference on Learning Representations Workshop Track, 2016. 1, 2, 3, 4, 7
59. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015. 1, 2, 3, 4, 7
60. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. 1, 2, 3, 4, 5, 7, 8, 12
61. D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. ArXiv preprint arXiv: 1607. 08022, 2016. 12
62. J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. ArXiv preprint arXiv: 1611. 05763, 2016. 2
63. T. Weyand, I. Kostrikov, and J. Philbin. Planet-photo geolocation with convolutional neural networks. In European Conference on Computer Vision, 2016. 8
64. O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. de Freitas, and J. Sohl-Dickstein. Learned optimizers that scale and generalize. ArXiv preprint arXiv: 1703. 04813, 2017. 2
65. D. Wierstra, F. J. Gomez, and J. Schmidhuber. Modeling systems with internal state using evolino. In The Genetic and Evolutionary Computation Conference, 2005. 2
66. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, 1992. 11
67. L. Xie and A. Yuille. Genetic CNN. ArXiv preprint arXiv: 1703. 01513, 2017. 2
68. S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 1, 2, 7
69. X. Zhang, Z. Li, C. C. Loy, and D. Lin. Polynet: A pursuit of structural diversity in very deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 5, 7, 8, 11
70. X. Zhang, X. Zhou, L. Mengxiao, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. ArXiv preprint arXiv: 1707. 01083, 2017. 2, 5, 7, 8
71. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017. 1, 2, 4, 6, 11