Efficient neural architecture search via proximal iterations

AAAI 2020 - 34th AAAI Conference on Artificial Intelligence

Volumn , Issue , 2020, Pages 6664-6671

  Yao, Quanming ^a   Xu, Ju ^b   Tu, Wei Wei ^a   Zhu, Zhanxing ^b,c  

^a 4PARADIGM INC   (China)

^b PEKING UNIVERSITY   (China)

^c Beijing Institute of Big Data Research   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATION COSTS; DISCRETE CONSTRAINTS; HIGH PERFORMANCE ARCHITECTURES; MODELING COMPLEXITY; NEURAL ARCHITECTURES; OPTIMIZATION PROBLEMS; PERFORMANCE; SEARCH METHOD; SEARCH PROCESS; STATE OF THE ART;

ARTIFICIAL INTELLIGENCE;

EID: 85106416134     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1609/aaai.v34i04.6143     Document Type: Conference Paper

References (37)

1. Akimoto, Y.; Shirakawa, S.; Yoshinari, N.; Uchida, K.; Saito, S.; and Nishida, K. 2019. Adaptive stochastic natural gradient method for one-shot neural architecture search. In ICML, 171–180.
2. Alistarh, D.; Grubic, D.; Li, J.; Tomioka, R.; and Vojnovic, M. 2017. QSGD: Communication-efficient sgd via gradient quantization and encoding. In NeurIPS, 1709–1720.
3. Amari, S. 1998. Natural gradient works efficiently in learning. Neural Computation 10(2):251–276.
4. Bai, Y.; Wang, Y.-X.; and Liberty, E. 2018. Proxquant: Quantized neural networks via proximal operators. In ICLR.
5. Baker, B.; Gupta, O.; Naik, N.; and Raskar, R. 2017. Designing neural network architectures using reinforcement learning. In ICLR.
6. Cai, H.; Zhu, L.; and Han, S. 2019. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR.
7. Courbariaux, M.; Bengio, Y.; and David, J.-P. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS, 3123–3131.
8. Dong, X., and Yang, Y. 2019. Searching for a robust neural architecture in four GPU hours. In CVPR, 1761–1770.
9. Guo, Z.; Zhang, X.; Mu, H.; Heng, W.; Liu, Z.; Wei, Y.; and Sun, J. 2019. Single path one-shot neural architecture search with uniform sampling. Technical report, Arvix.
10. Hou, L.; Yao, Q.; and Kwok, J. 2017. Loss-aware binarization of deep networks. In ICLR.
11. Howard, A.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CVPR.
12. Huang, G.; Liu, Z.; Van Der Maaten, L.; and Weinberger, K. 2017. Densely connected convolutional networks. In CVPR, 4700–4708.
13. Hutter, F.; Kotthoff, L.; and Vanschoren, J., eds. 2018. Automated Machine Learning: Methods, Systems, Challenges. Springer.
14. Jang, E.; Gu, S.; and Poole, B. 2016. Categorical reparameterization with gumbel-softmax. In ICLR.
15. Krizhevsky, A. 2009. Learning multiple layers of features from tiny images. Technical report, Citeseer.
16. Lee, D., and Seung, S. 1999. Learning the parts of objects by non-negative matrix factorization. Nature 401:788–791.
17. Liu, C.; Zoph, B.; Shlens, J.; Hua, W.; Li, L.; Li, F.-F.; Yuille, A.; Huang, J.; and Murphy, K. 2018. Progressive neural architecture search. In ECCV.
18. Liu, H.; Simonyan, K.; and Yang, Y. 2019. DARTS: Differentiable architecture search. In ICLR.
19. Luo, R.; Tian, F.; Qin, T.; Chen, E.; and Liu, T.-Y. 2018. Neural architecture optimization. In NeurIPS.
20. Ma, N.; Zhang, X.; Zheng, H.; and Sun, J. 2018. ShuffleNet V2: Practical guidelines for efficient CNN architecture design. ECCV 122–138.
21. Noy, A.; Nayman, N.; Ridnik, T.; Zamir, N.; Doveh, S.; Friedman, I.; Giryes, R.; and Zelnik-Manor, L. 2019. ASAP: Architecture search, anneal and prune. Technical report, arXiv preprint arXiv:1904.04123.
22. Parikh, N., and Boyd, S. 2013. Proximal algorithms. Foundations and Trends in Optimization 1(3):123–231.
23. Pham, H.; Guan, M.; Zoph, B.; Le, Q.; and Dean, J. 2018. Efficient neural architecture search via parameter sharing. Technical report, arXiv preprint.
24. Real, E.; Aggarwal, A.; Huang, T.; and Le, Q. 2018. Regularized evolution for image classifier architecture search. arXiv.
25. Sutton, R., and Barto, A. 1998. Reinforcement learning: An introduction. MIT press.
26. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S. E.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015. Going deeper with convolutions. CVPR 1–9.
27. Tan, M.; Chen, B.; Pang, R.; Vasudevan, V.; and Le, Q. 2018. Mnasnet: Platform-aware neural architecture search for mobile. Technical report, arXiv.
28. Xiao, L. 2010. Dual averaging methods for regularized stochastic learning and online optimization. JMLR 11(Oct):2543–2596.
29. Xie, L., and Yuille, A. 2017. Genetic CNN. In ICCV.
30. Xie, S.; Zheng, H.; Liu, C.; and Lin, L. 2019. SNAS: stochastic neural architecture search. In ICLR.
31. Yang, Z.; Dai, Z.; Salakhutdinov, R.; and Cohen, W. 2018. Breaking the softmax bottleneck: A high-rank rnn language model. In ICLR.
32. Yao, Q.; Kwok, J.; Gao, F.; Chen, W.; and Liu, T.-Y. 2017. Efficient inexact proximal gradient algorithm for nonconvex problems. In IJCAI, 3308–3314. AAAI Press.
33. Yao, Q.; Wang, M.; Chen, Y.; Dai, W.; Hu, Y.; Li, Y.; Tu, W.W.; Yang, Q.; and Yu, Y. 2018. Taking human out of learning applications: A survey on automated machine learning. Technical report, arXiv preprint.
34. Zhong, Z.; Yan, J.; Wu, W.; Shao, J.; and Liu, C.-L. 2018. Practical block-wise neural network architecture generation. In CVPR.
35. Zhou, H.; Yang, M.; Wang, J.; and Pan, W. 2019. BayesNAS: A bayesian approach for neural architecture search. In ICML, 7603–7613.
36. Zoph, B., and Le, Q. 2017. Neural architecture search with reinforcement learning. In ICLR.
37. Zoph, B.; Vasudevan, V.; Shlens, J.; and Le, Q. 2017. Learning transferable architectures for scalable image recognition. In CVPR.