Faster discovery of neural architectures by searching for paths in a large model

6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings

Volumn , Issue , 2018, Pages

  Pham, Hieu ^a,b   Guan, Melody Y ^c   Zoph, Barret ^a   Le, Quoc V ^a   Dean, Jeff ^a  

^a GOOGLE INC   (United States)

^b Carnegie Mellon University ^*  (United States)

^c Stanford University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FORESTRY; NEURAL NETWORKS; STATISTICAL TESTS;

AUTOMATED MODELING; AUTOMATIC MODELING; CROSS ENTROPY; NEURAL ARCHITECTURES; NOVEL ARCHITECTURE; OPTIMAL PATHS; POLICY GRADIENT; STATE OF THE ART;

NETWORK ARCHITECTURE;

EID: 85083952791     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (44)

1. Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In ICLR, 2017a.
2. Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. Arxiv, 1705.10823, 2017b.
3. Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In ICLR Workshop, 2017a.
4. Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with reinforcement learning. In ICML, 2017b.
5. Léon Bottou. Une Approche théorique de l’Apprentissage Connexioniste: Applications à la reconnaissance de la Parole. PhD thesis, 1991.
6. Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. SMASH: one-shot model architecture search through hypernetworks. Arxiv, 1708.05344, 2017.
7. Han Cai, Tianyao Chen, Weinan Zhang, Yong. Yu, and Jun Wang. Reinforcement learning for architecture search by network transformation. Arxiv, 1707.04873, 2017.
8. Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training. Arxiv, 1705.10823, 2017.
9. Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. Arxiv, 1708.04552, 2017.
10. Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. Arxiv, 1711.04528, 2017.
11. Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv, 1701.08734, 2017.
12. Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In NIPS, 2016.
13. Xavier Gastaldi. Shake-shake regularization of 3-branch residual networks. In ICLR Workshop Track, 2016.
14. David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks. In ICLR, 2017.
15. Kaiming He, Xiangyu Zhang, Shaoqing Rein, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR, 2015.
16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CPVR, 2016.
17. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. In Neural Computations, 1997.
18. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2016.
19. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: a loss framework for language modeling. In ICLR, 2017.
20. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
21. Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
22. Emmanuel Krause, Ben Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. Arxiv, 1709.07432, 2017.
23. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
24. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017.
25. Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. Arxiv, 1312.4400, 2013.
26. Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Arxiv, 1712.03351, 2017a.
27. Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. Arxiv, 1711.00436, 2017b.
28. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
29. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, 1994.
30. Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. Arxiv, 1707.05589, 2017.
31. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. Arxiv, 1708.02182, 2017.
32. Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures. In CPVR, 2017.
33. 2). Soviet Mathematics Doklady, 1983.
34. Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. Large-scale evolution of image classifiers. In ICML, 2017.
35. Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In NIPS, 2016.
36. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CPVR, 2016.
37. Tom Veniat and Ludovic Denoyer. Learning time-efficient deep architectures with budgeted super networks. Arxiv, 1706.00046, 2017.
38. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992.
39. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991.
40. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. Arxiv, 1409.2329, 2014.
41. Zhao Zhong, Junjie Yan, and Cheng-Lin Liu. Practical network blocks design with q-learning. Arxiv, 1708.05552, 2017.
42. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In ICML, 2017.
43. Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
44. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. Arxiv, 1707.07012, 2017.