SmaSH: One-shot model architecture search through hypernetworks

6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings

Volumn , Issue , 2018, Pages

  Brock, Andrew ^a   Lim, Theodore ^a   Ritchie, J M ^a   Weston, Nick ^b  

^a HERIOT WATT UNIVERSITY   (United Kingdom)

^b RENISHAW PLC   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP NEURAL NETWORKS;

ARCHITECTURE SELECTION; COMPETITIVE PERFORMANCE; COMPUTATION TIME; EXPERT KNOWLEDGE; FLEXIBLE MECHANISMS; MODEL ARCHITECTURE; NETWORK CONNECTIVITY;

NETWORK ARCHITECTURE;

EID: 85083953913     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (45)

1. D. Arpit, Y. Zhou, B.U. Kota, and V. Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift in deep networks. In ICML, 2016.
2. J.L. Ba, J.R. Kiros, and G.E. Hinton. Layer normalization. In NIPS, 2016.
3. B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. In ICLR, 2017.
4. J. Bergstra and Y. Bengio. Random search for hyperparameter optimization. In JMLR, 2012.
5. A. Brock, T. Lim, J.M. Ritchie, and N. Weston. Generative and discriminative voxel modeling with convolutional neural networks. 3D Deep Learning Workshop at NIPS 2016, 2016.
6. T. Chen, I. Goodfellow, and J. Shiens. Net2net: Accelerating learning via knowledge transfer. In ICLR, 2016.
7. P. Chrabaszcz, I. Loshchilov, and F. Hutter. A downsampled variant of imagennet as an altetrnative to the cifar datasets. arXiv Preprint arXiv: 1707.08819, 2017.
8. A. Coates, H. Lee, and A.Y. Ng. n analysis of single layer networks in unsupervised feature learning. In AISTATS, 2011.
9. M. Denil, B. Shakibi, L. Dinh, M. A. Ranzato, and N. de Freitas. Predicting parameters in deep learning. In NIPS, 2013.
10. D. Floreano, P. Durr, and C. Mattiussi. Neuroevolution: from architectures to learning. In Evolutionary Intelligence, 2008.
11. X. Gastaldi. Shake-shake regularization of 3-branch residual networks. ICLR 2017 Workshop, 2017.
12. D. Ha, A. Dai, and Q. Le. Hypernetworks. In ICLR, 2017.
13. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
14. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
15. G. Huang, Z. Liu, K.Q. Weinberger, and L. van der Maaten. Densely connected convolutional networks. In CVPR, 2017.
16. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
17. X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool. Dynamic filter networks. In NIPS, 2016.
18. D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv Preprint arXiv: 1412.6980, 2014.
19. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images, 2009.
20. G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017.
21. L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In ICLR, 2017.
22. I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
23. J.F. Miller and P. Thomson. Cartesian genetic programming. In EuroGP, 2000.
24. A. Paszke, S. Gross, and S. Chintala. Pytorch. github.com/pytorch/pytorch, 2017.
25. A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
26. E. Real, S. Moore, A. Selle, S. Saxena, Y.L. Suematsu, Q. Le, and A. Kurakin. Large-scale evolution of image classifiers. arXiv Preprint arXiv: 1703.01041, 2017.
27. S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. arXiv Preprint arXiv: 1705.08045, 2017.
28. T. Salimans and D.P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NIPS, 2016.
29. A.M. Saxe, P.W. Koh, Z. Chen, M. Bhand, B. Suresh, and A.Y. Ng. On random weights and unsupervised feature learning. In ICML, 2011.
30. S. Saxena and J. Verbeek. Convolutional neural fabrics. In NIPS, 2016.
31. J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. In Neural Computation, volume 4, pages 131–139, 1992.
32. S. Singh, D. Hoiem, and D. Forsyth. Swapout: Learning an ensemble of deep architectures. In NIPS, 2016.
33. J. Snoek, H. Larochelle, and R.P. Adams. Practical bayesian optimization of machinelearning algorithms. In NIPS, 2012.
34. J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M.M.A. Patwary, Prabhat, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In ICML, 2015.
35. N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv Preprint arXiv: 1207.0580, 2012.
36. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In NIPS, 2015.
37. K.O. Stanley, D.B. D’Ambrosio, and J Gauci. A hypercube-based encoding for evolving large-scale neural networks. In Artificial Life, 15(2):185-212, 2009.
38. M. Suganuma, S. Shirakawa, and T. Nagao. A genetic programming approach to designing convolutional neural network architectures. In GECCO, 2017.
39. C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv Preprint arXiv: 1602.07261, 2016.
40. D. Wierstra, F.J. Gomez, and J. Schmidhuber. Modeling systems with internal state using evolino. In GECCO, 2005.
41. Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
42. J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
43. S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv Preprint arXiv: 1605.07146, 2016.
44. B. Zoph and Q. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
45. B. Zoph, V. Vasudevan, J. Shlens, and Q. Le. Learning transferable architectures for scalable image recognition. arXiv Preprint arXiv: 1707.07012, 2017.