A genetic programming approach to designing convolutional neural network architectures

GECCO 2017 - Proceedings of the 2017 Genetic and Evolutionary Computation Conference

Volumn , Issue , 2017, Pages 497-504

  Suganuma, Masanori ^a   Shirakawa, Shinichi ^a   Nagao, Tomoharu ^a  

^a YOKOHAMA NATIONAL UNIVERSITY   (Japan)

Author keywords

Convolutional neural network; Deep learning; Designing neural network architectures; Genetic programming

Indexed keywords

CLASSIFICATION (OF INFORMATION); CONVOLUTION; DEEP LEARNING; DEEP NEURAL NETWORKS; EDUCATION; GENETIC ALGORITHMS; GENETIC PROGRAMMING; IMAGE CLASSIFICATION; NEURAL NETWORKS;

CARTESIAN GENETIC PROGRAMMING; CLASSIFICATION TASKS; CONVOLUTIONAL NEURAL NETWORK; ENCODING METHODS; EXPERT KNOWLEDGE; FUNCTIONAL MODULES; LEARNING MODELS; STATE OF THE ART;

NETWORK ARCHITECTURE;

EID: 85026354551     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3071178.3071229     Document Type: Conference Paper

References (39)

1. Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2016. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167v3 (2016).
2. Yoshua Bengio. 2000. Gradient-based optimization of hyperparameters. Neural computation 12, 8 (Aug. 2000), 1889-1900.
3. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 1 (Jan. 2012), 281-305.
4. James Bergstra, Daniel Yamins, and David D. Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning (ICML '13). Atlanta, Gerorgia, 115-123.
5. James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balâzs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24 (NIPS '11). Granada, Spain, 2546-2554.
6. Chrisantha Fernando, Dylan Banarse, Malcolm Reynolds, Frederic Besse, David Pfau, Max Jaderberg, Marc Lanctot, and Daan Wierstra. 2016. Convolution by evolution: Differentiable pattern producing networks. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO '16). Denver, Colorado, 109-116.
7. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 13). Atlanta, Gerorgia, 1319-1327.
8. Simon Harding. 2008. Evolution of image filters on graphics processor units using cartesian genetic programming. In Proceedings ofthe IEEE Congress on Evolutionary Computation (CEC '08). Hong Kong, China, 1921-1928.
9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV '15). Santiago, Chile, 1026-1034.
10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR '16). Las Vegas, Nevada, 770-778.
11. Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and others. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29, 6 (Nov. 2012), 82-97.
12. Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. 2016. Densely connected convolutional networks. arXivpreprint arXiv:1608.06993 (2016).
13. Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential model-based optimization for general algorithm configuration. In Proceedings of the 5th International Conference on Learning and Intelligent Optimization. Rome, Italy, 507-523.
14. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML '15). Lille, France, 448-456.
15. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representaions (ICLR '15). San Diego, 2452-2459.
16. Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Technical report (2009).
17. Alex Krizhevsky, IlyaSutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS '12). Nevada, 1097-1105.
18. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278-2324.
19. Min Lin, Qiang Chen, and Shuicheng Yan. 2014. Network in network. In Proceedings of the 2nd International Conference on Learning Representaions (ICLR '14). Banff, Canada.
20. Ilya Loshchilov and Frank Hutter. 2016. CMA-ES for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.07269 (2016).
21. Julian F. Miller and Stephen L. Smith. 2006. Redundancy and computational efficiency in cartesian genetic programming. IEEE Transactions on Evolutionary Computation 10, 2 (April 2006), 167-174.
22. Julian F. Miller and Peter Thomson. 2000. Cartesian genetic programming. In Proceedings of the European Conference on Genetic Programming (EuroGP '00). Scotland, UK, 121-132.
23. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. In Proceedings of Neural Information Processing Systems (NIPS '13) Workshop on Deep Learning. Nevada.
24. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, and others. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (Feb. 2015), 529-533.
25. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML '10). Haifa, Israel, 807-814.
26. J. David Schaffer, Darrell Whitley, and Larry J. Eshelman. 1992. Combinations of genetic algorithms and neural networks: A survey of the state of the art. In Proceedings of International Workshop on Combinations of Genetic Algorithms and Neural Networks (COGANN '92). Baltimore, Maryland, 1-37.
27. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
28. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25 (NIPS '12). Nevada, 2951-2959.
29. Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md Mostofa Ali Patwary, Mr Prabhat, and Ryan P. Adams. 2015. Scalable Bayesian optimization using deep neural networks. In Proceedings ofthe 32nd International Conference on Machine Learning(ICML 15). Lille, France, 2171-2180.
30. Kenneth O. Stanley. 2007. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines 8, 2 (June 2007), 131-162.
31. Kenneth O. Stanley, David B. D'Ambrosio, and Jason Gauci. 2009. A hypercube-based encoding for evolving large-scale neural networks. Artificial life 15, 2 (2009), 185-212.
32. Kenneth O. Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99-127.
33. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR '15). Boston, Massachusetts, 1-9.
34. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: a next-generation open source framework for deep learning. In Proceedings of Neural Information Processing Systems (NIPS '15) Workshop on Machine Learning Systems (LearningSys). Montreal, Canada.
35. Phillip Verbancsics and Josh Harguess. 2013. Generative neuroevolution for deep learning. arXiv preprint arXiv:1312.5355 ( 2013).
36. Phillip Verbancsics and Josh Harguess. 2015. Image classification using generative neuro evolution for deep learning. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV 15). Hawaii, 488-493.
37. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 15). Boston, Massachusetts, 3156-3164.
38. Richard Zhang, Phillip Isola, and Alexei A. Efros. 2016. Colorful image coloriza-tion. In Proceedings of the 14th European Conference on Computer Vision (ECCV 16). Amsterdam, The Netherlands, 649-666.
39. Barret Zoph and Quoc V. Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578v2 (2016).