Learned optimizers that scale and generalize

34th International Conference on Machine Learning, ICML 2017

Volumn 8, Issue , 2017, Pages 5744-5753

  Wichrowska, Olga ^a   Maheswaranathan, Niru ^a,b   Hoffman, Matthew W ^c   Colmenarejo, Sergio Gómez ^c   Denii, Misha ^c   De Freitas, Nando ^c   Sohl Dickstein, Jascha ^a  

^a GOOGLE INC   (United States)

^b STANFORD UNIVERSITY   (United States)

^c DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; NEURAL NETWORKS; STRUCTURAL OPTIMIZATION;

ARCHITECTURAL FEATURES; COMMON PROPERTY; COMPUTATION OVERHEADS; CONVOLUTIONAL NEURAL NETWORK; GRADIENT DESCENT; LEARNING TO LEARN; OPTIMIZATION PROBLEMS; OPTIMIZATION TASK;

NETWORK ARCHITECTURE;

EID: 85048532379     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Hoffman, Matthew W, Pfau, David, Schaul, Tom, Shillingford, Brendan, and de Freitas, Nando. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
2. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate, iclr, 2015.
3. Bengio, S., Bengio, Y., and Cloutier, J. On the search for new learning rules for ANNs. Neural Processing Letters, 2(4): 26-30, 1995.
4. Bengio, Yoshua, Bengio, Samy, and Cloutier, Jocelyn. Learning a synaptic learning rule. Université de Montréal, Département d'informatique et de recherche opérationnelle, 1990.
5. Bengio, Yoshua, Bengio, Samy, Cloutier, Jocelyn, and Gecsei, Jan. On the optimization of a synaptic learning rule. In in Conference on Optimality in Biological and Artificial Networks, 1992.
6. Chen, Yutian, Hoffman, Matthew W., Colmenarejo, Sergio Gomez, Denil, Misha, Lillicrap, Timothy P., and de Freitas, Nando. Learning to learn for global optimization of black box functions. arXiv Report 1611.03824, 2016.
7. Cho, Kyunghyun, Van Merriënboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409.1259, 2014.
8. 2: Fast reinforcement learning via slow reinforcement learning. Technical report, UC Berkeley and OpenAI, 2016.
9. Funk, Simon. RMSprop loses to SMORMS3 - beware the epsilon!, 2015. URL sifter.org/~simon/journal/20150420.html.
10. Harlow, Harry F. The formation of learning sets. Psychological review, 56(1): 51, 1949.
11. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630-645. Springer, 2016.
12. Hochreiter, Sepp, Younger, A Steven, and Conwell, Peter R. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
13. Kehoe, E James. A layered network model of associative learning: learning to learn and configuration. Psychological review, 95(4): 411, 1988.
14. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization, iclr, 2015.
15. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
16. Lake, Brenden M, Ullman, Tomer D, Tenenbaum, Joshua B, and Gershman, Samuel J. Building machines that learn and think like people. arXiv Report 1604.00289, 2016.
17. Li, SKe and Malik, Jitendra. Learning to optimize. In International Conference on Learning Representations, 2017.
18. Naik, Devang K and Mammone, RJ. Meta-neural networks that learn by learning. In International Joint Conference on Neural Networks, Volume 1, pp. 437-442. IEEE, 1992.
19. Nesterov, Yurii. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, Volume 27, pp. 372-376, 1983a.
20. Nesterov, Yurii. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, Volume 27, pp. 372-376, 1983b.
21. Ravi, Sachin and Larochelle, Hugo. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
22. Runarsson, Thomas Philip and Jonsson, Magnus Thor. Evolution and design of distributed learning rules. In IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, pp. 59-63. IEEE, 2000.
23. Santoro, ADAM, Bartunov, Sergey, Botvinick, Matthew, Wierstra, Daan, and Lillicrap, Timothy. Meta-leaming with memory-augmented neural networks. In International Conference on Machine Learning, 2016.
24. Schmidhuber, Jurgen. Evolutionary Principles in Self-Referential Learning. On Learning how to Learn: The Meta-Meta-Meta⋯-Hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
25. Silver, David, Huang, Aja, Maddison, Chris J, Guez, Arthur, Sifre, Laurent, Van Den Driessche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Veda, Lanctot, Marc, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587): 484-489, 2016.
26. Surjanovic, Sonja and Bingham, Derek. Optimization test functions and datasets, 2013. URL http://www.sfu.ca/ssurjano/optimization.html.
27. Sutton, Richard S. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for the Advancement of Artificial Intelligence, pp. 171-176, 1992.
28. Szegedy, Christian, Vanhoucke, Vincent, Ioffe, Sergey, Shlens, Jon, and Wojna, Zbigniew. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
29. Thrun, Sebastian and Pratt, Lorien. Learning to learn. Springer Science and Business Media, 1998.
30. Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4: 2, 2012.
31. Tseng, Paul. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal on Optimization, 8(2): 506-531, 1998.
32. Wang, Jane X., Kurth-Nelson, Zeb, Tirumala, Dhruva, Soyer, Hubert, Leibo, Joel Z., Munos, Rémi, Blundell, Charles, Kumaran, Dharshan, and Botvinick, Matt. Learning to reinforcement learn. arXiv Report 1611.05763, 2016.
33. Ward, Lewis B. Reminiscence and rote learning. Psychological Monographs, 49(4):i, 1937.
34. Zoph, Barret and Le, Quoc V. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.