Learning step size controllers for robust neural network training

30th AAAI Conference on Artificial Intelligence, AAAI 2016

Volumn , Issue , 2016, Pages 1519-1525

  Daniel, Christian ^a   Taylor, Jonathan ^b   Nowozin, Sebastian ^a  

^a DARMSTADT UNIVERSITY OF TECHNOLOGY   (Germany)

^b MICROSOFT RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; NEURAL NETWORKS; STOCHASTIC SYSTEMS;

ADAPTIVE CONTROLLERS; LEARNING METHODS; LEARNING PROBLEM; LEARNING RATES; NEURAL NETWORK TRAINING; NEURAL NETWORKS (NNS); PRIOR KNOWLEDGE; STOCHASTIC GRADIENT DESCENT;

LEARNING ALGORITHMS;

EID: 85007152018     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Amari, S. 1998. Natural gradient works efficiently in learning. Neural computation.
2. Bastian, M.; Gunther, J.; and Moon, T. 2011. A simplified natural gradient learning algorithm. Advances in Artificial Neural Systems.
3. Daniel, C.; Neumann, G.; Kroemer, O.; and Peters, J. 2013. Learning sequential motor tasks. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
4. Darken, C., and Moody, J. 1990. Fast adaptive k-means clustering: some empirical results. International Joint Conference on Neural Networks (IJCNN).
5. Dauphin, Y. N.; de Vries, H.; Chung, J.; and Bengio, Y. 2015. RMSProp and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390.
6. Duchi, J.; Hazan, E.; and Singer, Y. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research (JMLR) 12:2121-2159.
7. Hinton, G.; Deng, L.; Yu, D.; Dahl, G. E.; Mohamed, A.-r.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen, P.; Sainath, T. N.; and Others. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29(6):82- 97.
8. Jacobs, R. 1988. Increased rates of convergence through learning rate adaptation. Neural networks.
9. Kingma, D., and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
10. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. ImageNet Classification with Deep Convolutional Neural Networks. Advances In Neural Information Processing Systems (NIPS).
11. Kroemer, O.; Lampert, C.; and Peters, J. 2011. Learning Dynamic Tactile Sensing with Robust Vision-based Training. IEEE Transactions on Robotics (T-Ro) (3):545-557.
12. Le Roux, N.; Bengio, Y.; and Fitzgibbon, A. 2012. Improving First and Second-Order Methods by Modeling Uncertainty. Optimization for Machine Learning 403.
13. Peters, J.;Mülling, K.; and Altun, Y. 2010. Relative Entropy Policy Search. In Proceedings of the National Conference on Artificial Intelligence (AAAI).
14. Riedmiller, M., and Braun, H. 1993. A direct adaptive method for faster backpropagation learning: The RPROP algorithm. Neural Networks.
15. Schaul, T.; Zhang, S.; and LeCun, Y. 2013. No More Pesky Learning Rates. International Conference on Machine Learning (ICML).
16. Senior, A.; Heigold, G.; Ranzato, M.; and Yang, K. 2013. An empirical study of learning rates in deep neural networks for speech recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
17. Snoek, J.; Larochelle, H.; and Adams, R. P. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NIPS).
18. Sutton, R., and Barto, A. 1998. Reinforcement learning: An introduction. MIT press Cambridge.
19. Sutton, R. 1992. Adapting bias by gradient descent: An incremental version of delta-bar-delta. Proceedings of the National Conference on Artificial Intelligence (AAAI).
20. Thiery, C., and Scherrer, B. 2009. Building controllers for Tetris. International Computer Games Association Journal.
21. Tieleman, T., and Hinton, G. 2012. Lecture 6.5 RMSprop: Divide the Gradient by The Running Average of its recent magnitude. In COURSERA: Neural Networks for Machine Learning.
22. Vedaldi, A., and Lenc, K. 2014. MatConvNet - Convolutional Neural Networks for MATLAB. CoRR abs/1412.4.
23. Zeiler, M. D. 2012. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701.
24. Zhang, Y.; Duchi, J. C.; and Wainwright, M. J. 2012. Communication-efficient algorithms for statistical optimization. In Advances in Neural Information Processing Systems (NIPS).