Cyclical learning rates for training neural networks

Proceedings - 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017

Volumn , Issue , 2017, Pages 464-472

  Smith, Leslie N ^a  

^a NAVAL RESEARCH LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; DEEP NEURAL NETWORKS; LEARNING ALGORITHMS; STOCHASTIC SYSTEMS;

BEST VALUE; BOUNDARY VALUES; CLASSIFICATION ACCURACY; GLOBAL LEARNING; HYPER-PARAMETER; LEARNING RATES;

DEEP LEARNING;

EID: 85020166041     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WACV.2017.58     Document Type: Conference Paper

References (29)

1. K. Bache, D. DeCoste, and P. Smyth. Hot swapping for online adaptation of optimization hyperparameters. arXiv preprint arXiv:1412.6599, 2014.
2. Y. Bengio. Neural Networks: Tricks of the Trade, chapter Practical recommendations for gradient-based training of deep architectures, pages 437-478. Springer Berlin Heidelberg, 2012.
3. T. M. Breuel. The effects of hyperparameters on sgd training of neural networks. arXiv preprint arXiv:1508.02788, 2015. 2.
4. Y. N. Dauphin, H. de Vries, J. Chung, and Y. Bengio. Rmsprop and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390, 2015.
5. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121-2159, 2011.
6. A. P. George and W. B. Powell. Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming. Machine learning, 65(1):167-198, 2006.
7. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 580-587. IEEE, 2014.
8. A. Graves and N. Jaitly. Towards end-To-end speech recognition with recurrent neural networks. In Proceedings of the 31st International Conference on Machine Learning (ICML- 14), pages 1764-1772, 2014.
9. C. Gulcehre and Y. Bengio. Adasecant: Robust adaptive secant method for stochastic gradient. arXiv preprint arXiv:1412.7419, 2014.
10. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, 2015.
11. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027, 2016.
12. G. Huang, Z. Liu, and K. Q.Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016.
13. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382, 2016.
14. B. Huval, T. Wang, S. Tandon, J. Kiske, W. Song, J. Pazhayampallil, M. Andriluka, R. Cheng-Yue, F. Mujica, A. Coates, et al. An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716, 2015.
15. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
16. D. Kingma and J. Lei-Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2015.
17. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012.
18. I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.
19. Y. Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pages 372-376, 1983.
20. S. Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1600.04747, 2016.
21. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Image Net Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015. 6.
22. T. Schaul, S. Zhang, and Y. LeCun. No more pesky learning rates. arXiv preprint arXiv:1206.1106, 2012.
23. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
24. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112, 2014.
25. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.
26. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1701-1708. IEEE, 2014.
27. T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012.
28. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014.
29. M. D. Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.