No more pesky learning rates

30th International Conference on Machine Learning, ICML 2013

Volumn , Issue PART 2, 2013, Pages 1380-1388

  Schaul, Tom ^a   Zhang, Sixin ^a   LeCun, Yann ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; SOFTWARE ENGINEERING;

ADAPTIVE APPROACH; LEARNING RATES; LEARNING TASKS; LOCAL GRADIENTS; NON-STATIONARY PROBLEMS; ONE-TIME; STOCHASTIC GRADIENT DESCENT; SYSTEMATIC SEARCHES;

LEARNING SYSTEMS;

EID: 84897487847     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Almeida, LB and Langlois, T. Parameter adaptation in stochastic optimization. On-line learning in neural networks, 1999.
2. Amari, Shun-ichi, Park, Hyeyoung, and Fukumizu, Kenji. Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons. Neural Computation, 12(6):1399-1409, June 2000. ISSN 0899-7667.
3. Bach, F. and Moulines, E. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems (NIPS), 2011.
4. Bordes, Antoine, Bottou, Léon, and Gallinari, Patrick. SGD-QN: Careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 10: 1737-1754, July 2009.
5. Bottou, Léon. Online algorithms and stochastic approximations. In Saad, David (ed.), Online Learning and Neural Networks. Cambridge University Press, Cambridge, UK, 1998.
6. Bottou, Léon and Bousquet, Olivier. The tradeoffs of large scale learning. In Sra, Suvrit, Nowozin, Sebastian, and Wright, Stephen J. (eds.), Optimization for Machine Learning, pp. 351-368. MIT Press, 2011.
7. Bottou, Léon and LeCun, Yann. Large scale online learning. In Thrun, Sebastian, Saul, Lawrence, and Schölkopf, Bernhard (eds.), Advances in Neural Information Processing Systems 16. MIT Press, Cambridge, MA, 2004.
8. Chapelle, Olivier and Erhan, Dumitru. Improved preconditioner for hessian free optimization. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
9. Duchi, John C., Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. 2010.
10. George, Abraham P. and Powell, Warren B. Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming. Machine Learning, 65(1):16798, May 2006. ISSN 0885-6125.
11. Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10).
12. Krizhevsky, Alex. Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto, 2009.
13. Le Roux, N., Manzagol, P.A., and Bengio, Y. Topmoumoute online natural gradient algorithm, 2008.
14. Le Roux, Nicolas and Fitzgibbon, Andrew. A fast natural newton method. In Proceedings of the 27th International Conference on Machine Learning. Citeseer, 2010.
15. LeCun, Y., Bottou, L., Orr, G., and Muller, K. Efficient backprop. In Orr, G. and K., Muller (eds.), Neural Networks: Tricks of the trade. Springer, 1998.
16. LeCun, Yann and Cortes, Corinna. The mnist dataset of handwritten digits. 1998. http://yann.lecun.com/exdb/mnist/.
17. Martens, J, Sutskever, I, and Swersky, K. Estimating the Hessian by Back-propagating Curvature. arXiv preprint arXiv:1206.6464, 2012.
18. Robbins, H. and Monro, S. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
19. Schaul, Tom and LeCun, Yann. Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients. In Proceedings of the International Conference on Learning Representations (ICLR), Scottsdale, AZ, 2013.
20. Schraudolph, Nicol N. Local gain adaptation in stochastic gradient descent. In Artificial Neural Networks, 1999. ICANN 99. Ninth International Conference on (Conf. Publ. No. 470), volume 2, pp. 569-574. IET, 1999.
21. Schraudolph, Nicol N. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1723-1738, 2002.
22. Xu, Wei. Towards optimal one pass large scale learning with averaged stochastic gradient descent. ArXiv-CoRR, abs/1107.2490, 2011.