Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients

1st International Conference on Learning Representations, ICLR 2013 - Conference Track Proceedings

Volumn , Issue , 2013, Pages

  Schaul, Tom ^a   LeCun, Yann ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FINITE DIFFERENCE METHOD; GRADIENT METHODS; ORTHOGONAL FUNCTIONS;

ADAPTIVE LEARNING RATES; ESTIMATION PROCEDURES; FINITE DIFFERENCE APPROXIMATIONS; LINEAR COMPLEXITY; ORTHOGONAL GRADIENTS; PARALLELIZATIONS; STATIONARY PROBLEM; STOCHASTIC GRADIENT DESCENT;

STOCHASTIC SYSTEMS;

EID: 85083952393     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (12)

1. Schaul, T, Zhang, S, and LeCun, Y. No More Pesky Learning Rates. Technical report, June 2012.
2. Jacobs, R. A. Increased rates of convergence through learning rate adaptation. Neural Networks, 1(4):295–307, January 1988.
3. Almeida, L and Langlois, T. Parameter adaptation in stochastic optimization. On-line learning in neural . . . , 1999.
4. George, A. P and Powell, W. B. Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming. Machine Learning, 65(1):167–198, May 2006.
5. Nicolas Le Roux, A. F. A fast natural Newton method.
6. Duchi, J. C, Hazan, E, and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. 2010.
7. Amari, S, Park, H, and Fukumizu, K. Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Computation, 12(6):1399–1409, 2000.
8. LeCun, Y, Bottou, L, Orr, G, and Muller, K. Efficient backprop. In Orr, G and K., M, editors, Neural Networks: Tricks of the trade. Springer, 1998.
9. Byrd, R, Chin, G, Nocedal, J, and Wu, Y. Sample size selection in optimization methods for machine learning. Mathematical Programming, 2012.
10. Niu, F, Recht, B, Re, C, and Wright, S. J. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Matrix, (1):21, 2011.
11. Bordes, A, Bottou, L, and Gallinari, P. Sgd-qn: Careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 10:1737–1754, July 2009.
12. Le Roux, N, Manzagol, P, and Bengio, Y. Topmoumoute online natural gradient algorithm, 2008.