Krylov subspace descent for deep learning

Journal of Machine Learning Research

Volumn 22, Issue , 2012, Pages 1261-1268

  Vinyals, Oriol ^a   Povey, Daniel ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; ITERATIVE METHODS; MATRIX ALGEBRA; OPTIMIZATION; STOCHASTIC SYSTEMS;

CONJUGATE GRADIENT DESCENTS; DAMPING PARAMETERS; DEEP NEURAL NETWORKS; FASTER CONVERGENCE; POSITIVE DEFINITE; POSITIVE SEMIDEFINITE; SECOND ORDER OPTIMIZATION; STOCHASTIC GRADIENT DESCENT;

CONJUGATE GRADIENT METHOD;

EID: 84954239313     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (14)

1. Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10:251-276, 1998.
2. Yoshua Bengio and Xavier Glorot. Understanding the difficulty of training deep feedforward neural networks. In AISTATS 2010, volume 9, pages 249-256, May 2010.
3. Richard H. Byrd, Gillian M. Chiny, Will Neveitt, and Jorge Nocedal. On the use of stochastic hessian information in optimization methods for machine learning. (submitted for publication), 2010.
4. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-AntoineManzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research, 11:625-660, 2010.
5. Geoffrey Hinton and Ruslan Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786):504-507, 2006.
6. Quoc V. Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y. Ng. On optimization methods for deep learning. In ICML, 2011.
7. Nicolas Le Roux, Yoshua Bengio, and Pierre antoine Manzagol. Topmoumoute online natural gradient algorithm. In NIPS, 2007.
8. James Martens. Deep learning via Hessian-free optimization. In ICML, 2010.
9. James Martens and Ilya Sutskever. Learning Recurrent Neural Networks with Hessian-Free Optimization. In ICML, 2011.
10. José Luis Morales and Jorge Nocedal. Enriched Methods for Large-Scale Unconstrained Optimization. Computational Optimization and Applications, 21: 143-154, 2000.
11. Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, New York, 2nd edition, 2006.
12. Barak A. Pearlmutter. Fast exact multiplication by the Hessian. Neural Computation, 6:147-160, 1994.
13. Nicol N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. In Neural Computation, 2002.
14. Oriol Vinyals and Suman Ravuri. Comparing Multilayer Perceptron to Deep Belief Network Tandem Features for Robust ASR. In ICASSP, 2011.