Preconditioned spectral descent for deep learning

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 2971-2979

  Carlson, David E ^a   Collins, Edo ^b   Hsieh, Ya Ping ^b   Carin, Lawrence ^c   Cevher, Volkan ^b  

^a Columbia University ^*  (United States)

^b EPFL   (Switzerland)

^c DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; GRADIENT METHODS; INFORMATION SCIENCE; NEURAL NETWORKS;

COMPUTATIONAL CHALLENGES; COMPUTATIONAL TIME; LEARNING OBJECTIVES; OBJECTIVE FUNCTIONS; OPTIMIZATION ALGORITHMS; OPTIMIZATION SOLUTION; PRECONDITIONED METHOD; RESTRICTED BOLTZMANN MACHINE;

OPTIMIZATION;

EID: 84965148019     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. D. Carlson, V. Cevher, and L. Carin. Stochastic Spectral Descent for Restricted Boltzmann Machines. AISTATS, 2015.
2. D. Carlson, Y.-P. Hsieh, E. Collins, L. Carin, and V. Cevher. Stochastic Spectral Descent for Discrete Graphical Models. IEEE J. Special Topics in Signal Processing, 2016.
3. K. Cho, T. Raiko, and A. Ilin. Enhanced Gradient for Training Restricted Boltzmann Machines. Neural Computation, 2013.
4. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. Le Cun. The Loss Surfaces of Multilayer Networks. AISTATS 2015.
5. Y. N. Dauphin, H. de Vries, J. Chung, and Y. Bengio. RMSProp and equilibrated adaptive learning rates for non-convex optimization. arXiv:1502.04390 2015.
6. Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NIPS, 2014.
7. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2010.
8. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. Why Does Unsupervised Pre-training Help Deep Learning? JMLR 2010.
9. N. Halko, P. G. Martinsson, and J. A. Tropp. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions. SIAM Review 2011.
10. G. Hinton. A Practical Guide to Training Restricted Boltzmann Machines. U. Toronto Technical Report, 2010.
11. G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 2006.
12. G. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 2002.
13. J. A. Kelner, Y. T. Lee, L. Orecchia, and A. Sidford. An Almost-Linear-Time Algorithm for Approximate Max Flow in Undirected Graphs, and its Multicommodity Generalizations 2013.
14. A. Krizhevsky and G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. NIPS, 2012.
15. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. University of Toronto, Tech. Rep, 2009.
16. Q. V. Le, A. Coates, B. Prochnow, and A. Y. Ng. On Optimization Methods for Deep Learning. ICML, 2011.
17. B. Marlin and K. Swersky. Inductive principles for restricted Boltzmann machine learning. ICML, 2010.
18. J. Martens and R. Grosse. Optimizing Neural Networks with Kronecker-factored Approximate Curvature. arXiv:1503.05671 2015.
19. J. Martens and I. Sutskever. Parallelizable Sampling of Markov Random Fields. AISTATS, 2010.
20. V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
21. R. M. Neal. Annealed Importance Sampling. U. Toronto Technical Report, 1998.
22. V. Rokhlin, A. Szlam, and M. Tygert. A Randomized Algorithm for Principal Component Analysis. SIAM Journal on Matrix Analysis and Applications 2010.
23. R. Salakhutdinov and G. Hinton. Deep Boltzmann Machines. AISTATS, 2009.
24. R. Salakhutdinov and I. Murray. On the Quantitative Analysis of Deep Belief Networks. ICML, 2008.
25. T. Schaul, S. Zhang, and Y. Le Cun. No More Pesky Learning Rates. arXiv 1206.1106 2012.
26. P. Smolensky. Information Processing in Dynamical Systems: Foundations of Harmony Theory, 1986.
27. J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In NIPS, 2012.
28. T. Tieleman and G. Hinton. Using fast weights to improve persistent contrastive divergence. ICML, 2009.
29. L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus. Regularization of neural networks using dropconnect. In ICML, 2013.
30. M. D. Zeiler. ADADELTA: An Adaptive Learning Rate Method. arXiv 1212.5701 2012.