Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Advances in Neural Information Processing Systems

Volumn 4, Issue January, 2014, Pages 2933-2941

  Dauphin, Yann N ^a   Pascanu, Razvan ^a   Gulcehre, Caglar ^a   Cho, Kyunghyun ^a   Ganguli, Surya ^b   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONVEX OPTIMIZATION; ERRORS; GLOBAL OPTIMIZATION; INFORMATION SCIENCE; NEURAL NETWORKS; NEWTON-RAPHSON METHOD; RANDOM VARIABLES; RECURRENT NEURAL NETWORKS;

HIGH DIMENSIONAL SPACES; HIGH-DIMENSIONAL PROBLEMS; NONCONVEX OPTIMIZATION; QUASI-NEWTON METHODS; RANDOM MATRIX THEORY; SADDLE POINT PROBLEMS; SCIENCE AND ENGINEERING; SECOND ORDER OPTIMIZATION;

OPTIMIZATION;

EID: 84928534967     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Baldi, P. and Hornik, K. (1989). Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1), 53-58.
2. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., and Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements.
3. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. 5(2), 157-166. Special Issue on Recurrent Neural Networks, March 94.
4. Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 281-305.
5. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: aCPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy).
6. Bray, A. J. and Dean, D. S. (2007). Statistics of critical points of gaussian fields on large-dimensional spaces. Physics Review Letter, 98, 150201.
7. Callahan, J. (2010). Advanced Calculus: A Geometric View. Undergraduate Texts in Mathematics. Springer.
8. Fyodorov, Y. V. and Williams, I. (2007). Replica symmetry breaking condition exposed by random matrix calculation of landscape complexity. Journal of Statistical Physics, 129(5-6), 1081-1116.
9. Inoue, M., Park, H., and Okada, M. (2003). On-line learning theory of soft committee machines with correlated hidden units steepest gradient descent and natural gradient descent. Journal of the Physical Society of Japan, 72(4), 805-810.
10. Le Roux, N., Manzagol, P.-A., and Bengio, Y. (2007). Topmoumoute online natural gradient algorithm. Advances in Neural Information Processing Systems.
11. Martens, J. (2010). Deep learning via hessian-free optimization. In International Conference in Machine Learning, pages 735-742.
12. Mizutani, E. and Dreyfus, S. (2010). An analysis on negative curvature induced by singularity in multi-layer neural-network learning. In Advances in Neural Information Processing Systems, pages 1669-1677.
13. Murray, W. (2010). Newton-type methods. Technical report, Department of Management Science and Engineering, Stanford University.
14. Nocedal, J. and Wright, S. (2006). Numerical Optimization. Springer.
15. Parisi, G. (2007). Mean field theory of spin glasses: statistics and dynamics. Technical Report Arxiv 0706.0094.
16. Pascanu, R. and Bengio, Y. (2014). Revisiting natural gradient for deep networks. In International Conference on Learning Representations.
17. Pascanu, R., Mikolov, T., and Bengio, Y. (2013). Onthe difficulty of training recurrent neural networks. In ICML'2013.
18. Pascanu, R., Dauphin, Y., Ganguli, S., and Bengio, Y. (2014). On the saddle point problem for non-convex optimization. Technical Report Arxiv 1405.4604.
19. Rasmussen, C. E. and Williams, C. K. I. (2005). Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press.
20. Rattray, M., Saad, D., and Amari, S. I. (1998). Natural Gradient Descent for On-Line Learning. Physical Review Letters, 81(24), 5461-5464.
21. Saad, D. and Solla, S. A. (1995). On-line learning in soft committee machines. Physical Review E, 52, 4225-4243.
22. Saxe, A., McClelland, J., and Ganguli, S. (2013). Learning hierarchical category structure in deep neural networks. Proceedings of the 35th annual meeting of the Cognitive Science Society, pages 1271-1276.
23. Saxe, A., McClelland, J., and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In International Conference on Learning Representations.
24. Sohl-Dickstein, J., Poole, B., and Ganguli, S. (2014). Fast large-scale optimization by unifying stochastic gradient and quasi-newton methods. In ICML'2014.
25. Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E. (2013). On the importance of initialization and momentum in deep learning. In S. Dasgupta and D. Mcallester, editors, Proceedings of the 30th International Conference on Machine Learning (ICML-13), Volume 28, pages 1139-1147. JMLR Workshop and Conference Proceedings.
26. Vinyals, O. and Povey, D. (2012). Krylov Subspace Descent for Deep Learning. In AISTATS.
27. Wigner, E. P. (1958). On the distribution of the roots of certain symmetric matrices. The Annals of Mathematics, 67(2), 325-327.