Efficient approaches for escaping higher order saddle points in non-convex optimization

Journal of Machine Learning Research

Volumn 49, Issue June, 2016, Pages 81-102

  Anandkumar, Animashree ^a   Ge, Rong ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b DUKE UNIVERSITY   (United States)

Author keywords

Degenerate saddle points; Higher order conditions for local optimality; Non convex optimization; Trust region methods

Indexed keywords

HEURISTIC ALGORITHMS; MACHINE LEARNING;

APPLIED MACHINE LEARNING; HIGHER ORDER DERIVATIVES; LOCAL OPTIMALITY; LOCAL SEARCH HEURISTICS; NONCONVEX OPTIMIZATION; SADDLE POINT; SECOND ORDER DERIVATIVES; TRUST-REGION METHODS;

CONVEX OPTIMIZATION;

EID: 85072254591     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (24)

1. Shun-Ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamics of learning in neuromanifolds. Neural computation, 18(5):1007-1065, 2006.
2. Peter Auer, Mark Herbster, Manfred K Warmuth, et al. Exponentially many local minima for single neurons. Advances in neural information processing systems, pages 316-322, 1996.
3. Antonio Auffinger, Gerard Ben Arous, et al. Complexity of random smooth functions on the high-dimensional sphere. The Annals of Probability, 41(6):4214-4247, 2013.
4. Michel Baes. Estimate sequence methods: extensions and approximations. Institute for Operations Research, ETH, Zürich, Switzerland, 2009.
5. Dennis S Bernstein. A systematic approach to higher-order necessary conditions in optimization theory. SIAM journal on control and optimization, 22(2):211-238, 1984.
6. Anthony Carbery and James Wright. Distributional and l^ q norm inequalities for polynomials over convex bodies in r^ n. Mathematical Research Letters, 8(3):233-248, 2001.
7. Dustin Cartwright and Bernd Sturmfels. The number of eigenvalues of a tensor. Linear algebra and its applications, 438(2):942-952, 2013.
8. Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in neural information processing systems, pages 2933-2941, 2014.
9. Peter JC Dickinson and Luuk Gijben. On the computational complexity of membership problems for the completely positive cone and its dual. Computational optimization and applications, 57(2):403-415, 2014.
10. Alan Frieze, Mark Jerrum, and Ravi Kannan. Learning linear transformations. In focs, page 359. IEEE, 1996.
11. Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. In Proceedings of The 28th Conference on Learning Theory, pages 797-842, 2015.
12. Rich Caruana Steve Lawrence Lee Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, volume 13, page 402, 2001.
13. Christopher J Hillar and Lek-Heng Lim. Most tensor problems are np-hard. Journal of the ACM (JACM), 60(6):45, 2013.
14. Masato Inoue, Hyeyoung Park, and Masato Okada. On-line learning theory of soft committee machines with correlated hidden units-steepest gradient descent and natural gradient descent-. Journal of the Physical Society of Japan, 72(4):805-810, 2003.
15. Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. Mathematical programming, 39(2):117-129, 1987.
16. Yurii Nesterov. Squared functional systems and optimization problems. In High performance optimization, pages 405-440. Springer, 2000.
17. Yurii Nesterov and Boris T Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming, 108(1):177-205, 2006.
18. Jiawang Nie. The hierarchy of local minimums in polynomial optimization. Mathematical Programming, 151(2):555-583, 2015.
19. David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52(4):4225, 1995.
20. Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural networks. arXiv preprint arXiv:1511.04210, 2015.
21. Ju Sun, Qing Qu, and John Wright. When are nonconvex problems not scary? arXiv preprint arXiv:1510.06096, 2015.
22. Santosh S Vempala and Ying Xiao. Structure from local optima: Learning subspace juntas via higher order pca. arXiv preprint arXiv:1108.3329, 2011.
23. Jack Warga. Higher order conditions with and without lagrange multipliers. SIAM journal on control and optimization, 24(4):715-730, 1986.
24. Haikun Wei, Jun Zhang, Florent Cousseau, Tomoko Ozeki, and Shun-ichi Amari. Dynamics of learning near singularities in layered networks. Neural computation, 20(3):813-843, 2008.