Finding approximate local minima faster than gradient descent

Proceedings of the Annual ACM Symposium on Theory of Computing

Volumn Part F128415, Issue , 2017, Pages 1195-1199

  Agarwal, Naman ^a   Allen Zhu, Zeyuan ^b   Bullins, Brian ^a   Hazan, Elad ^a   Ma, Tengyu ^a  

^a PRINCETON UNIVERSITY   (United States)

^b INSTITUTE FOR ADVANCED STUDY   (United States)

Author keywords

Cubic regularization; Deep learning; Non convex optimization; Second order optimization

Indexed keywords

COMPLEX NETWORKS; COMPUTATIONAL COMPLEXITY; CONVEX OPTIMIZATION; DEEP LEARNING; EDUCATION;

CONVEX OBJECTIVES; CUBIC REGULARIZATION; GRADIENT DESCENT; NONCONVEX OPTIMIZATION; OPTIMIZATION PROBLEMS; SECOND ORDER OPTIMIZATION; TIME COMPLEXITY; TRAINING EXAMPLE;

OPTIMIZATION;

EID: 85024401503     PISSN: 07378017     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3055399.3055464     Document Type: Conference Paper

References (31)

1. Naman Agarwal, Brian Bullins, and Elad Hazan. Second order stochastic optimization for machine learning in linear time. arXiv preprint arXiv: 1602.03943, 2016.
2. Zeyuan Allen-Zhu. Natasha: Faster Stochastic Non-Convex Optimization via Strongly Non-Convex Parameter. ArXiv e-prints, abs/1702.00763, February 2017.
3. Zeyuan Allen-Zhu and Elad Hazan. Variance Reduction for Faster Non-Convex Optimization. In ICML, 2016.
4. Afonso S Bandeira, Nicolas Boumal, and Vladislav Voroninski. On the low-rank approach for semidefinite programs arising in synchronization and community detection. arXiv preprint arXiv: 1602.04426, 2016.
5. S. Bhojanapalli, B. Neyshabur, and N. Srebro. Global Optimality of Local Search for Low Rank Matrix Recovery. ArXiv e-prints, May 2016.
6. Yair Carmon, John C. Duchi, Oliver Hinder, and Aaron Sidford. Accelerated methods for non-convex optimization. arXiv preprint 1611.00756, 2016.
7. Coralia Cartis, Nicholas IM Gould, and Philippe L Toint. Adaptive cubic regularisation methods for unconstrained optimization. part i: motivation, convergence and numerical results. Mathematical Programming, 127(2): 245-295, 2011.
8. Coralia Cartis, Nicholas IM Gould, and Philippe L Toint. Adaptive cubic regularisation methods for unconstrained optimization. part ii: worst-case function-and derivative-evaluation complexity. Mathematical Programming, 130(2): 295-319, 2011.
9. Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In AISTATS, 2015.
10. Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in neural information processing systems, pages 2933-2941, 2014.
11. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12: 2121-2159, 2011.
12. Dan Garber and Elad Hazan. Fast and simple PCA via convex optimization. ArXiv e-prints, September 2015.
13. Dan Garber, Elad Hazan, Chi Jin, Sham M. Kakade, Cameron Musco, Praneeth Netrapalli, and Aaron Sidford. Robust shift-and-invert preconditioning: Faster and more sample efficient algorithms for eigenvector computation. In ICML, 2016.
14. Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. arXiv:1503.02101, 2015.
15. Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. In Proceedings of the 28th Annual Conference on Learning Theory, COLT 2015, 2015.
16. Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. In Proceedings of The 28th Conference on Learning Theory, COLT 2015, Paris, France, July 3-6, 2015, pages 797-842, 2015.
17. Rong Ge, Jason Lee, and Tengyu Ma. Matrix Completion has No Spurious Local Minimum. ArXiv e-prints, May 2016.
18. Rong Ge and Tengyu Ma. On the optimization landscape of tensor decompositions, 2016.
19. Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming, pages 1-26, feb 2015.
20. I.J. Goodfellow, O. Vinyals, and A. M. Saxe. Qualitatively characterizing neural network optimization problems. ArXiv e-prints, December 2014.
21. Elad Hazan and Tomer Koren. A linear-time algorithm for trust region problems. Mathematical Programming, pages 1-19, 2015.
22. Christopher J. Hillar and Lek-Heng Lim. Most tensor problems are np-hard. J. ACM, 60(6): 45, 2013.
23. Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Proceedings of the 29th Conference on Learning Theory, COLT2016, New York, USA, June 23-26, 2016, pages 1246-1257, 2016.
24. Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. Mathematical programming, 39(2): 117-129, 1987.
25. 2). In Doklady AN SSSR (translated as Soviet Mathematics Doklady), Volume 269, pages 543-547, 1983.
26. Yurii Nesterov. Introductory Lectures on Convex Programming Volume: A Basic course, Volume I. Kluwer Academic Publishers, 2004.
27. Yurii Nesterov and Boris T Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming, 108(1): 177-205, 2006.
28. Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1): 147-160, 1994.
29. Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
30. Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. arXiv preprint arXiv: 1309.2388, pages 1-45, 2013. Preliminary version appeared in NIPS 2012.
31. Jonathan Richard Shewchuk. An introduction to the conjugate gradient method without the agonizing pain, 1994.