Global convergence of stochastic gradient descent for some non-convex matrix problems

32nd International Conference on Machine Learning, ICML 2015

Volumn 3, Issue , 2015, Pages 2322-2331

  De Sa, Christopher ^a   Olukotun, Kunle ^a   Ré, Christopher ^a  

^a Stanford University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; ITERATIVE METHODS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS; TIME VARYING NETWORKS;

GLOBAL CONVER-GENCE; LEAST SQUARES PROBLEMS; MATRIX COMPLETION; MATRIX PROBLEM; SAMPLING CONDITIONS; SDP RELAXATION; STOCHASTIC GRADIENT DESCENT; SUBSPACE TRACKING;

STOCHASTIC SYSTEMS;

EID: 84969963449     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (38)

1. Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008. ISBN 978-0-691-13298-3.
2. Agarwal, Alekh, Chapelle, Olivier, Dudfk, Miroslav, and Langford, John. A reliable effective terascale linear learning system. CoRR, abs/1110.4198, 2011.
3. Arora, R., Cotter, A., Livescu, K., and Srebro, N. Stochastic optimization for PCA and PLS. In Communication, Control, and Computing (Allerton), 2012 50th Annual Allerton Conference on, pp. 861-868, Oct 2012.
4. Arora, Raman, Cotter, Andy, and Srebro, Nati. Stochastic optimization of PCA with capped MSG. In Burges, C.j.c., Bottou, L., Welling, M., Ghahramani, TL., and Weinberger, K.q. (eds.), Advances in Neural Information Processing Systems 26, pp. 1815-1823. 2013.
5. Balsubramani, Akshay, Dasgupta, Sanjoy, and Freund, Yoav. The fast convergence of incremental PCA. In NIPS, pp. 3174-3182, 2013.
6. Balzano, Laura, Nowak, Robert, and Recht, Benjamin. Online identification and tracking of subspaces from highly incomplete information. In Communication, Control, and Computing (Allerton), 2010 48th Annual Allerton Conference on, pp. 704-711. IEEE, 2010.
7. Bottou, Lon. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pp. 177-186. 2010.
8. Bottou, Lon and Bousquet, Olivier. The tradeoffs of large scale learning. In IN: ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 20, pp. 161-168, 2008.
9. Burer, Samuel and Monteiro, Renato DC. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329-357, 2003.
10. Burer, Samuel and Monteiro, Renato DC. Local minima and convergence in low-rank semidefinite programming. Mathematical Programming, 103(3):427-4144, 2005.
11. Candès, Emmanuel, Li, Xiaodong, and Soltanolkotabi, Mahdi. Phase retrieval via wirtinger flow: Theory and algorithms. arXiv preprint arXiv: 1407.1065, 2014.
12. Candès, Emmanuel J. and Recht, Benjamin. Exact matrix completion via convex optimization. FoCM, 9(6):717-772, 2009. ISSN 1615-3375.
13. Candès, Emmanuel J. and Li, Xiaodong. Solving quadratic equations via phaselift when there are about as many equations as unknowns. FoCM, 14(5): 1017-1026, 2014.
14. Chen, Caihua, He, Bingsheng, and Yuan, Xiaoming. Matrix completion via an alternating direction method. IMAJNA, 2011.
15. do Carmo, M.P. Riemannian Geometry. Mathematics (Birkhäuser) theory. Birkhauser Boston, 1992. ISBN 9780817634902.
16. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121-2159, July 2011.
17. Fleming, Thomas R and Harrington, David P. Counting processes and survival analysis, volume 169, pp. 56-57. John Wiley & Sons, 1991.
18. Funk, Simon. Netflix Update: Try this at Home. 2006.
19. Gupta, Pankaj, Goel, Ashish, Lin, Jimmy, Sharma, Aneesh, Wang, Dong, and Zadeh, Reza. WTF: The who to follow service at twitter. WWW '13, pp. 505-514, 2013.
20. Hardt, Moritz. Understanding alternating minimization for matrix completion. In Foundations of Computer Science (FOCS), 2014 IEEE 55th Annual Symposium on, pp. 651-660. IEEE, 2014.
21. Hardt, Moritz and Price, Eric. The noisy power method: A meta algorithm with applications. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.d., and Weinberger, K.q. (eds.), Advances in Neural Information Processing Systems 27, pp. 2861-2869. Curran Associates, Inc., 2014.
22. Horstmeyer, R., Chen, R. Y., Ou, X., Ames, B., Tropp, J. A., and Yang, C. Solving ptychography with a convex relaxation. ArXiv e-prints, dec 2014.
23. Hu, Chonghai, Kwok, James T., and Pan, Weike. Accelerated gradient methods for stochastic optimization and online learning. In Advances in Neural Information Processing Systems 22, pp. 781-789, 2009.
24. Jain, Prateek and Netrapalli, Praneeth. Fast exact matrix completion with finite samples. arXiv preprint arXiv:1411.1087, 2014.
25. Jain, Prateek, Netrapalli, Praneeth, and Sanghavi, Sujay. Low-rank matrix completion using alternating minimization. In Proceedings of the Forty-fifth Annual ACM STOC, pp. 665-674. ACM, 2013.
26. John Goes, Teng Zhang, Raman Arora and Lerman, Gilad. Robust stochastic principal component analysis. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics, pp. 266-274, 2014.
27. Journée, M., Bach, F., Absil, P.-A., and Sepulchre, R. Low-rank optimization on the conc of positive scmidcfinite matrices. SIAM J. on Optimization, 20(5):2327-2351, May 2010.
28. Keshavan, R.H., Montanari, A., and Oh, Sewoong. Matrix completion from a few entries. Information Theory, IEEE Transactions on, 56(6):2980-2998, June 2010.
29. Mishra, Bamdev, Meyer, Gilles, Bach, Francis, and Sepulchre, Rodolphe. Low-rank optimization with trace norm penalty. SIAM Journal on Optimization, 23(4):2124-2149, 2013.
30. Niu, Feng, Recht, Benjamin, Ré, Christopher, and Wright, Stephen J. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In In NIPS, 2011.
31. Oja, Erkki. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of Mathematical Analysis and Applications, 106, 1985.
32. Oscar Boykin, Sam Ritchie, Ian O'Connell, Jimmy Lin. Summingbird: A framework for integrating batch and online mapreduce computations. In Proceedings of the VLDB Endowment, volume 7, pp. 1441-1451, 2013-2014.
33. Recht, Benjamin and Re, Christopher. Parallel stochastic gradient algorithms for large-scale matrix completion. Mathematical Programming Computation, 5(2): 201-226, 2013. ISSN 1867-2949.
34. Shamir, Ohad. Making gradient descent optimal for strongly convex stochastic optimization. CoRR, abs/1109.5647, 2011.
35. Shamir, Ohad. A stochastic PCA algorithm with an exponential convergence rate. CoRR, abs/1409.2848, 2014.
36. Sun, Ruoyu and Luo, Zhi-Quan. Guaranteed matrix completion via non-convex factorization. arXiv preprint arXiv: 1411.8003, 2014.
37. Teflioudi, Christina, Makari, Faraz, and Gemulla, Rainer. Distributed matrix completion. 2013 IEEE 13th 1CDM, 0:655-664, 2012. ISSN 1550-4786.
38. Zou, Hui, Hastie, Trevor, and Tibshirani, Robert. Sparse principal component analysis. J. Comp. Graph. Stat., 15:2006, 2004.