Riemannian SVRG: Fast stochastic optimization on riemannian manifolds

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 4599-4607

  Zhang, Hongyi ^a   Reddi, Sashank J ^b   Sra, Suvrit ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

VECTOR SPACES;

CONVERGENCE ANALYSIS; NON-ASYMPTOTIC; RIEMANNIAN MANIFOLD; RIEMANNIAN OPTIMIZATIONS; SMOOTH FUNCTIONS; SPACE PROBLEM; STOCHASTIC OPTIMIZATIONS; VARIANCE REDUCTION TECHNIQUES;

OPTIMIZATION;

EID: 85018894780     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2009.
2. A. Agarwal and L. Bottou. A lower bound for the optimization of finite sums. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 78-86, 2015.
3. Z. Allen-Zhu and E. Hazan. Variance reduction for faster non-convex optimization. arXiv:1603.05643, 2016.
4. F. Bach and E. Moulines. Non-strongly-convex smooth stochastic approximation with convergence rate o (1/n). In Advances in Neural Information Processing Systems, pages 773-781, 2013.
5. R. Bhatia. Positive Definite Matrices. Princeton University Press, 2007.
6. D. A. Bini and B. Iannazzo. Computing the karcher mean of symmetric positive definite matrices. Linear Algebra and its Applications, 438(4):1700-1710, 2013.
7. S. Bonnabel. Stochastic gradient descent on Riemannian manifolds. Automatic Control, IEEE Transactions on, 58(9):2217-2229, 2013.
8. A. Cherian and S. Sra. Riemannian dictionary learning and sparse coding for positive definite matrices. arXiv:1507.02772, 2015.
9. M. Congedo, B. Afsari, A. Barachant, and M. Moakher. Approximate joint diagonalization and geometric mean of symmetric positive definite matrices. PloS one, 10(4):e0121423, 2015.
10. A. Defazio, F. Bach, and S. Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In NIPS, pages 1646-1654, 2014.
11. A. Edelman, T. A. Arias, and S. T. Smith. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20(2):303-353, 1998.
12. D. Garber and E. Hazan. Fast and simple pca via convex optimization. arXiv preprint arXiv:1509.05647, 2015.
13. S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
14. P. Gong and J. Ye. Linear convergence of variance-reduced stochastic gradient without strong convexity. arXiv preprint arXiv:1406.1102, 2014.
15. R. Hosseini and S. Sra. Matrix manifold optimization for Gaussian mixtures. In NIPS, 2015.
16. B. Jeuris, R. Vandebril, and B. Vandereycken. A survey and comparison of contemporary algorithms for computing the matrix geometric mean. Electronic Transactions on Numerical Analysis, 39:379-402, 2012.
17. C. Jin, S. M. Kakade, C. Musco, P. Netrapalli, and A. Sidford. Robust shift-and-invert preconditioning: Faster and more sample efficient algorithms for eigenvector computation. arXiv:1510.08896, 2015.
18. R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, pages 315-323, 2013.
19. H. Kasai, H. Sato, and B. Mishra. Riemannian stochastic variance reduced gradient on grassmann manifold. arXiv preprint arXiv:1605.07367, 2016.
20. J. Konečnỳ and P. Richtárik. Semi-stochastic gradient descent methods. arXiv:1312.1666, 2013.
21. X. Liu, A. Srivastava, and K. Gallivan. Optimal linear representations of images for object recognition. IEEE TPAMI, 26(5):662-666, 2004.
22. M. Moakher. Means and averaging in the group of rotations. SIAM journal on matrix analysis and applications, 24(1):1-16, 2002.
23. E. Oja. Principal components, minor components, and linear neural networks. Neural Networks, 5(6): 927-935, 1992.
24. P. Petersen. Riemannian geometry, Volume 171. Springer Science & Business Media, 2006.
25. S. J. Reddi, A. Hefny, S. Sra, B. Póczós, and A. Smola. Stochastic variance reduction for nonconvex optimization. arXiv:1603.06160, 2016.
26. H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22: 400-407, 1951.
27. R. Y. Rubinstein and D. P. Kroese. Simulation and the Monte Carlo method, Volume 707. John Wiley & Sons, 2011.
28. M. Schmidt, N. L. Roux, and F. Bach. Minimizing finite sums with the stochastic average gradient. arXiv:1309.2388, 2013.
29. O. Shamir. A Stochastic PCA and SVD Algorithm with an Exponential Convergence Rate. In International Conference on Machine Learning (ICML-15), pages 144-152, 2015.
30. S. Sra and R. Hosseini. Geometric optimisation on positive definite matrices for elliptically contoured distributions. In Advances in Neural Information Processing Systems, pages 2562-2570, 2013.
31. J. Sun, Q. Qu, and J. Wright. Complete dictionary recovery over the sphere ii: Recovery by riemannian trust-region method. arXiv:1511.04777, 2015.
32. M. Tan, I. W. Tsang, L. Wang, B. Vandereycken, and S. J. Pan. Riemannian pursuit for big matrix recovery. In International Conference on Machine Learning (ICML-14), pages 1539-1547, 2014.
33. C. Udriste. Convex functions and optimization methods on Riemannian manifolds, Volume 297. Springer Science & Business Media, 1994.
34. B. Vandereycken. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimization, 23(2):1214-1236, 2013.
35. A. Wiesel. Geodesic convexity and covariance estimation. IEEE Transactions on Signal Processing, 60(12):6182-6189, 2012.
36. L. Xiao and T. Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization, 24(4):2057-2075, 2014.
37. X. Yuan, W. Huang, P.-A. Absil, and K. Gallivan. A riemannian limited-memory bfgs algorithm for computing the matrix geometric mean. Procedia Computer Science, 80:2147-2157, 2016.
38. H. Zhang and S. Sra. First-order methods for geodesically convex optimization. arXiv:1602.06053, 2016.
39. T. Zhang, A. Wiesel, and M. S. Greco. Multivariate generalized Gaussian distribution: Convexity and graphical models. Signal Processing, IEEE Transactions on, 61(16):4141-4148, 2013.