Convex optimization: Algorithms and complexity

Foundations and Trends in Machine Learning

Volumn 8, Issue 3-4, 2015, Pages 231-357

  Bubeck, Sébastien ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTATIONAL COMPLEXITY; CONVEX OPTIMIZATION; LEARNING SYSTEMS; MIRRORS; RANDOM PROCESSES; RELAXATION PROCESSES; STOCHASTIC SYSTEMS; STRUCTURAL OPTIMIZATION;

ALGORITHMS AND COMPLEXITY; BLACK-BOX OPTIMIZATION; COMBINATORIAL PROBLEM; CUTTING PLANE METHODS; INTERIOR-POINT METHOD; NESTEROV'S SMOOTHING; STOCHASTIC GRADIENT DESCENT; STOCHASTIC OPTIMIZATIONS;

OPTIMIZATION;

EID: 84983143287     PISSN: 19358237     EISSN: 19358245     Source Type: Journal    
DOI: 10.1561/2200000050     Document Type: Article

References (89)

1. A. Agarwal and L. Bottou. A lower bound for the optimization of finite sums. Arxiv preprint arXiv:1410.0723, 2014.
2. Z. Allen-Zhu and L. Orecchia. Linear coupling: An ultimate unification of gradient and mirror descent. Arxiv preprint arXiv:1407.1537, 2014.
3. K. M. Anstreicher. Towards a practical volumetric cutting plane method for convex programming. SIAM Journal on Optimization, 9(1):190-206, 1998.
4. J.Y Audibert, S. Bubeck, and R. Munos. Bandit view on noisy optimization. In S. Sra, S. Nowozin, and S. Wright, editors, Optimization for Machine Learning. MIT press, 2011.
5. J.Y. Audibert, S. Bubeck, and G. Lugosi. Regret in online combinatorial optimization. Mathematics of Operations Research, 39:31-45, 2014.
6. F. Bach. Learning with submodular functions: A convex optimization perspective. Foundations and Trends R in Machine Learning, 6(2-3):145-373, 2013.
7. F. Bach and E. Moulines. Non-strongly-convex smooth stochastic approximation with convergence rate o(1/n). In Advances in Neural Information Processing Systems (NIPS), 2013.
8. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations and Trends R in Machine Learning, 4(1):1-106, 2012.
9. B. Barak. Sum of squares upper bounds, lower bounds, and open questions. Lecture Notes, 2014.
10. A. Beck and M. Teboulle. Mirror Descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167- 175, 2003.
11. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183-202, 2009.
12. A. Ben-Tal and A. Nemirovski. Lectures on modern convex optimization: Analysis, algorithms, and engineering applications. Society for Industrial and Applied Mathematics (SIAM), 2001.
13. D. Bertsimas and S. Vempala. Solving convex programs by random walks. Journal of the ACM, 51:540-556, 2004.
14. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
15. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends R in Machine Learning, 3(1):1-122, 2011.
16. S. Bubeck. Introduction to online optimization. Lecture Notes, 2011.
17. S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends R in Machine Learning, 5(1):1-122, 2012.
18. S. Bubeck and R. Eldan. The entropic barrier: A simple and optimal universal self-concordant barrier. Arxiv preprint arXiv:1412.1587, 2014.
19. S. Bubeck, R. Eldan, and J. Lehec. Sampling from a log-concave distribution with projected langevin monte carlo. Arxiv preprint arXiv:1507.02564, 2015a.
20. S. Bubeck, Y.-T. Lee, and M. Singh. A geometric alternative to nesterov's accelerated gradient descent. Arxiv preprint arXiv:1506.08187, 2015b.
21. E. Candès and B. Recht. Exact matrix completion via convex optimization. Foundations of Computational mathematics, 9(6):717-772, 2009.
22. A. Cauchy. Méthode générale pour la résolution des systemes d'équations simultanées. Comp. Rend. Sci. Paris, 25(1847):536-538, 1847.
23. N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
24. A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40(1):120-145, 2011.
25. K. Clarkson, E. Hazan, and D. Woodruff. Sublinear optimization for machine learning. Journal of the ACM, 2012.
26. A. Conn, K. Scheinberg, and L. Vicente. Introduction to Derivative-Free Optimization. Society for Industrial and Applied Mathematics (SIAM), 2009.
27. T. M. Cover. 1990 shannon lecture. IEEE information theory society newsletter, 42(4), 1992.
28. A. d'Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Optimization, 19(3):1171-1183, 2008.
29. A. Defazio, F. Bach, and S. Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems (NIPS), 2014.
30. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research, 13:165-202, 2012.
31. J. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In Proceedings of the 23rd Annual Conference on Learning Theory (COLT), 2010.
32. J. C. Dunn and S. Harshbarger. Conditional gradient algorithms with open loop step size rules. Journal of Mathematical Analysis and Applications, 62 (2):432-444, 1978.
33. M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95-110, 1956.
34. M. P. Friedlander and P. Tseng. Exact regularization of convex programs. SIAM Journal on Optimization, 18(4):1326-1350, 2007.
35. M. Goemans and D. Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM, 42(6):1115-1145, 1995.
36. M. D. Grigoriadis and L. G. Khachiyan. A sublinear-time randomized approximation algorithm for matrix games. Operations Research Letters, 18: 53-58, 1995.
37. B. Grünbaum. Partitions of mass-distributions and of convex bodies by hyperplanes. Pacific J. Math, 10(4):1257-1261, 1960.
38. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, 2001.
39. E. Hazan. The convex optimization approach to regret minimization. In S. Sra, S. Nowozin, and S.Wright, editors, Optimization for Machine Learning, pages 287-303. MIT press, 2011.
40. M. Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 427-435, 2013.
41. P. Jain, P. Netrapalli, and S. Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the Forty-fifth Annual ACM Symposium on Theory of Computing, STOC '13, pages 665-674, 2013.
42. R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems (NIPS), 2013.
43. L. K. Jones. A simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training. Annals of Statistics, pages 608-613, 1992.
44. A. Juditsky and A. Nemirovski. First-order methods for nonsmooth convex large-scale optimization, i: General purpose methods. In S. Sra, S. Nowozin, and S. Wright, editors, Optimization for Machine Learning, pages 121-147. MIT press, 2011a.
45. A. Juditsky and A. Nemirovski. First-order methods for nonsmooth convex large-scale optimization, ii: Utilizing problem's structure. In S. Sra, S. Nowozin, and S. Wright, editors, Optimization for Machine Learning, pages 149-183. MIT press, 2011b.
46. N. Karmarkar. A new polynomial-time algorithm for linear programming. Combinatorica, 4:373-395, 1984.
47. S. Lacoste-Julien, M. Schmidt, and F. Bach. A simpler approach to obtaining an o (1/t) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002, 2012.
48. N. Le Roux, M. Schmidt, and F. Bach. A stochastic gradient method with an exponential convergence rate for strongly-convex optimization with finite training sets. In Advances in Neural Information Processing Systems (NIPS), 2012.
49. Y.-T. Lee and A. Sidford. Path finding i :solving linear programs with ÃT(sqrt(rank)) linear system solves. Arxiv preprint arXiv:1312.6677, 2013.
50. Y.-T. Lee, A. Sidford, and S. C.-W Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. abs/1508.04874, 2015.
51. A. Levin. On an algorithm for the minimization of convex functions. In Soviet Mathematics Doklady, volume 160, pages 1244-1247, 1965.
52. L. Lovász. Hit-and-run mixes fast. Math. Prog., 86:443-461, 1998.
53. G. Lugosi. Comment on: 1-penalization for mixture regression models. Test, 19(2):259-263, 2010.
54. N. Maculan and G. G. de Paula. A linear-time median-finding algorithm for projecting a vector on the simplex of rn. Operations research letters, 8(4): 219-222, 1989.
55. A. Nemirovski. Orth-method for smooth convex optimization. Izvestia AN SSSR, Ser. Tekhnicheskaya Kibernetika, 2, 1982.
56. A. Nemirovski. Information-based complexity of convex programming. Lecture Notes, 1995.
57. A. Nemirovski. Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1):229-251, 2004a.
58. A. Nemirovski. Interior point polynomial time methods in convex programming. Lecture Notes, 2004b.
59. A. Nemirovski and D. Yudin. Problem Complexity and Method Efficiency in Optimization. Wiley Interscience, 1983.
60. Y. Nesterov. A method of solving a convex programming problem with convergence rate o(1/k2). Soviet Mathematics Doklady, 27(2):372-376, 1983.
61. Y. Nesterov. Quality of semidefinite relaxation for nonconvex quadratic optimization. CORE Discussion Papers 1997019, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE), 1997.
62. Y. Nesterov. Introductory lectures on convex optimization: A basic course. Kluwer Academic Publishers, 2004a.
63. Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 103(1):127-152, 2004b.
64. Y. Nesterov. Gradient methods for minimizing composite objective function. Core discussion papers, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE), 2007.
65. Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22:341-362, 2012.
66. Y. Nesterov and A. Nemirovski. Interior-point polynomial algorithms in convex programming. Society for Industrial and Applied Mathematics (SIAM), 1994.
67. D. Newman. Location of the maximum on unimodal surfaces. Journal of the ACM, 12(3):395-398, 1965.
68. J. Nocedal and S. J. Wright. Numerical Optimization. Springer, 2006.
69. N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends R in Optimization, 1(3):123-231, 2013.
70. A.Rakhlin. Lecture notes on online learning. 2009.
71. J. Renegar. A mathematical view of interior-point methods in convex optimization, volume 3. Siam, 2001.
72. P. Richtárik and M. Takác. Parallel coordinate descent methods for big data optimization. Arxiv preprint arXiv:1212.0873, 2012.
73. H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
74. R. Rockafellar. Convex Analysis. Princeton University Press, 1970.
75. M. Rudelson. Random vectors in the isotropic position. Journal of Functional Analysis, 164:60-72, 1999.
76. M. Schmidt, N. Le Roux, and F. Bach. Convergence rates of inexact proximalgradient methods for convex optimization. In Advances in neural information processing systems, pages 1458-1466, 2011.
77. B. Schölkopf and A. Smola. Learning with kernels. MIT Press, 2002.
78. S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
79. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14:567-599, 2013a.
80. S. Shalev-Shwartz and T. Zhang. Accelerated mini-batch stochastic dual coordinate ascent. In Advances in Neural Information Processing Systems (NIPS), 2013b.
81. W. Su, S. Boyd, and E. Candès. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. In Advances in Neural Information Processing Systems (NIPS), 2014.
82. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):pp. 267-288, 1996.
83. P. Tseng. On accelerated proximal gradient methods for convex-concave optimization. 2008.
84. A. Tsybakov. Optimal rates of aggregation. In Conference on Learning Theory (COLT), pages 303-313. 2003.
85. P. M. Vaidya. A new algorithm for minimizing convex functions over convex sets. In Foundations of Computer Science, 1989., 30th Annual Symposium on, pages 338-343, 1989.
86. P. M. Vaidya. A new algorithm for minimizing convex functions over convex sets. Mathematical programming, 73(3):291-341, 1996.
87. S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo. Sparse reconstruction by separable approximation. IEEE Transactions on Signal Processing, 57 (7):2479-2493, 2009.
88. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 11:2543-2596, 2010.
89. Y. Zhang and L. Xiao. Stochastic primal-dual coordinate method for regularized empirical risk minimization. Arxiv preprint arXiv:1409.3257, 2014.