Dual averaging methods for regularized stochastic learning and online optimization

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 2543-2596

  Xiao, Lin ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

1 regularization; Accelerated gradient methods; Dual averaging methods; Online optimization; Stochastic learning; Structural convex optimization

Indexed keywords

ACCELERATED GRADIENT METHODS; AVERAGING METHOD; ONLINE OPTIMIZATION; STOCHASTIC LEARNING; STRUCTURAL CONVEX OPTIMIZATION;

GRADIENT METHODS; STOCHASTIC SYSTEMS; STRUCTURAL OPTIMIZATION;

CONVEX OPTIMIZATION;

EID: 78649396336     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (68)

1. 1-regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning (ICML), pages 33-40, Corvallis, OR, USA, 2007.
2. A. Auslender and M. Teboulle. Interior gradient and proximal methods for convex and conic optimization. SIAM Journal on Optimization, 16:697-725, 2006.
3. K. Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, 19:357-367, 1967.
4. S. Balakrishnan and D. Madigan. Algorithms for sparse linear classifiers in the massive data setting. Journal of Machine Learning Research, 9:313-337, 2008.
5. P. Bartlett, E. Hazan, and A. Rakhlin. Adaptive online gradient descent. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 65-72. MIT Press, Cambridge, MA, 2008.
6. A. Beck and M. Teboulle. A fast iterative shrinkage-threshold algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2 (1):183-202, 2009.
7. L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 161-168. MIT Press, Cambridge, MA, 2008.
8. L. Bottou and Y. LeCun. Large scale online learning. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16, pages 217-224. MIT Press, Cambridge, MA, 2004.
9. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
10. D. M. Bradley and J. A. Bagnell. Differentiable sparse coding. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 113-120. MIT Press, Cambridge, MA, USA, 2009.
11. K. Bredies and D. A. Lorenz. Iterated hard shrinkage for minimization problems with sparsity constraints. SIAM Journal on Scientific Computing, 30 (2):657-683, 2008.
12. P. Carbonetto, M. Schmidt, and N. De Freitas. An interior-point stochastic approximation method and an l1-regularized delta rule. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 233-240. MIT Press, 2009.
13. N. Cesa-Bianchi and G. Lugosi. Predictioin, Learning, and Games. Cambridge University Press, 2006.
14. G. Chen and M. Teboulle. Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM Journal on Optimization, 3 (3):538-543, August 1993.
15. G. H.-G. Chen and R. T. Rockafellar. Convergence rates in forward-backward splitting. SIAM Journal on Optimization, 7 (2):421-444, 1997.
16. S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20:33-61, 1998.
17. J. Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10:2873-2898, 2009.
18. 1-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning (ICML), pages 272-279, 2008.
19. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. To appear in Journal of Machine Learning Research, 2010.
20. M. C. Ferris and T. S. Munson. Interior-point methods for massive support vector machines. SIAM Journal on Optimization, 13 (3):783-804, 2003.
21. M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE Journal on Selected Topics in Signal Processing, 1 (4):586-597, 2007.
22. D. A. Freedman. On tail probabilities for martingales. The Annals of Probability, 3 (1):100-118, 1975.
23. C. Gentile. The robustness of the p-norm algorithms. Machine Learning, 53:265-299, 2003.
24. R. Goebel and R. T. Rockafellar. Local strong convexity and local Lipschitz continuity of the gradient of convex functions. Journal of Convex Analysis, 15 (2):263-270, 2008.
25. E. Hazan, A. Kalai, S. Kale, and A. Agarwal. Logarithmic regret algorithms for online convex optimization. In Proceedings of 19th Annual Conference on Computational Learning Theory (COLT), pages 499-513, Pittsburgh, PA, USA, 2006.
26. J.-B. Hiriart-Urruty and C. Lemaréchal. Fundamentals of Convex Analysis. Springer, 2001.
27. C. Hu, J. T. Kwok, and W. Pan. Accelerated gradient methods for stochastic optimization and online learning. In Y Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 781-789. 2009.
28. B. Johansson, M. Rabi, and M. Johansson. A randomized incremental subgradient method for distributed optimization in networked systems. SIAM Journal on Optimization, 20 (3):1157-1170, 2009.
29. A. Juditsky and A. Nemirovski. Large deviations of vector-valued martingales in 2-smooth normed spaces. Manuscript submitted to The Annals of Probability, 2008. arXiv:0809.0813vl.
30. A. Juditsky, A. Nazin, A. Tsybakov, and N. Vayatis. Recursive aggregation of estimators by mirror descent algorithm with averaging. Problems of Information Transmission, 41 (4):368-384, 2005.
31. S. M. Kakade and A. Tewari. On the generalization ability of online strongly convex programming algorithms. In D. Koller, D. Schuurmans, Y Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 801-808. MIT Press, Cambridge, MA, USA, 2009.
32. J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. The Annuals of Mathematical Statistics, 23:462-466, 1952.
33. J. Kivinen and M. K. Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132 (1):1-63, 1997.
34. 1-regularized logistic regression. Journal of Machine Learning Research, 8:1519-1555, 2007. (Pubitemid 47210273)
35. G. Lan. An optimal method for stochastic composite optimization. To appear in Mathematical Programming, 2010.
36. G. Lan, A. Nemirovski, and A. Shapiro. Validation analysis of robust stochastic approximation methods. Submitted to Mathematical Programming, 2008.
37. G. Lan, Z. Lu, and R. D. C. Monteiro. Primal-dual first-order methods with O (1/ε) iterationcomplexity for cone programming. Mathematical Programming, February 2009. Published online, DOI 10.1007/sl0107-008-0261-6.
38. J. Langford, L. Li, and T. Zhang. Sparse online learning via truncated gradient. Journal of Machine Learning Research, 10:777-801, 2009.
39. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 (11):2278-2324, 1998. Dataset available at http://yann. lecun.com/exdb/mnist.
40. P.-L. Lions and B. Mercier. Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis, 16:964-979, 1979.
41. Z. Lu. Primal-dual first-order methods for a class of cone programming with applications to the Dantzig selector. Submitted manuscript, 2009.
42. A. Nedic and D. P. Bertsekas. Incremental subgradient methods for nondifferentiable optimization. SIAM Journal on Optimization, 12 (1):109-138, 2001.
43. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19 (4):1574-1609, 2009.
44. A. Nemirovsky and D. Yudin. Problem Complexity and Method Efficiency in Optimization. J. Wiley & Sons, New York, 1983.
45. 2). Soviet Math. Doklady, 27 (2):372-376, 1983. Translated from Russian by A. Rosa.
46. Yu. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer, Boston, 2004.
47. Yu. Nesterov. Smooth minimization of nonsmooth functions. Mathematical Programming, 103:127-152, 2005.
48. Yu. Nesterov. Gradient methods for minimizing composite objective function. Technical Report 2007/76, Catholic University of Louvain, Center for Operations Research and Econometrics, 2007.
49. Yu. Nesterov. How to advance in structural convex optimization. OPTIMA: Mathematical Programming Society Newsletter, 78:2-5, November 2008.
50. Yu. Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming, 120 (1):221-259, 2009. Appeared early as CORE discussion paper 2005/67, Catholic University of Louvain, Center for Operations Research and Econometrics.
51. Yu. Nesterov and J.-Ph. Vial. Confidence level solutions for stochastic programming. Automatica, 44 (6):1559-1568, 2008.
52. B. T. Polyak and A. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30:838-855, 1992.
53. S. Sundhar Ram, A. Nedic, and V. V. Veeravalli. Incremental stochastic subgradient algorithms for convex optimization. SIAM Journal on Optimization, 20 (2):691-717, 2009.
54. H. Robbins and S. Monro. A stochastic approximation method. The Annuals of Mathematical Statistics, 22:400-407, 1951.
55. R. T. Rockafellar. Convex Analysis. Princeton University Press, 1970.
56. R. T. Rockafellar and R. J. B. Wets. On the interchange of subdifferentiation and conditional expectation for convex functionals. Stochastics An International Journal of Probability and Stochastic Processes, 7 (3):173-182, 1982.
57. S. Shalev-Shwartz and S. M. Kakade. Mind the duality gap: Logarithmic regret algorithms for online optimization. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 1457-1464. MIT Press, 2009.
58. S. Shalev-Shwartz and Y. Singer. Convex repeated games and Fenchel duality. In B. Schölkopf, J. Platt, and T. Hofmann, editors, Advances in Neural Information Processing Systems, volume 19, pages 1265-1272. MIT Press, 2006.
59. 1 regularized loss minimization. In Proceedings of the 26th International Conference on Machine Learning (ICML), pages 929-936, Montreal, Canada, 2009.
60. S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated sub-gradient solver for SVM. In Proceedings of the 24th International Conference on Machine Learning (ICML), pages 807-814, 2007.
61. R. Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, 58:267-288, 1996.
62. P. Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. SIAM Journal on Optimization, 8 (2):506-531, 1998.
63. P. Tseng. A modified forward-backward splitting method for maximal monotone mappings. SIAM Journal on Control and Optimization, 38 (2):431-446, 2000.
64. P. Tseng. On accelerated proximal gradient methods for convex-concave optimization. Manuscript submitted to SIAM Journal on Optimization, 2008.
65. P. Tseng and D. P. Bertsekas. On the convergence of exponential multiplier method for convex programming. Mathematical Programming, 60:1-19, 1993.
66. S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo. Sparse reconstruction by separable approximation. IEEE Transactions on Signal Processing, 57 (7):2479-2493, 2009.
67. T. Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the 21st International Conference on Machine Learning (ICML), pages 116-123, Banff, Alberta, Canada, 2004.
68. M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML), pages 928-936, Washington DC, 2003.