Adaptive subgradient methods for online learning and stochastic optimization

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 2121-2159

  Duchi, John ^a   Hazan, Elad ^b   Singer, Yoram ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^c GOOGLE INC   (United States)

Author keywords

Adaptivity; Online learning; Stochastic convex optimization; Subgradient methods

Indexed keywords

ADAPTIVITY; DOMAIN CONSTRAINT; EFFICIENT ALGORITHM; EMPIRICAL RISK MINIMIZATION; GRADIENT-BASED LEARNING; LEARNING RATES; ONLINE LEARNING; REGULARIZATION FUNCTION; STOCHASTIC CONVEX OPTIMIZATION; STOCHASTIC OPTIMIZATIONS; SUB-GRADIENT ALGORITHM; SUBGRADIENT METHODS;

ALGORITHMS; CONVEX OPTIMIZATION; OPTIMIZATION; STOCHASTIC SYSTEMS;

E-LEARNING;

EID: 80052250414     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (44)

1. J. Abernethy, P. L. Bartlett, A. Rakhlin, and A. Tewari. Optimal strategies and minimax lower bounds for online convex games. In Proceedings of the Twenty First Annual Conference on Computational Learning Theory, 2008.
2. T. Ando. Concavity of certain maps on positive definite matrices and applications to Hadamard products. Linear Algebra and its Applications, 26:203-241, 1979.
3. A. Asuncion and D. J. Newman. UCI machine learning repository, 2007. URL http: //www. ics. uci.edu/~mlearn/MLRepository.html.
4. P. Auer and C. Gentile. Adaptive and self-confident online learning algorithms. In Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, 2000.
5. P. L. Bartlett, E. Hazan, and A. Rakhlin. Adaptive online gradient descent. In Advances in Neural Information Processing Systems 20, 2007.
6. A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31:167-175, 2003.
7. J. V. Bondar. Comments on and complements to Inequalities: Theory of Majorization and Its Applications. Linear Algebra and its Applications, 199:115-129, 1994.
8. A. Bordes, L. Bottou, and P. Gallinari. Sgd-qn: Careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 10:1737-1754, 2009.
9. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
10. P. Brucker. An O(n) algorithm for quadratic knapsack problems. Operations Research Letters, 3 (3):163-166, 1984.
11. N. Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050-2057, September 2004.
12. N. Cesa-Bianchi, A. Conconi, and C. Gentile. A second-order perceptron algorithm. SIAM Journal on Computing, 34(3):640-668, 2005. (Pubitemid 40844236)
13. N. Cesa-Bianchi, Y. Mansour, and G. Stoltz. Improved second-order bounds for prediction with expert advice. Machine Learning, 66:321-352, 2007. (Pubitemid 46360617)
14. K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive aggressive algorithms. Journal of Machine Learning Research, 7:551-585, 2006.
15. K. Crammer, M. Dredze, and F. Pereira. Exact convex confidence-weighted learning. In Advances in Neural Information Processing Systems 22, 2008.
16. K. Crammer, M. Dredze, and A. Kulesza. Adaptive regularization of weight vectors. In Advances in Neural Information Processing Systems 23, 2009.
17. C. Davis. Notions generalizing convexity for functions defined on spaces of matrices. In Proceedings of the Symposia in Pure Mathematics, volume 7, pages 187-201. American Mathematical Society, 1963.
18. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. ImageNet: a large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009.
19. J. Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10:2873-2908, 2009.
20. J. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In Proceedings of the Twenty Third Annual Conference on Computational Learning Theory, 2010.
21. R. Fletcher. A new approach to variable metric algorithms. Computer Journal, 13:317-322, 1970.
22. D. Grangier and S. Bengio. A discriminative kernel-based model to rank images from text queries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(8):1371-1384, 2008.
23. E. Hazan and S. Kale. Extracting certainty from uncertainty: regret bounded by variation in costs. In Proceedings of the Twenty First Annual Conference on Computational Learning Theory, 2008.
24. E. Hazan, A. Kalai, S. Kale, and A. Agarwal. Logarithmic regret algorithms for online convex optimization. In Proceedings of the Nineteenth Annual Conference on Computational Learning Theory, 2006.
25. J. B. Hiriart-Urruty and C. Lemaréchal. Convex Analysis and Minimization Algorithms II. Springer-Verlag, 1996.
26. R. A. Horn and C. R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
27. A. Juditsky, A. Nemirovski, and C. Tauvel. Solving variational inequalities with the stochastic mirror-prox algorithm. http://arxiv.org/abs/ 0809.0815, 2008.
28. A. Kalai and S. Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 71(3):291-307, 2003.
29. G. Lan. An optimal method for stochastic composite optimization. Mathematical Programming Series A, 2010. Online first; to appear.
30. D. Lewis, Y. Yang, T. Rose, and F. Li. RCV1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361-397, 2004.
31. H. B. McMahan and M. Streeter. Adaptive bound optimization for online convex optimization. In Proceedings of the Twenty Third Annual Conference on Computational Learning Theory, 2010.
32. A. Nedić. Subgradient Methods for Convex Minimization. PhD thesis, Massachusetts Institute of Technology, 2002.
33. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609, 2009.
34. A. S. Nemirovski and D. B. Yudin. Problem Complexity and Efficiency in Optimization. John Wiley and Sons, 1983.
35. Y. Nesterov. Smooth minimization of nonsmooth functions. Mathematical Programming, 103: 127-152, 2005.
36. Y. Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming, 120(1):221-259, 2009.
37. G. Obozinski, B. Taskar, and M. Jordan. Joint covariate selection for grouped classification. Technical Report 743, Dept. of Statistics, University of California Berkeley, 2007.
38. P. M. Pardalos and J. B. Rosen. An algorithm for a singly constrained class of quadratic programs subject to upper and lower bounds. Mathematical Programming, 46:321-328, 1990.
39. A. Rakhlin. Lecture notes on online learning. For the Statistical Machine Learning Course at University of California, Berkeley, 2009.
40. G. Salton and C. Buckley. Term weighting approaches in automatic text retrieval. Information Processing and Management, 24(5), 1988.
41. N. Z. Shor. Utilization of the operation of space dilation in the minimization of convex functions. Cybernetics and Systems Analysis, 6(1):7-15, 1972. Translated from Kibernetika.
42. P. Tseng. On accelerated proximal gradient methods for convex-concave optimization. Technical report, Department of Mathematics, University of Washington, 2008.
43. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Technical Report MSR-TR-2010-23, Microsoft Research, 2010.
44. M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning, 2003.