Adaptive subgradient methods for online learning and stochastic optimization

COLT 2010 - The 23rd Conference on Learning Theory

Volumn , Issue , 2010, Pages 257-269

  Duchi, John ^a   Hazan, Elad ^b   Singer, Yoram ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b IBM ALMADEN RESEARCH CENTER   (United States)

^c GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

GRADIENT-BASED LEARNING; LEARNING RATES; ONLINE LEARNING; STOCHASTIC OPTIMIZATIONS; SUB-GRADIENT METHODS; TEXT CLASSIFICATION;

CLASSIFICATION (OF INFORMATION); OPTIMIZATION; TEXT PROCESSING;

E-LEARNING;

EID: 80052423377     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. J. Abernethy, P. Bartlett, A. Rakhlin, and A. Tewari. Optimal strategies and minimax lower bounds for online convex games. In Proceedings of the Twenty First Annual Conference on Computational Learning Theory, 2008.
2. P. Auer, N. Cesa-Bianchi, and C. Gentile. Adaptive and self-confident on-line learning algorithms. Journal of Computer and System Sciences, 64(1):48-75, 2002.
3. P. L. Bartlett, E. Hazan, and A. Rakhlin. Adaptive online gradient descent. InAdvances in Neural Information Processing Systems 20, 2007.
4. A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31:167-175, 2003.
5. A. Bordes, L. Bottou, and P. Gallinari. Sgd-qn: Careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 10:1737-1754, 2009.
6. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
7. N. Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050-2057, September 2004.
8. N. Cesa-Bianchi, A. Conconi, and C. Gentile. A second-order perceptron algorithm. SIAM Journal on Computing, 34(3):640-668, 2005.
9. K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. Online passive aggressive algorithms. Journal of Machine Learning Research, 7:551-585, 2006.
10. K. Crammer, M. Dredze, and A. Kulesza. Adaptive regularization of weight vectors. In Advances in Neural Information Processing Systems 23, 2009.
11. C. Davis. Notions generalizing convexity for functions defined on spaces of matrices. In Proceedings of the Symposia in Pure Mathematics, Volume 7, pages 187-201. American Mathematical Society, 1963.
12. J. Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10:2873-2908, 2009.
13. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Technical Report 2010-24, UC Berkeley Electrical Engineering and Computer Science, 2010a. URL cs.berkeley.edu/~jduchi/projects/DuchiHaSi10.pdf.
14. J. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In Proceedings of the Twenty Third Annual Conference on Computational Learning Theory, 2010b.
15. R. Fletcher. A new approach to variable metric algorithms. Computer Journal, 13:317-322, 1970.
16. E. Hazan, A. Kalai, S. Kale, and A. Agarwal. Logarithmic regret algorithms for online convex optimization. In Proceedings of the Nineteenth Annual Conference on Computational Learning Theory, 2006.
17. A. Kalai and S. Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 71(3):291-307, 2003.
18. Y. Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming, 120(1): 221-259, 2009.
19. G. Salton and C. Buckley. Term weighting approaches in automatic text retrieval. Information Processing and Management, 24(5), 1988.
20. S. Shalev-Shwartz and Y. Singer. Logarithmic regret algorithms for strongly convex repeated games. Technical report, The Hebrew University, 2007. URL http://www.cs.huji.ac.il/~shais.
21. N. Z. Shor. Utilization of the operation of space dilation in the minimization of convex functions. Cybernetics and Systems Analysis, 6(1):7-15, 1972. Translated from Kibernetika.
22. P. Tseng. On accelerated proximal gradient methods for convex-concave optimization. Technical report, Department of Mathematics, University of Washington, 2008.
23. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. In Advances in Neural Information Processing Systems 23, 2009.
24. M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the Twentieth International Conference on Machine Learning, 2003.