Sparse Q-learning with mirror descent

Uncertainty in Artificial Intelligence - Proceedings of the 28th Conference, UAI 2012

Volumn , Issue , 2012, Pages 564-573

  Mahadevan, Sridhar ^a   Liu, Bo ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

GENERATING FUNCTIONS; HIGH DIMENSIONAL SPACES; LEGENDRE TRANSFORMS; MAHALANOBIS DISTANCES; MARKOV DECISION PROCESSES; ONLINE CONVEX OPTIMIZATIONS; REINFORCEMENT LEARNING METHOD; TEMPORAL DIFFERENCES;

ARTIFICIAL INTELLIGENCE; CONVEX OPTIMIZATION; FUNCTIONS; GRADIENT METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; MIRRORS;

REINFORCEMENT LEARNING;

EID: 84886008156     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. Dimitri P. Bertsekas. Nonlinear Programming. Athena Scientific, 2nd edition, September 1999.
2. V. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, 2008.
3. L. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7(3):200-217, 1967.
4. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, Massachusetts, 1996.
5. A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 2003.
6. A. Ben-Tal, T. Margalit, and A. Nemirovski. The ordered subsets mirror descent optimization method with applications to tomography. SIAM Journal of Optimization, Jan 2001.
7. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, July 2011.
8. Claudio Gentile. The robustness of the p-norm algorithms. Mach. Learn., 53:265-299, December 2003.
9. Mohammad Ghavamzadeh, Alessandro Lazaric, Remi Munos, and Matthew Hoffman. Finite-Sample Analysis of Lasso-TD In Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML '11, 2011.
10. M. Geist and B. Scherrer. L1-penalized projected bellman residual. In Proceedings of the European Workshop on Reinforcement Learning, 2011.
11. J. Johns, C. Painter-Wakefield, and R. Parr. Linear complementarity for regularized policy evaluation and improvement. In Proceedings of Advances in Neural Information Processing Systems, 23, 2010.
12. J. Zico Kolter and Andrew Y. Ng. Regularization and feature selection in least-squares temporal difference learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 521-528, New York, NY, USA, 2009. ACM.
13. G. Konidaris, S. Osentoski, and PS Thomas. Value function approximation in reinforcement learning using the fourier basis. Computer Science Department Faculty Publication Series, page 101, 2008.
14. Jyrki Kivinen and Manfred K. Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132, 1995.
15. Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linearthreshold algorithm. In Machine Learning, pages 285-318, 1988.
16. John Langford, Lihong Li, and Tong Zhang. Sparse online learning via truncated gradient. Journal of Machine Learning Research, 10:777-801, 2009.
17. S. Mahadevan. Learning Representation and Control in Markov Decision Processes: New Frontiers. Foundations and Trends in Machine Learning, 1(4):403-565, 2009.
18. S. Mahadevan and M. Maggioni. Proto-Value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes. Journal of Machine Learning Research, 8:2169-2231, 2007.
19. Y Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming, Jan 2009.
20. A Nemirovski, A Juditsky, G Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 14(4):1574-1609, 2009.
21. A. Nemirovksi and D. Yudin. Problem Complexity and Method Efficiency in Optimization. John Wiley Press, 1983.
22. D. Precup and R. Sutton. Exponentiated gradient methods for reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 1995.
23. M. Petrik, G. Taylor, R. Parr, and S. Zilberstein. Feature selection using regularization in approximate linear programs for markov decision processes. In ICML, pages 871-878, 2010.
24. Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvri, and Eric Wiewiora. Fast gradient-descent methods for temporaldifference learning with linear function approximation. In In Proceedings of the 26th International Conference on Machine Learning, 2009.
25. Shai Shalev-shwartz and Ambuj Tewari. Stochastic methods for l1 regularized loss minimization. In International Conference on Machine Learning, pages 117-936, 2009.
26. S. Shalev-Shwartz and A. Tewari. Stochastic methods for l1 regularized loss minimization. Journal of Machine Learning Research, 2011.
27. N. Srebro, K. Sridharan, and A. Tewari. On the universality of online mirror descent. Arxiv preprint arXiv:1107.4080, Jan 2011.
28. R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44, 1988.
29. Robert Schapire Schapire and Manfred K. Warmuth. On the worst-case analysis of temporal-difference learning algorithms. In Machine Learning, pages 266-274. Morgan Kaufmann, 1994.
30. J. Si and Y.T. Wang. Online learning control by association and reinforcement. Neural Networks, IEEE Transactions on, 12(2):264-276, 2001.
31. C. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, England, 1989.
32. M Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML), 2003.