Regularized off-policy TD-learning

Advances in Neural Information Processing Systems

Volumn 2, Issue , 2012, Pages 836-844

  Liu, Bo ^a   Mahadevan, Sridhar ^a   Liu, Ji ^b  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

^b University of Wisconsin Madison   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMIC FRAMEWORK; COMPUTATIONAL COSTS; CONVEX REGULARIZATIONS; LOW COMPUTATIONAL COMPLEXITY; NON-SMOOTH CONVEX OPTIMIZATIONS; SADDLE-POINT FORMULATIONS; SPARSE REPRESENTATION; THEORETICAL AND EXPERIMENTAL;

CONVEX OPTIMIZATION;

ALGORITHMS;

EID: 84877748309     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. A. Ben-Tal and A. Nemirovski. Non-Euclidean restricted memory level method for large-scale convex optimization. Mathematical Programming, 102(3):407-456, 2005.
2. T. Degris, M. White, and R. S. Sutton. Linear off-policy actor-critic. In International Conference on Machine Learning, 2012.
3. M. Geist, B. Scherrer, A. Lazaric, and M. Ghavamzadeh. A Dantzig Selector Approach to Temporal Difference Learning. In International Conference on Machine Learning, 2012.
4. M. Ghavamzadeh, A. Lazaric, R. Munos, and M. Hoffman. Finite-Sample Analysis of Lasso- TD. In Proceedings of the 28th International Conference on Machine Learning, 2011.
5. J. Johns, C. Painter-Wakefield, and R. Parr. Linear complementarity for regularized policy evaluation and improvement. In Proceedings of the International Conference on Neural Information Processing Systems, 2010.
6. A. Juditsky and A. Nemirovski. Optimization for Machine Learning, chapter First-Order Methods for Nonsmooth Convex Large-Scale Optimization. MIT Press, 2011.
7. J. Zico Kolter and A. Y. Ng. Regularization and feature selection in least-squares temporal difference learning. In Proceedings of 27 th International Conference on Machine Learning, 2009.
8. G. Konidaris, S. Osentoski, and PS Thomas. Value function approximation in reinforcement learning using the fourier basis. In Proceedings of the Twenty-Fifth Conference on Artificial Intelligence, 2011.
9. G. M. Korpelevich. The extragradient method for finding saddle points and other problems. 1976.
10. H.R. Maei and R.S. Sutton. GQ (σ): A general gradient algorithm for temporal-difference prediction learning with eligibility traces. In Proceedings of the Third Conference on Artificial General Intelligence, pages 91-96, 2010.
11. S. Mahadevan and B. Liu. Sparse Q-learning with Mirror Descent. In Proceedings of the Conference on Uncertainty in AI, 2012.
12. A. Nedić and A. Ozdaglar. Subgradient methods for saddle-point problems. Journal of optimization theory and applications, 142(1):205-228, 2009.
13. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19:1574-1609, 2009.
14. Y. Nesterov. Gradient methods for minimizing composite objective function. In www.optimization-online.org, 2007.
15. C. Painter-Wakefield and R. Parr. Greedy algorithms for sparse reinforcement learning. In International Conference on Machine Learning, 2012.
16. C. Painter-Wakefield and R. Parr. L1 regularized linear temporal difference learning. Technical report, Duke CS Technical Report TR-2012-01, 2012.
17. M. Petrik, G. Taylor, R. Parr, and S. Zilberstein. Feature selection using regularization in approximate linear programs for Markov decision processes. In Proceedings of the International Conference on Machine learning (ICML), 2010.
18. J. Si and Y. Wang. Online learning control by association and reinforcement. IEEE Transactions on Neural Networks, 12:264-276, 2001.
19. R. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
20. R.S. Sutton, H.R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In International Conference on Machine Learning, pages 993-1000, 2009.
21. J. Zico Kolter. The Fixed Points of Off-Policy TD. In Advances in Neural Information Processing Systems 24, pages 2169-2177, 2011.