Fast gradient-descent methods for temporal-difference learning with linear function approximation

ACM International Conference Proceeding Series

Volumn 382, Issue , 2009, Pages

  Sutton, Richard S ^a   Maei, Hamid Reza ^a   Precup, Doina ^b   Bhatnagar, Shalabh ^c   Silver, David ^a   Szepesv́ari, Csaba ^a   Wiewiora, Eric ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

^b MCGILL UNIVERSITY   (Canada)

^c INDIAN INSTITUTE OF SCIENCE   (India)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL REQUIREMENTS; COMPUTER GO; CONVERGENCE RATES; FUNCTION APPROXIMATORS; GRADIENT-DESCENT; LEARNING RATES; LINEAR FUNCTIONS; OBJECTIVE FUNCTIONS; POLICY PROBLEM; TEMPORAL DIFFERENCE LEARNING; TEMPORAL DIFFERENCES; TEST PROBLEM;

APPROXIMATION ALGORITHMS; COMPUTER APPLICATIONS; CONVERGENCE OF NUMERICAL METHODS; EDUCATION; ROBOT LEARNING;

LEARNING ALGORITHMS;

EID: 70049090437     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1553374.1553501     Document Type: Conference Paper

References (18)

1. Antos, A., Szepesv́ari, Cs., Munos, R. (2008). Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path.Machine Learning 71:89-129.
2. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the 12th Int. Conf. on Machine Learning, pp. 30-37.
3. Baird, L. C. (1999). Reinforcement Learning Through Gradient Descent. PhD thesis, Carnegie-Mellon University.
4. Barnard, E. (1993). Temporal-difference methods and Markov models. IEEE Transactions on Systems, Man, and Cybernetics 23(2):357-365.
5. Borkar, V. S. (1997). Stochastic approximation with two timescales. Systems and Control Letters 29:291-294.
6. Borkar, V. S. and Meyn, S. P. (2000). The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control And Optimization 38(2):447-469.
7. Boyan, J. (2002). Technical update: Least-squares temporal difference learning. Machine Learning 49:233-246.
8. Bradtke, S., Barto, A. G. (1996). Linear least-squares algorithms for temporal difference learning. Machine Learning 22:33-57.
9. Dayan, P. (1992). The convergence of TD(λ) for general λ. Machine Learning 8:341-362.
10. Geramifard, A., Bowling, M., Sutton, R. S. (2006). Incremental least-square temporal difference learning. Proceedings AAAI, pp. 356-361.
11. Hirsch, M. W. (1989). Convergent activation dynamics in continuous time networks. Neural Networks 2:331-349.
12. Precup, D., Sutton, R. S. and Dasgupta, S. (2001). Offpolicy temporal-difference learning with function approximation. Proceedings of the 18th International Conference on Machine Learning, pp. 417-424.
13. Precup, D., Sutton, R. S., Paduraru, C., Koop, A., Singh, S. (2006). Off-policy learning with recognizers. Advances in Neural Information Processing Systems 18.
14. Silver, D., Sutton, R. S., Müller, M. (2007). Reinforcement learning of local shape in the game of Go. Proceedings of the 20th IJCAI, pp. 1053-1058.
15. Sturtevant, N. R., White, A. M. (2006). Feature construction for reinforcement learning in hearts. In Proceedings of the 5th International Conf. on Computers and Games.
16. Sutton, R. S. (1988). Learning to predict by the method of temporal differences. Machine Learning 3:9-44.
17. Sutton, R. S., Precup D., and Singh, S (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112:181-211.
18. Sutton, R. S., Szepesv́ari, Cs., Maei, H. R. (2009). A convergent O(n) algorithm for off-policy temporaldifference learning with linear function approximation. Advances in Neural Information Processing Systems 21.