A convergent O(n) algorithm for off-policy temporal-difference learning with linear function approximation

Advances in Neural Information Processing Systems 21 - Proceedings of the 2008 Conference

Volumn , Issue , 2009, Pages 1609-1616

  Sutton, Richard S ^a   Szepesvári, Csaba ^a   Maei, Hamid Reza ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; IMPORTANCE SAMPLING; LEAST SQUARES APPROXIMATIONS; MARKOV PROCESSES; STOCHASTIC SYSTEMS;

BEHAVIOR POLICY; FINITE MARKOV DECISION PROCESS; FUNCTIONS APPROXIMATIONS; LINEAR FUNCTIONS; POLICY EVALUATION; PROCESS BEHAVIOR; STOCHASTIC APPROXIMATIONS; STOCHASTIC GRADIENT DESCENT; TEMPORAL DIFFERENCE LEARNING; TEMPORAL-DIFFERENCE ALGORITHM;

LEARNING ALGORITHMS;

EID: 77956513316     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 30-37. Morgan Kaufmann.
2. Baxter, J., Tridgell, A., Weaver, L. (1998). Experiments in parameter learning using temporal differences. International Computer Chess Association Journal, 21, 84-99.
3. Bertsekas, D. P., Tsitsiklis. J. (1996). Neuro-Dynamic Programming. Athena Scientific, 1996.
4. Borkar, V. S. and Meyn, S. P. (2000). The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control And Optimization, 38(2):447-469. (Pubitemid 30594546)
5. Boyan, J. (2002). Technical update: Least-squares temporal difference learning. Machine Learning, 49:233-246. (Pubitemid 34325688)
6. Bradtke, S., Barto, A. G. (1996). Linear least-squares algorithms for temporal difference learning. Machine Learning, 22:33-57. (Pubitemid 126724362)
7. Dayan, P. (1992). The convergence of TD(λ) for general λ. Machine Learning, 8:341-362.
8. Geramifard, A., Bowling, M., Sutton, R. S. (2006). Incremental least-square temporal difference learning. Proceedings of the National Conference on Artificial Intelligence, pp. 356-361. (Pubitemid 44705310)
9. Gordon, G. J. (1995). Stable function approximation in dynamic programming. Proceedings of the Twelfth International Conference on Machine Learning, pp. 261-268. Morgan Kaufmann, San Francisco.
10. Lagoudakis, M., Parr, R. (2003). Least squares policy iteration. Journal of Machine Learning Research, 4:1107-1149.
11. Peters, J., Vijayakumar, S. and Schaal, S. (2005). Natural Actor-Critic. Proceedings of the 16th European Conference on Machine Learning, pp. 280-291.
12. Precup, D., Sutton, R. S. and Dasgupta, S. (2001). Off-policy temporal-difference learning with function approximation. Proceedings of the 18th International Conference on Machine Learning, pp. 417-424.
13. Precup, D., Sutton, R. S., Paduraru, C., Koop, A., Singh, S. (2006). Off-policy Learning with Recognizers. Advances in Neural Information Processing Systems 18.
14. Precup, D., Sutton, R. S., Singh, S. (2000). Eligibility traces for off-policy policy evaluation. Proceedings of the 17th International Conference on Machine Learning, pp. 759-766. Morgan Kaufmann.
15. Schaeffer, J., Hlynka, M., Jussila, V. (2001). Temporal difference learning applied to a high-performance gameplaying program. Proceedings of the International Joint Conference on Artificial Intelligence, pp. 529-534.
16. Silver, D., Sutton, R. S., Müller, M. (2007). Reinforcement learning of local shape in the game of Go. Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 1053-1058.
17. Sturtevant, N. R., White, A. M. (2006). Feature construction for reinforcement learning in hearts. In Proceedings of the 5th International Conference on Computers and Games.
18. Sutton, R. S. (1988). Learning to predict by the method of temporal differences. Machine Learning, 3:9-44.
19. Sutton, R. S., Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press.
20. Sutton, R.S., Precup D. and Singh, S (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112:181-211. (Pubitemid 32079890)
21. Sutton, R. S., Rafols, E.J., and Koop, A. 2006. Temporal abstraction in temporal-difference networks. Advances in Neural Information Processing Systems 18.
22. Tadic, V. (2001). On the convergence of temporal-difference learning with linear function approximation. In Machine Learning 42:241-267 (Pubitemid 32188797)
23. Tsitsiklis, J. N., and Van Roy, B. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42:674-690. (Pubitemid 127760263)
24. Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. Ph.D. thesis, Cambridge University.