An analysis of temporal-difference learning with function approximation

IEEE Transactions on Automatic Control

Volumn 42, Issue 5, 1997, Pages 674-690

  Tsitsiklis, John N ^a,b   Van Roy, Benjamin ^b  

^a IEEE   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Dynamic programming; Function approximation; Markov chains; Neuro dynamic programming; Renforcement learning; Temporal difference learning

Indexed keywords

APPROXIMATION THEORY; CONVERGENCE OF NUMERICAL METHODS; COSTS; DYNAMIC PROGRAMMING; MARKOV PROCESSES; OPTIMAL CONTROL SYSTEMS; PARAMETER ESTIMATION; PROBABILITY; STATE SPACE METHODS;

COST TO GO FUNCTION APPROXIMATION; MARKOV CHAIN; REINFORCEMENT LEARNING; TEMPORAL DIFFERENCE LEARNING;

LEARNING ALGORITHMS;

EID: 0031143730     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/9.580874     Document Type: Article

References (25)

1. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995.
2. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learning, vol. 3, pp. 9-44, 1988.
3. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learning, vol. 8, pp. 279-292, 1992.
4. J. N. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning," Mach. Learning, vol. 16, pp. 185-202, 1994.
5. T. Jaakkola, M. I. Jordan, and S. P. Singh, "On the convergence of stochastic iterative dynamic programming algorithms," Neural Comp., vol. 6, no. 6, pp. 1185-1201, 1994.
6. P. D. Dayan and T. J. Sejnowski, "TD(λ) converges with probability 1," Mach. Learning, vol. 14, pp. 295-301, 1994.
7. L. Gurvits, L. J. Lin, and S. J. Hanson, "Incremental learning of evaluation functions for absorbing Markov chains: New methods and theorems," 1994, preprint.
8. P. D. Dayan, "The convergence of TD(λ) for general λ" Mach. Learning, vol. 8, pp. 341-362, 1992.
9. R. E. Schapire and M. K. Warmuth, "On the worst-case analysis of temporal-difference learning algorithms," Mach. Learning, vol. 22, pp. 95-122, 1996.
10. J. N. Tsitsiklis and B. Van Roy, "Feature-based methods for large scale dynamic programming," Mach. Learning, vol. 22, pp. 59-94, 1996.
11. G. J. Gordon, "Stable function approximation in dynamic programming," Carnegie Melon Univ., Tech. Rep. CMU-CS-95-103, 1995.
12. S. P. Singh, T. Jaakkola, and M. I. Jordan, "Reinforcement learning with soft state aggregation," in Advances in Neural Information Processing Systems, vol. 7, G. Tesauro, D. S. Touretzky, and T. K. Leen, Eds Cambridge, MA: MIT Press, 1995.
13. L. C. Baird, " Residual algorithms Reinforcement learning with function approximation," in Machine Learning: Proceedings 12th Int. Conf., July 9-12, Prieditis and Russell, Eds. San Francisco, CA: Morgan Kaufman, 1995.
14. J. A. Boyan and A. W. Moore, "Generalization in reinforcement learning: Safely approximating the value function," in Advances in Neural Information Processing Systems, vol. 7. MIT Press, 1995.
15. R. S. Sutton, "On the virtues of linear learning and trajectory distributions," in Proc. Wkshp. Value Function Approximation, Mach. Learning Conf., Carnegie Mellon Univ., Tech. Rep. CMU-CS-95-206, 1995.
16. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
17. L. Gurvits, 1996, private communication.
18. F. Pineda, "Mean-field analysis for batched TD(λ)," 1996, preprint.
19. A. Benveniste, M. Métivier, and P. Prioret, Adaptive Algorithms and Stochastic Approximations. Berlin: Springer-Verlag, 1990.
20. J. Walrand, An Introduction to Queueing Networks. Englewood Cliffs, NJ: Prentice Hall, 1988.
21. G. D. Stamoulis and J. N. Tsitsiklis, "On the settling time of the congested GI/G/1 queue," Adv. Appl. Probability, vol. 22, pp. 929-956, 1990.
22. P. Konstantopoulos and F. Baccelli, "On the cut-off phenomenon in some queueing systems," J. Appl. Probability, vol. 28, pp. 683-694, 1991.
23. D. P. Bertsekas, "A counterexample to temporal-difference learning," Neural Comp., vol. 7, pp. 270-279, 1994.
24. S. P. Singh and R. S. Sutton, "Reinforcement learning with replacing eligibility traces," Mach. Learning, vol. 22, pp. 123-158, 1996.
25. R. S. Sutton, "Generalization in reinforcement learning: Successful examples using sparse coarse coding," in Advances in Neural Information Processing Systems, vol. 8, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996.