On the convergence of temporal-difference learning with linear function approximation

Machine Learning

Volumn 42, Issue 3, 2001, Pages 241-267

  Tadić, Vladislav ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; ASYMPTOTIC STABILITY; CONVERGENCE OF NUMERICAL METHODS; DYNAMIC PROGRAMMING; ERROR ANALYSIS; FUNCTION EVALUATION; MARKOV PROCESSES; STATE SPACE METHODS;

LINEAR FUNCTION APPROXIMATION; NEURODYNAMIC PROGRAMMING; POSITIVE HARRIS RECURRENCE; REINFORCEMENT LEARNING; TEMPORAL DIFFERENCE LEARNING;

LEARNING ALGORITHMS;

EID: 0035283402     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1007609817671     Document Type: Article

References (21)

1. Asmussen, S. (1987). Applied Probability and Queues. New York: Wiley.
2. Benveniste, A., Metivier, M., Priouret, P. (1990). Adaptive Algorithms and Stochastic Approximation. Berlin: Springer Verlag.
3. Bertsekas, D. P. (1976). Dynamic Programming and Optimal Control. New York: Academic Press.
4. Bertsekas, D. P. & Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Belmont, MA: Athena Scientific.
5. Clark, D. S. (1984). Necessary and sufficient conditions for the Robbins-Monro method. Stochastic Processes and their Applications, 17, 359-367.
6. Chen, H. F. & Guo, L. (1991). Identification and Stochastic Adaptive Control. Basel: Birkhäuser Verlag.
7. Dai, J. G. (1995). On positive Harris recurrence of multiclass queueing networks: A unified approach via fluid limit models. Annals of Applied Probability, 5, 49-77.
8. Dayan, P. D. (1992). The convergence of TD(λ) for general λ. Machine Learning, 8, 341-362.
9. Dayan, P. D. & Sejnowski, T. J. (1994). TD(λ) converges with probability 1. Machine Learning, 14, 295-301.
10. Jaakola, T., Jordan, M. I., & Singh, S. P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6, 1185-1201.
11. Kulkarni, S. R. & Horn, C. S. (1996). An alternative proof for convergence of stochastic approximation algorithms. IEEE Transactions of Automatic Control, 41, 419-424.
12. Kumar, P. R. & Varaiya, P. (1986). Stochastic Systems: Estimation, Identification and Adaptive Control. Englewood Cliffs, NJ: Prentice Hall.
13. Kushner, H. J. & Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Berlin: Springer Verlag.
14. Ljung, L., Pflug, G., & Walk, H. (1992). Stochastic Approximation and Optimization of Random Systems. Basel: Birkhäuser Verlag.
15. Meyn, S. P. & Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Berlin: Springer-Verlag.
16. Solo, V. & Kong, X. (1995). Adaptive Signal Processing Algorithms: Stability and Performance. Englewood Cliffs, NJ: Prentice Hall.
17. Sutton, R. S. (1988). Learning to predict by the methods of temporal-differences. Machine Learning, 3, 9-44.
18. Sutton, R. S. & Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
19. Tadić, V. (1997). Convergence of stochastic approximation under general noise and stability conditions. In Proceedings of the 36 IEEE Conference on Decision and Control.
20. Tsitsiklis, J. N. & Van Roy, B. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42, 674-690.
21. Wang, I.-J., Chong, E. K. P., & Kulkarni, S. R. (1996). Equivalent and sufficient conditions on noise sequences for stochastic approximation algorithms. Advances in Applied Probability, 28, 784-801.