Improved temporal difference methods with linear function approximation

Handbook of Learning and Approximate Dynamic Programming

Volumn , Issue , 2004, Pages 235-259

  Bertsekas, Dimitri P ^a   Nedich, Angelia ^b   Borkar, Vivek S ^c  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b BAE SYSTEMS   (United Kingdom)

^c TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

Author keywords

Argon; Convergence; Eigenvalues and eigenfunctions; Function approximation; Markov processes; Trajectory; Vectors

Indexed keywords

APPROXIMATION ALGORITHMS; ARGON; COST FUNCTIONS; DYNAMIC PROGRAMMING; EIGENVALUES AND EIGENFUNCTIONS; MARKOV PROCESSES; TRAJECTORIES; VECTORS;

CONVERGENCE; DISCOUNTED COSTS; FUNCTION APPROXIMATION; INFINITE HORIZONS; LINEAR COST FUNCTIONS; LINEAR FUNCTIONS; TEMPORAL DIFFERENCE METHODS; TEMPORAL-DIFFERENCE ALGORITHM;

ITERATIVE METHODS;

EID: 85036496976     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1109/9780470544785.ch9     Document Type: Chapter

References (19)

1. A. Benveniste, M. Metivier, and P. Priouret, Adaptive Algorithms and Stochastic Approximations, Springer-Verlag, New York, 1990.
2. D. P. Bertsekas and S. Ioffe, Temporal differences-based policy iteration and applications in neuro-dynamic programming, Lab. for Info, and Decision Systems Report LIDS-P-2349, MIT, Cambridge, MA, 1996.
3. D. P. Bertsekas, Dynamic Programming and Optimal Control, 2nd Edition, Athena Scientific, Belmont, MA, 2001.
4. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, Belmont, MA, 1996.
5. D. P. Bertsekas and J. N. Tsitsiklis, Gradient convergence in gradient methods with errors, SIAM Journal Optimization, vol. 10, pp. 627-642, 2000.
6. J. A. Boyan, Technical update: Least-squares temporal difference learning, Machine Learning, vol. 49, pp. 1-15,2002.
7. S. J. Bradtke and A. G. Barto, Linear least-squares algorithms for temporal difference learning, Machine Learning, vol. 22, pp. 33-57, 1996.
8. P. D. Dayan, The convergence of TD(A) for general A, Machine Learning, vol. 8, pp. 341-362,1992.
9. D. P. de Farias and B. Van Roy, On the existence of fixed points for approximate value iteration and temporal-difference learning, Journal of Optimization Theory and Applications, vol. 105,2000.
10. L. Gurvits, L. J. Lin, and S. J. Hanson, Incremental learning of evaluation functions for absorbing Markov chains: New methods and theorems, Preprint, 1994.
11. V. R. Konda and J. N. Tsitsiklis, The asymptotic mean squared error of temporal difference learning, Unpublished Report, Lab. for Information and Decision Systems, MIT, Cambridge, MA, 2003.
12. V. R. Konda, Actor-Critic Algorithms, Ph.D. Thesis, Dept, of Electrical Engineering and Computer Science, MIT, Cambridge, MA, 2002.
13. A. Nedic and D. P. Bertsekas, Least squares policy evaluation algorithms with linear function approximation, Discrete Event Dynamic Systems: Theory and Applications, vol. 13, pp. 79-110, 2003.
14. F. Pineda, Mean-field analysis for batched TD(A), Neural Computation, pp.1403-1419,1997.
15. M. L. Puterman, Markov Decision Processes, Wiley, New York, 1994.
16. R. S. Sutton and A. G. Barto, Reinforcement Learning, MIT Press, Cambridge, MA, 1998.
17. R. S. Sutton, Learning to predict by the methods of temporal differences, Machine Learning, vol. 3, pp. 9-44,1988.
18. B. Van Roy, Learning and Value Function Approximation in Complex Decision Processes, Ph.D. Thesis, MIT, Cambridge, MA, 1998.
19. J. N. Tsitsiklis and B. Van Roy, An analysis of temporal-difference learning with function approximation, IEEE Trans, on Automatic Control, vol. 42, pp. 674-690,1997.