Convergence results for some temporal difference methods based on least squares

IEEE Transactions on Automatic Control

Volumn 54, Issue 7, 2009, Pages 1515-1531

  Yu, Huizhen ^a,b   Bertsekas, Dimitri P ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Approximation methods; Convergence of numerical methods; Dynamic programming; Markov processes

Indexed keywords

APPROXIMATION METHODS; AVERAGE COST; CONVERGENCE RATES; CONVERGENCE RESULTS; DISCOUNTED COSTS; FINITE-STATE; INFINITE HORIZONS; LEAST SQUARE; LINEAR FUNCTIONS; MARKOV DECISION PROCESS; POLICY EVALUATION; RATE OF CONVERGENCE; STATIONARY POLICY; STEPSIZE; TEMPORAL DIFFERENCES;

APPROXIMATION THEORY; COSTS; DYNAMIC PROGRAMMING; MARKOV PROCESSES; NUMBER THEORY; SYSTEMS ENGINEERING;

CONVERGENCE OF NUMERICAL METHODS;

EID: 67949109470     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAC.2009.2022097     Document Type: Article

References (30)

1. V. R. Konda and J. N. Tsitsiklis, "Actor-critic algorithms," SIAM J. Control Optim., vol. 42, no. 4, pp. 1143-1166, 2003.
2. R. S. Sutton, "Learning to predict by the methods of temporal differences," Machine Learning, vol. 3, pp. 9-44, 1988.
3. P. D. Dayan, "The convergence of TD(λ) for general λ," Machine Learning, vol. 8, pp. 341-362, 1992.
4. L. Gurvits, L. J. Lin, and S. J. Hanson, Incremental Learning of Evaluation Functions for Absorbing Markov Chains: NewMethods and Theorems Siemans Corporate Research, Princeton, NJ, 1994.
5. F. Pineda, "Mean-field analysis for batched TD(λ)," Neural Computation, pp. 1403-1419, 1997.
6. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Automat. Control vol. 42, no. 5, pp. 674-690, May 1997.
7. J. N. Tsitsiklis and B. Van Roy, "Average cost temporal-difference learning," Automatica, vol. 35, no. 11, pp. 1799-1808, 1999.
8. J. N. Tsitsiklis and B. Van Roy, "On average versus discounted reward temporal-difference learning," Machine Learning, vol. 49, pp. 179-191, 2002.
9. P. Marbach and J. N. Tsitsiklis, "Simulation-based optimization of Markov reward processes," IEEE Trans. Automat. Control, vol. 46, no. 2, pp. 191-209, Feb. 2001.
10. S. J. Bradtke and A. G. Barto, "Linear least-squares algorithms for temporal difference learning," Machine Learning, vol. 22, no. 2, pp. 33-57, 1996.
11. J. A. Boyan, "Least-squares temporal difference learning," in Proc. 16th Int. Conf. Machine Learning, 1999, pp. 49-56.
12. D. P. Bertsekas and S. Ioffe, Temporal Differences-Based Policy Iteration and Applications in Neuro-Dynamic Programming MIT, Cambridge, MA, LIDS Tech. Rep. LIDS-P-2349, 1996.
13. V. R. Konda, "Actor-critic algorithms," Ph.D. dissertation, Dept. Comput. Sci. Elect. Eng., MIT, Cambridge, MA, 2002.
14. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
15. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Machine Learning Res., vol. 4, pp. 1107-1149, 2003.
16. A. Nedić and D. P. Bertsekas, "Least squares policy evaluation algorithms with linear function approximation," Discrete Event Dynamic Systems: Theory and Applications, vol. 13, pp. 79-110, 2003.
17. D. P. Bertsekas, V. S. Borkar, and A. Nedic', "Improved Temporal Difference Methods With Linear Function Approximation," in Learning and Approximate Dynamic Programming, A. Barto, W. Powell, and J. Si, Eds. New York: IEEE Press, 2004, LIDS Tech. Rep. 2573, 2003.
18. H.Yu, "A function approximation approach to estimation of policy gradient for POMDP with structured polices," in Proc. 21st Conf. Uncertainty Artif. Intell., 2005, pp. 642-657.
19. R. S. Sutton and A. G. Barto, Reinforcement Learning. Cambridge, MA: MIT Press, 1998.
20. H. Yu and D. P. Bertsekas, New Error Bounds for Approximations From Projected Linear Equations Univ. Helsinki, Helsinki, Finland, Tech. Rep. C-2008-43, 2008.
21. J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic Press, 1970.
22. M. Duflo, Random Iterative Models. Berlin, Germany: Springer-Verlag, 1997.
23. H. J. Kushner and G. G. Yin, Stochastic Approximation and Recursive Algorithms and Applications, 2nd ed. New York: Springer-Verlag, 2003.
24. D. P. de Farias and B. Van Roy, "On the existence of fixed points for approximate value iteration and temporal-difference learning," J. Optim. Theory Appl., vol. 105, no. 3, pp. 589-608, 2000.
25. J. N. Tsitsiklis and B. Van Roy, "Optimal stopping of Markov processes: Hilbert space theory, approximation algorithms, and an application to pricing financial derivatives," IEEE Trans. Automat. Control, vol. 44, no. 10, pp. 1840-1851, Oct. 1999.
26. H. Yu and D. P. Bertsekas, A Least Squares Q-Learning Algorithm for Optimal Stopping Problems MIT, Cambridge, MA, LIDS Tech. Rep. 2731, 2006.
27. H. Yu and D. P. Bertsekas, "Q-learning algorithms for optimal stopping based on least squares," in Proc. Eur. Control Conf., 2007, pp. 2368-2375.
28. V. S. Borkar, "Stochastic approximation with 'controlled Markov' noise," Syst. Control Lett., vol. 55, pp. 139-145, 2006.
29. V. S. Borkar, Stochastic Approximation: A Dynamic Viewpoint. New Delhi, India: Hindustan Book Agency, 2008.
30. D. P. Bertsekas and H. Yu, "Projected equation methods for approximate solution of large linear systems," J. Comput. Sci. Appl. Math., vol. 227, no. 1, pp. 27-50, May 2009.