Q-learning and enhanced policy iteration in discounted dynamic programming

Proceedings of the IEEE Conference on Decision and Control

Volumn , Issue , 2010, Pages 1409-1416

  Bertsekas, Dimitri P ^a   Yu, Huizhen ^b  

^a Department of Electrical Engineering and Computer Science   (United States)

^b UNIVERSITY OF HELSINKI   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; DYNAMIC PROGRAMMING; ITERATIVE METHODS; LINEAR SYSTEMS; Q FACTOR MEASUREMENT; REINFORCEMENT LEARNING; STOCHASTIC SYSTEMS; TABLE LOOKUP;

LARGE-SCALE PROBLEM; LINEAR BASIS FUNCTION; LINEAR SYSTEM OF EQUATIONS; MARKOVIAN DECISION PROBLEMS; OPTIMAL STOPPING PROBLEM; POLICY EVALUATION; Q-LEARNING ALGORITHMS; TEMPORAL DIFFERENCES;

LEARNING ALGORITHMS;

EID: 79953158573     PISSN: 07431546     EISSN: 25762370     Source Type: Conference Proceeding    
DOI: 10.1109/CDC.2010.5717930     Document Type: Conference Paper

References (42)

1. Abounadi, J., Bertsekas, D. P., and Borkar, V., 2002. "Stochastic Approximation for Non-Expansive Maps: Application to QLearning Algorithms," SIAM J. on Control and Optimization, Vol. 41, pp. 1-22.
2. Bertsekas, D. P., and Ioffe, S., 1996. "Temporal Differences-Based Policy Iteration and Applications in Neuro-Dynamic Programming," Lab. for Info. and Decision Systems Report LIDSP-2349, MIT, Cambridge, MA.
3. Bertsekas, D. P., and Tsitsiklis, J. N., 1989. Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Englewood Cliffs, N. J; republished by Athena Scientific, Belmont, MA, 1997.
4. Bertsekas, D. P., and Tsitsiklis, J. N., 1996. Neuro-Dynamic Programming, Athena Scientific, Belmont, MA.
5. Bertsekas, D. P., and Yu, H., 2010. "Q-Learning and Enhanced Policy Iteration in Discounted Dynamic Programming" Lab. for Information and Decision Systems Report 2831, MIT.
6. Bertsekas, D. P., 1982. "Distributed Dynamic Programming," IEEE Trans. Automatic Control, Vol. AC-27, pp. 610-616.
7. Bertsekas, D. P., 1983. "Asynchronous Distributed Computation of Fixed Points," Math. Programming, Vol. 27, pp. 107-120.
8. Bertsekas, D. P., 2005. Dynamic Programming and Optimal Control, 3rd Edition, Vol. I, Athena Scientific, Belmont, MA.
9. Bertsekas, D. P., 2007. Dynamic Programming and Optimal Control, 3rd Edition, Vol. II, Athena Scientific, Belmont, MA.
10. Bertsekas, D. P., 2010. Approximate Dynamic Programming, online at http://web.mit.edu/dimitrib/www/dpchapter.html.
11. Bhatnagar, S., and Babu, K. M., 2008. "New Algorithms of the Q-Learning Type," Automatica, Vol. 44, pp. 1111-1119.
12. Borkar, V. S., 1998. "Asynchronous Stochastic Approximations," SIAM J. on Control and Optimization, Vol. 36, pp. 840-851; correction note in ibid., Vol. 38, pp. 662-663.
13. Borkar, V. S., 2008. Stochastic Approximation: A Dynamical Systems Viewpoint, Hindustan Book Agency, New Delhi, India.
14. Boyan, J. A., 2002. "Technical Update: Least-Squares Temporal Difference Learning," Machine Learning, Vol. 49, pp. 1-15.
15. Bradtke, S. J., and Barto, A. G., 1996. "Linear Least-Squares Algorithms for Temporal Difference Learning," Machine Learning, Vol. 22, pp. 33-57.
16. Chang, H. S., Fu, M. C., Hu, J., Marcus, S. I., 2007. Simulation-Based Algorithms for Markov Decision Processes, Springer, N.Y.
17. Cao, X. R., 2007. Stochastic Learning and Optimization: A Sensitivity-Based Approach, Springer, N.Y.
18. Choi, D. S., and Van Roy, B., 2006. "A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning, Discrete Event Dynamic Systems, Vol. 16, pp. 207-239.
19. Gordon, G. J., 1995. "Stable Function Approximation in Dynamic Programming," Proc. ICML.
20. Gosavi, A., 2003. Simulation-Based Optimization Parametric Optimization Techniques and Reinforcement Learning, Springer-Verlag, N.Y.
21. Jaakkola, T., Jordan, M. I., and Singh, S. P., 1994. "On the Convergence of Stochastic Iterative Dynamic Programming Algorithms," Neural Computation, Vol. 6, pp. 1185-1201.
22. Jaakkola, T., Singh, S. P., and Jordan, M. I., 1995. "Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems," Proc. NIPS.
23. Menache, I., Mannor, S., and Shimkin, N., 2005. "Basis Function Adaptation in Temporal Difference Reinforcement Learning," Ann. Oper. Res., Vol. 134, pp. 215-238.
24. Maei, H. R., Szepesvari, C., Bhatnagar, S., Silver, D., Precup, D., and Sutton, R. S., 2009. "Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation," Proc. NIPS.
25. Meyn, S., 2007. Control Techniques for Complex Networks, Cambridge University Press, N.Y.
26. Powell, W. B., 2007. Approximate Dynamic Programming: Solving the Curses of Dimensionality, Wiley, N.Y.
27. Puterman, M. L., 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, N.Y.
28. Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvari, C., and Wiewiora, E., 2009. "Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation," Proc. ICML.
29. Sutton, R. S., Szepesvari, C., and Maei, H. R., 2008. "A Convergent O(n) Algorithm for Off-Policy Temporal-Difference Learning with Linear Function Approximation," Proc. NIPS.
30. Sutton, R. S., and Barto, A. G., 1998. Reinforcement Learning, MIT Press, Cambridge, MA.
31. Sutton, R. S., 1988. "Learning to Predict by the Methods of Temporal Differences," Machine Learning, Vol. 3, pp. 9-44.
32. Tsitsiklis, J. N., Bertsekas, D. P., and Athans, M., 1986. "Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms," IEEE Trans. on Aut. Control, Vol. AC-31, pp. 803-812.
33. Tsitsiklis, J. N., and Van Roy, B., 1996. "Feature-Based Methods for Large-Scale Dynamic Programming," Machine Learning, Vol. 22, pp. 59-94.
34. Tsitsiklis, J. N., and Van Roy, B., 1999. "Optimal Stopping of Markov Processes: Hilbert Space Theory, Approximation Algorithms, and an Application to Pricing Financial Derivatives," IEEE Trans. on Aut. Control, Vol. 44, pp. 1840-1851.
35. Tsitsiklis, J. N., 1994. "Asynchronous Stochastic Approximation and Q-Learning," Machine Learning, Vol. 16, pp. 185-202.
36. Tsitsiklis, J. N., 2002. "On the Convergence of Optimistic Policy Iteration," J. of Machine Learning Research, Vol. 3, pp. 59-72.
37. Van Roy, B., 2009. "On Regression-Based Stopping Times," Discrete Event Dynamic Systems, to appear.
38. Watkins, C. J. C. H., Learning from Delayed Rewards, Ph.D. Thesis, Cambridge Univ., England.
39. Williams, R. J., and Baird, L. C., 1993. "Analysis of Some Incremental Variants of Policy Iteration: First Steps Toward Understanding Actor-Critic Learning Systems," Report NU-CCS-93-11, College of Computer Science, Northeastern University, Boston, MA.
40. Yu, H., and Bertsekas, D. P., 2007. "A Least Squares Q-Learning Algorithm for Optimal Stopping Problems," Lab. for Information and Decision Systems Report 2731, MIT;
41. also in Proc. European Control Conference 2007, Kos, Greece.
42. Yu, H., and Bertsekas, D. P., 2009. "Basis Function Adaptation Methods for Cost Approximation in MDP," Proc. IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2009), Nashville, Tenn.