Q-learning and Pontryagin's minimum principle

Proceedings of the IEEE Conference on Decision and Control

Volumn , Issue , 2009, Pages 3598-3605

  Mehta, Prashant ^a   Meyn, Sean ^a  

^a University of Illinois at Urbana Champaign   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; APPROXIMATION THEORY; CONTINUOUS TIME SYSTEMS; DISTRIBUTED PARAMETER CONTROL SYSTEMS; HAMILTONIANS; LEARNING SYSTEMS; MARKOV CHAINS; MULTI AGENT SYSTEMS; OPTIMIZATION; REINFORCEMENT LEARNING; STOCHASTIC CONTROL SYSTEMS; STOCHASTIC SYSTEMS;

CONSISTENT ALGORITHM; CONTINUOUS TIME MODELS; CONTROLLED MARKOV CHAINS; OPTIMAL APPROXIMATION; OPTIMALITY EQUATION; PONTRYAGIN'S MINIMUM PRINCIPLES; Q-LEARNING ALGORITHMS; STOCHASTIC APPROXIMATIONS;

LEARNING ALGORITHMS;

EID: 77950806766     PISSN: 07431546     EISSN: 25762370     Source Type: Conference Proceeding    
DOI: 10.1109/CDC.2009.5399753     Document Type: Conference Paper

References (21)

1. D.P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Atena Scientific, Cambridge, Mass, 1996.
2. V. S. Borkar and S. P. Meyn. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim., 38(2):447-469, 2000. (also presented at the IEEE CDC, December, 1998).
3. S. J. Bradtke and A. G. Barto. Linear least-squares algorithms for temporal difference learning. Mach. Learn., 22(1-3):33-57, 1996.
4. S.J. Bradtke, B.E. Ydstie, and A.G. Barto. Adaptive linear quadratic control using policy iteration. In Proceedings of the 1994 American Control Conference, volume 3, pages 3475-3479, 1994.
5. R. Cogill, M. Rotkowitz, B. Van Roy, and S Lall. An approximate dynamic programming approach to decentralized control of stochastic systems. In Control of Uncertain Systems: Modelling, Approximation, and Design, pages 243-256. Springer, 2006.
6. D. P. Pucci de Farias and B. Van Roy. A cost-shaping linear program for average-cost approximate dynamic programming with performance guarantees. Math. Oper. Res., 31(3):597-620, 2006.
7. V. F. Farias, D. Saure, and G. Y. Weintraub. The linear programming approach to solving large scale dynamic stochastic games (working paper). http://www.stanford.edu/ bvr/publ-all.html, 2008.
8. J. Han and B. Van Roy. Control of diffusions via linear programming. To appear in a volume on stochastic programming in honor of George Dantzig, edited by Gerd Infanger. Preprint available at http://www.stanford.edu/~bvr/, 2009.
9. M. Huang, P. E. Caines, and R. P. Malhame. Large-population cost-coupled LQG problems with nonuniform agents: Individual-mass behavior and decentralized ε-Nash equilibria. IEEE Trans. Automat. Control, 52(9):1560-1571, 2007.
10. D. H. Jacobson and D. Q. Mayne. Differential dynamic programming. American Elsevier Pub. Co., New York, NY, 1970.
11. F. S. Melo, S. Meyn, and M. Isabel Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of ICML, pages 664-671, 2008.
12. S. P. Meyn. Control Techniques for Complex Networks. Cambridge University Press, Cambridge, 2007.
13. S. P. Meyn and G. Mathew. Shannon meets Bellman: Feature based Markovian models for detection and optimization. In Proc. 47th IEEE CDC, pages 5558-5564, 2008.
14. S. P. Meyn and R. L. Tweedie. Markov chains and stochastic stability. Cambridge University Press, Cambridge, second edition, 2009. Published in the Cambridge Mathematical Library. 1993 edition online: http://black.csl.uiuc.edu/ ~meyn/pages/book.html.
15. C.C. Moallemi, S. Kumar, and B. Van Roy. Approximate and data-driven dynamic programming for queueing networks. Preprint available at http://moallemi.com/ciamac/research-interests.php, 2008.
16. Y. Tassa and T. Erez. Least squares solutions of the HJB equation with neural network value-function approximators. IEEE Transactions on Neural Networks, 18(4):1031-1041, 2007.
17. J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Trans. Automat. Control, 42(5):674-690, 1997.
18. D. Vrabie, M. Abu-Khalaf, F.L. Lewis, and Y. Wang. Continuous-time ADP for linear systems with partially unknown dynamics. In Proc. IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pages 247-253, April 2007.
19. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F.L. Lewis. Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica, 45(2):477-484, 2009.
20. C. J. C. H. Watkins and P. Dayan. Q-learning. Machine Learning, 8(3-4):279-292, 1992.
21. H. Yu and D. P. Bertsekas. Q-learning algorithms for optimal stopping based on least squares. In Proc. European Control Conference (ECC), July 2007.