Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics

Automatica

Volumn 50, Issue 4, 2014, Pages 1167-1175

  Kiumarsi, Bahare ^a   Lewis, Frank L ^b   Modares, Hamidreza ^a   Karimpour, Ali ^a   Naghibi Sistani, Mohammad Bagher ^a  

^a FERDOWSI UNIVERSITY OF MASHHAD   (Iran)

^b UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

Author keywords

Algebraic Riccati equation; Linear quadratic tracker; Policy iteration; Reinforcement learning

Indexed keywords

ALGEBRA; DIFFERENCE EQUATIONS; DIGITAL CONTROL SYSTEMS; DISCRETE TIME CONTROL SYSTEMS; ITERATIVE METHODS; NAVIGATION; REINFORCEMENT LEARNING; RICCATI EQUATIONS;

ALGEBRAIC RICCATI EQUATIONS; DISCRETE TIME SYSTEM; LINEAR DISCRETE-TIME SYSTEMS; LINEAR QUADRATIC TRACKERS; OPTIMAL TRACKING CONTROL; POLICY ITERATION; Q-LEARNING ALGORITHMS; REFERENCE TRAJECTORIES;

LEARNING ALGORITHMS;

EID: 84898853127     PISSN: 00051098     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.automatica.2014.02.015     Document Type: Article

References (30)

1. A. Al-Tamimi, F.L. Lewis, and M. Abu-Khalaf Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control Automatica 43 3 2007 473 481
2. A. Al-Tamimi, F.L. Lewis, and M. Abu-Khalaf Discrete-time nonlinear HJB solution using approximate dynamic programming: convergence proof IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics 38 4 2008 943 949
3. Baird, L. C. III (1994). Reinforcement learning in continuous time: advantage updating. In Proc. of ICNN.
4. J.Si.A. Barto, W. Powell, and D. Wunch Handbook of learning and approximate dynamic programming 2004 John Wiley
5. D.P. Bertsekas, and J.N. Tsitsiklis Neuro-dynamic programming 1996 Athena Scientific MA
6. Bradtke, S. J.; Ydestie, B. E.; Barto, A. G. (1994). Adaptive linear quadratic control using policy iteration. In: Proc. of ACC (pp. 3475-3476).
7. Dierks, T.; Jagannathan, S. (2010). Optimal control of affine nonlinear continuous-time systems. In Proc. Am. control conf. (pp. 1568-1573).
8. G.A. Hewer An iterative technique for the computation of steady state gains for the discrete optimal regulator IEEE Transactions on Automatic Control 16 4 1971 382 384
9. Y. Jiang, and Z.P. Jiang Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics Automatica 48 2012 2699 2704
10. Landelius, T.; Knutsson, H. (1996). Greedy adaptive critics for LQR problem: convergence proof. Technical report, Linkoping, Sweden, Computer vision laboratory.
11. J.Y. Lee, J.B. Park, and Y.H. Choi Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems Automatica 48 2012 2850 2859
12. F.L. Lewis, and D. Liu Reinforcement learning and approximate dynamic programming for feedback control 2013 Wiley Hoboken, NJ
13. F.L. Lewis, and K. Vamvoudakis Reinforcement learning for partially observable dynamic processes: adaptive dynamic programming using measured output data IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics 41 1 2011 14 23
14. F.L. Lewis, K. Vamvoudakis, and D. Vrabie Optimal adaptive control and differential games by reinforcement learning principles 2013 Institution of Engineering and Technology London
15. F.L. Lewis, and D. Vrabie Reinforcement learning and adaptive dynamic programming for feedback control IEEE Circuits and Systems Magazine 9 3 2009 32 50
16. F.L. Lewis, D. Vrabie, and V. Syrmos Optimal control 2012 John Wiley
17. F.L. Lewis, D. Vrabie, and K.G. Vamvoudakis Reinforcement learning and feedback control using natural decision methods to design optimal adaptive controllers IEEE Systems Magazine 32 6 2012 76 105
18. W.B. Powell Approximate dynamic programming: solving the curses of dimensionality 2009 John Wiley
19. R.S. Sutton, and A.G. Barto Reinforcement learning - an introduction 1998 MIT Press Cambridge, MT
20. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F.L. Lewis Adaptive optimal control for continuous-time linear systems based on policy iteration Automatica 45 2009 477 484
21. D. Wang, D. Liu, and Q. Wei Finite-horizon neurooptimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach Neurocomputing 78 2012 14 22
22. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Ph.D. thesis, England: University of Cambridge.
23. Werbos, P. J. (1989). Neural networks for control and system identification. In: Proc. of CDC (pp. 260-265).
24. P.J. Werbos A menu of designs for reinforcement learning over time Neural networks for control 1991 MIT Press Cambridge, MA 67 95
25. P.J. Werbos Approximate dynamic programming for real-time control and neural modeling D.A. White, D.A. Sofge, Handbook of intelligent control 1992 Van Nostrand Reinhold New York
26. D.A. White, and D.A. Sofge Handbook of intelligent control 1992 Van Nostrand Reinhold New York
27. H. Zhang, L. Cui, X. Zhang, and X. Luo Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method IEEE Transactions on Neural Networks 22 2011 2226 2236
28. H. Zhang, D. Liu, Y. Luo, and D. Wang Adaptive dynamic programming for control-algorithms and stability 2012 Springer-Verlag London
29. H. Zhang, Q. Wei, and D. Liu An iterative approximate dynamic programming method to solve for a class of nonlinear zero-sum differential games Automatica 47 1 2011 207 214
30. H. Zhang, Q. Wei, and Y. Luo A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 38 2008 937 942