Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems

Automatica

Volumn 50, Issue 1, 2014, Pages 193-202

  Modares, Hamidreza ^a   Lewis, Frank L ^b   Naghibi Sistani, Mohammad Bagher ^a  

^a FERDOWSI UNIVERSITY OF MASHHAD   (Iran)

^b UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

Author keywords

Experience replay; Input constraints; Integral reinforcement learning; Neural networks; Optimal control

Indexed keywords

ADAPTIVE OPTIMAL CONTROL; EXPERIENCE REPLAY; FEEDBACK CONTROL LAW; HAMILTON JACOBI BELLMAN EQUATION; INPUT CONSTRAINTS; NEAR-OPTIMAL CONTROL; OPTIMAL CONTROLS; PERSISTENCE OF EXCITATION;

CONTROL; CONTROL THEORY; DYNAMIC PROGRAMMING; NEURAL NETWORKS; ONLINE SYSTEMS; REINFORCEMENT LEARNING;

OPTIMAL CONTROL SYSTEMS;

EID: 84893708995     PISSN: 00051098     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.automatica.2013.09.043     Document Type: Article

References (32)

1. M. Abu-Khalaf, and F.L. Lewis Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach Automatica 41 2005 779 791
2. S. Adam, L. Busoniu, and R. Babuska Experience replay for real-time reinforcement learning control IEEE Transactions on Systems Man, and Cybernetics, Part C: Applications and Reviews 42 2012 201 212
3. Beard, R.W. (1995). Improving the closed-loop performance of nonlinear systems. Ph.D. dissertation, Elec. Eng. Dep., Rensselaer Polytech. Ins., Troy, NY.
4. D.P. Bertsekas, and J.N. Tsitsiklis Neuro-dynamic programming 1996 Athena Scientific MA
5. S. Bhasin, R. Kamalapurkar, M. Johnson, K.G Vamvoudakis, F.L Lewis, and W.E. Dixon A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems Automatica 49 2012 82 92
6. Chowdhary, G.V. (2010). Concurrent learning for convergence in adaptive control without persistency of excitation, PhD Thesis, Georgia institute of technology.
7. Chowdhary, G.V., & Johnson, E. (2010). Concurrent learning for convergence in adaptive control without. In IEEE CDC. Atlanta GA (pp. 3675-3679).
8. K. Doya Reinforcement learning in continuous time and space Neural Computation 12 2000 219 245
9. Dung, L.T., Komeda, T., & Takagi, M. (2008). Efficient experience reuse in non-Markovian environments. In Proc. int. conf. instrum. control inf. technol. Tokyo, Japan (pp. 3327-3332).
10. B.A. Finlayson The method of weighted residuals and variational principles 1990 Academic Press New York
11. G. Hardy, J. Littlewood, and G. Polya Inequalities 2nd ed. 1989 Cambridge Univ. Press Cambridge, U.K.
12. P. Ioannou, and J. Sun Robust adaptive control 1996 Prentice Hall New Jersey
13. Kalyanakrishnan, S., & Stone, P. (2007). Batch reinforcement learning in a complex domain. In Proc. 6th Int. Conf. Auton. Agents Multi-Agent Syst. Honolulu, HI (pp. 650-657).
14. H.K. Khalil Nonlinear systems 3rd ed. 2002 Prentice Hall
15. F.L. Lewis, S. Jagannathan, and A. Yesildirek Neural network control of robot manipulators and nonlinear systems 1999 Taylor & Francis
16. F.L. Lewis, and D. Vrabie Reinforcement learning and adaptive dynamic programming for feedback control IEEE Circuits & Systems Magazine, Invited Feature Article 9 2009 32 50
17. F.L. Lewis, D. Vrabie, and K.G. Vamvoudakis Reinforcement learning and feedback control IEEE Control Systems magazine 2012
18. F.L. Lewis, D. Vrabie, and V. Syrmos Optimal control 3rd ed. 2012 Wiley
19. L.J. Lin Self-improving reactive agents based on reinforcement learning, planning and teaching Machine Learning 8 1992 293 321
20. Lyshevski, S.E. (1998). Optimal control of nonlinear continuous-time systems: design of bounded controllers via generalized nonquadratic functionals. In Proceedings of American control conference (pp. 205-209).
21. H. Modares, F.L. Lewis, and M.B. Naghibi-Sistani Online solution of nonquadratic two-player zero-sum games arising in the H control of constrained input systems International Journal of Adaptive Control and Signal Processing 2012 10.1002/acs.2348
22. J.J. Murray, C.J. Cox, G.G. Lendaris, and R. Saeks Adaptive dynamic programming IEEE Transactions on Systems Man, and Cybernetics, Part C: Applications and Reviews 32 2002 140 153
23. Polycarpou, M., & Ioannou, P. (1993). A robust adaptive nonlinear control design. In Proceedings of American control conference (pp. 1365-1369).
24. W.B. Powell Approximate dynamic programming: solving the curses of dimensionality 2007 Wiley-Interscience
25. R.S Sutton, and A.G. Barto Reinforcement learning - an introduction 1998 MIT Press Cambridge, MA
26. K. Vamvoudakis, and F.L. Lewis Online actor-critic algorithm to solve the continuous infinite-time horizon optimal control problem Automatica 46 2010 878 888
27. K.G. Vamvoudakis, D. Vrabie, and F.L. Lewis Online adaptive algorithm for optimal control with integral reinforcement learning Proc. International (pp. 250-257, 2011). Journal of robust and Nonlinear control 2013 10.1002/rnc.3018
28. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F.L. Lewis Adaptive optimal control for continuous-time linear systems based on policy iteration Automatica 45 2009 477 484
29. D. Vrabie, and F.L. Lewis Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems Neural Networks 22 2009 237 246
30. P. Wawrzynski Real-time reinforcement learning by sequential actor-critics and experience replay Neural Networks 22 2009 1484 1497
31. P.J. Werbos Approximate dynamic programming for real time control and neural modeling D.A. White, D.A. Sofge, Handbook of intelligent control 1992 Multiscience Press
32. H. Xu, S. Jagannathan, and F.L. Lewis Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses Automatica 48 2012 1017 1030