Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 5, 2015, Pages 916-932

  Lee, Jae Young ^a   Park, Jin Bae ^a   Choi, Yoon Ho ^b  

^a Yonsei University   (South Korea)

^b Kyonggi University   (South Korea)

Author keywords

Adaptive optimal control; continuous time (CT); exploration; policy iteration (PI); Q learning; reinforcement learning (RL)

Indexed keywords

ALGORITHMS; CLOSED LOOP SYSTEMS; CONTINUOUS TIME SYSTEMS; ITERATIVE METHODS; LEARNING ALGORITHMS; NATURAL RESOURCES EXPLORATION; NONLINEAR SYSTEMS; NUMERICAL METHODS; OPTIMAL CONTROL SYSTEMS; SOCIAL NETWORKING (ONLINE);

ADAPTIVE OPTIMAL CONTROL; CONTINUOUSTIME; CONVERGENCE PROPERTIES; EXCITATION CONDITIONS; INPUT-TO-STATE STABILITY; NONLINEAR OPTIMAL CONTROL PROBLEMS; POLICY ITERATION; Q-LEARNING;

REINFORCEMENT LEARNING;

EID: 85027928575     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2014.2328590     Document Type: Article

References (36)

1. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Jun. 2009.
2. J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Handbook of Learning and Approximate Dynamic Programming. New York, NY, USA: Wiley, 2004.
3. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, U.K.: Cambridge Univ. Press, 1998.
4. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. New York, NY, USA: Wiley, 2007.
5. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, nos. 3-4, pp. 279-292, 1992.
6. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, May 1996.
7. G. N. Saridis and C.-S. G. Lee, "An approximation theory of optimal control for trainable manipulators," IEEE Trans. Syst., Man Cybern., vol. 9, no. 3, pp. 152-159, Mar. 1979.
8. R. W. Beard, G. N. Saridis, and J. T. Wen, "Galerkin approximation of the generalized Hamilton-Jacobi equation," Automatica, vol. 33, no. 12, pp. 2159-2177, 1996.
9. R. W. Beard, "Improving the closed-loop performance of nonlinear systems," Ph.D. dissertation, Dept. Phys., Rensselaer Polytechnic Institute, Troy, NY, USA, 1995.
10. J. Y. Lee, J. B. Park, and Y. H. Choi. (2014, Apr.). Invariantly admissible policy iteration for a class of nonlinear optimal control problems. Syst. Control Lett. [Online]. Available: http://arxiv.org/abs/1402.4187
11. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern., Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
12. D. Vrabie and F. Lewis, "Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, 2009.
13. J. Y. Lee, J. B. Park, and Y. H. Choi, "Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems," Automatica, vol. 48, no. 11, pp. 2850-2859, 2012.
14. J. Y. Lee, J. B. Park, and Y. H. Choi, "Integral reinforcement learning with explorations for continuous-time nonlinear systems," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), 2012, pp. 1042-1047.
15. Y. Jiang and Z.-P. Jiang, "Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics," Automatica, vol. 48, no. 10, pp. 2699-2704, 2012.
16. S. J. Bradtke, B. E. Ydstie, and A. G. Barto, "Adaptive linear quadratic control using policy iteration," in Proc. Amer. Control Conf. (ACC), vol. 3. Jul. 1994, pp. 3475-3479.
17. J. Y. Lee, J. B. Park, and Y. H. Choi, "On integral generalized policy iteration for continuous-time linear quadratic regulations," Automatica, vol. 50, no. 2, pp. 475-489, 2014.
18. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control," Automatica, vol. 43, no. 3, pp. 473-481, 2007.
19. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand, 1992, ch. 13, pp. 493-525.
20. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
21. D. Vrabie, "Online adaptive optimal control for continuous-time systems," Ph.D. dissertation, Dept. History, Univ. Texas Arlington, Arlington, TX, USA, 2010.
22. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, 2010.
23. K. G. Vamvoudakis, D. Vrabie, and F. L. Lewis, "Online adaptive algorithm for optimal control with integral reinforcement learning," Int. J. Robust Nonlinear Control, 2013, doi: 10.1002/rnc.3018.
24. S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon, "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 82-92, 2013.
25. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 10, pp. 1513-1525, Oct. 2013.
26. F. L. Lewis and V. L. Syrmos, Optimal Control. New York, NY, USA: Wiley, 1995.
27. D. E. Kirk, Optimal Control Theory: An Introduction. New York, NY, USA: Dover, 2004.
28. R. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ., 1957.
29. F. Ornelas-Tellez, E. N. Sanchez, and A. G. Loukianov, "Discrete-time neural inverse optimal control for nonlinear systems via passivation," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 8, pp. 1327-1339, Aug. 2012.
30. L. J. Ricalde and E. N. Sanchez, "Inverse optimal neural control of a class of nonlinear systems with constrained inputs for trajectory tracking," Optim. Control Appl. Methods, vol. 33, no. 2, pp. 176-198, 2012.
31. M. Fu and B. Barmish, "Adaptive stabilization of linear systems via switching control," IEEE Trans. Autom. Control, vol. 31, no. 12, pp. 1097-1103, Dec. 1986.
32. E. B. Kosmatopoulos, "Control of unknown nonlinear systems with efficient transient performance using concurrent exploitation and exploration," IEEE Trans. Neural Netw., vol. 21, no. 8, pp. 1245-1261, Aug. 2010.
33. P. A. Ioannou and J. Sun, Stable and Robust Adaptive Control. Englewood Cliffs, NJ, USA: Prentice-Hall, 1995.
34. H. K. Khalil, Nonlinear Systems. Englewood Cliffs, NJ, USA: Prentice-Hall, 2002.
35. V. Nevistíc and J. A. Primbs, "Constrained nonlinear optimal control: A converse HJB approach," Dept. Control & Dynamical Syst., California Inst. Technology, Pasadena, CA, USA, Tech. Rep. CIT-CDS 96-021, 1996.
36. D. Kleinman, "On an iterative technique for Riccati equation computations," IEEE Trans. Autom. Control, vol. 13, no. 1, pp. 114-115, Feb. 1968.