Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics

IEEE Transactions on Automation Science and Engineering

Volumn 11, Issue 3, 2014, Pages 706-714

  Li, Hongliang ^a   Liu, Derong ^a   Wang, Ding ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive critic designs; adaptive dynamic programming; approximate dynamic programming; policy iteration; reinforcement learning; zero sum games

Indexed keywords

ALGEBRA; ALGORITHMS; CONTINUOUS TIME SYSTEMS; DYNAMIC PROGRAMMING; GAME THEORY; LEAST SQUARES APPROXIMATIONS; NEWTON-RAPHSON METHOD; RICCATI EQUATIONS; STATE FEEDBACK;

ADAPTIVE CRITIC DESIGNS; ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE DYNAMIC PROGRAMMING; POLICY ITERATION; ZERO-SUM GAME;

REINFORCEMENT LEARNING;

EID: 84904398037     PISSN: 15455955     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASE.2014.2300532     Document Type: Article

References (49)

1. P. J. Werbos, "Intelligence in the brain: A theory of how it works and how to build it," Neural Netw., vol. 22, no. 3, pp. 200-212, Apr. 2009.
2. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
3. F. L. Lewis and D. Vrabie, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst. Mag., vol. 32, no. 6, pp. 76-105, Dec. 2012.
4. F. L. Lewis and D. Liu, Approximate Dynamic Programming and Reinforcement Learning for Feedback Control. Hoboken, NJ: Wiley, 2012.
5. R. S. Sutton and A. G. Barto, Reinforcement Learning : An Introduction. Cambridge, MA: MIT Press, 1998.
6. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
7. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systemswith control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
8. F. Y. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear systems with ∈-error bound," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1854-1862, Dec. 2011.
9. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
10. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
11. H. Li and D. Liu, "Optimal control for discrete-time affine nonlinear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.
12. Q. Wei and D. Liu, "An iterative-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, no. 6, pp. 236-244, Aug. 2012.
13. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithmfor optimal control of unknown discrete-time nonlinear systemswith constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
14. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, June 2013.
15. D. Liu and Q. Wei, "Finite-approximation-error based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
16. J. Si and Y. T. Wang, "On-line learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001. (Pubitemid 32371483)
17. D. Liu, Y. Zhang, and H. Zhang, "Aself-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw., vol. 16, no. 5, pp. 1219-1228, Sep. 2005. (Pubitemid 41444623)
18. D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
19. G. G. Yen and P. G. Delima, "Improving the performance of globalized dual heuristic programming for fault tolerant control through an online learning supervisor," IEEE Trans. Autom. Sci. Eng., vol. 2, no. 2, pp. 121-131, Apr. 2005.
20. R. Ganesan, T. K. Das, and K. M. Ramachandran, "A multiresolution analysis-assisted reinforcement learning approach to run-by-run control," IEEE Trans. Autom. Sci. Eng., vol. 4, no. 2, pp. 182-193, Apr. 2007. (Pubitemid 46574023)
21. W. S. Lin and J.W. Sheu, "Optimization of train regulation and energy usage of metro lines using an adaptive-optimal-control algorithm," IEEE Trans. Autom. Sci. Eng., vol. 8, no. 4, pp. 855-864, Oct. 2011.
22. S. K. Pradhan and B. Subudhi, "Real-time adaptive control of a flexible manipulator using reinforcement learning," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 2, pp. 237-249, Apr. 2012.
23. Q. Kang, M. Zhou, J. An, and Q. Wu, "Swarm intelligence approaches to optimal power flow problem with distributed generator failures in power networks," IEEE Trans. Autom. Sci. Eng., vol. 10, no. 2, pp. 343-353, Apr. 2013.
24. B. Sun, P. B. Luh, Q. Jia, Z. Jiang, F. Wang, and C. Song, "Building energy management: Integrated control of active and passive heating, cooling, lighting, shading, and ventilation systems," IEEE Trans. Autom. Sci. Eng., vol. 10, no. 3, pp. 588-602, Jul. 2013.
25. Z. Xu, Q. Jia, H. Guan, and J. Shen, "Smart management of multiple energy systems in automotive painting shop," IEEE Trans. Autom. Sci. Eng., vol. 10, no. 3, pp. 603-614, Jul. 2013.
26. H. He, Z. Ni, and J. Fu, "A three-network architecture for on-line learning and optimization based on adaptive dynamic programming," Neurocomputing, vol. 78, no. 1, pp. 3-13, Feb. 2012.
27. Z. Ni, H. He, and J. Wen, "Adaptive learning in tracking control based on the dual critic network design," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 6, pp. 913-928, Jun. 2013.
28. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
29. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
30. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
31. S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon, "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
32. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-time linear systems based on policy iteration," Automatica, vol. 45, no. 2, pp. 477-484, Feb. 2009.
33. D. Vrabie and F. L. Lewis, "Neural network approach to continuoustime direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, Apr. 2009.
34. P. Mehta and S. Meyn, "Q-learning and pontryagins minimum principle," in Proc. IEEE Conf. Decision Control, Shanghai, China, Dec. 2009, pp. 3598-3605.
35. J. Y. Lee, J. B. Park, and Y. H. Choi, "Integral Q-learning and explorized policy iteration for adaptive optimal control of continuoustime linear systems," Automatica, vol. 48, no. 11, pp. 2850-2859, Nov. 2012.
36. Y. Jiang and Z.-P. Jiang, "Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics," Automatica, vol. 48, no. 10, pp. 2699-2704, Oct. 2012.
37. T. Basar and G. J. Olsder, Dynamic Noncooperative Game Theory, 2nd ed. Philadelphia, PA, USA: Society for Industrial and Applied Mathematics, 1999.
38. T. Basar and P. Bernhard, H∞ Optimal Conrol and Related Minimax Design Problems: A Dynamic Game Approach, 2nd ed. Boston, MA, USA: Birkhäuser, 1995.
39. A. Al-Tamimi, M. Abu-Khalaf, and F. L. Lewis, "Adaptive critic designs for discrete-time zero-sum games with application to H∞ control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 1, pp. 240-247, Feb. 2007. (Pubitemid 46358495)
40. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Model-free Q-learning designs for linear discrete-time zero-sum games with application to H∞ control," Automatica, vol. 43, no. 3, pp. 473-481, Mar. 2007. (Pubitemid 46209050)
41. J.-H. Kim and F. L. Lewis, "Model-free H∞ control design for unknown linear discrete-time systems via Q-learning with LMI," Automatica, vol. 46, no. 8, pp. 1320-1326, Aug. 2010.
42. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Policy iterations and the Hamilton-Jacobi-Isaacs equation for H∞ state feedback control with input saturation," IEEE Trans. Autom. Control, vol. 51, no. 12, pp. 1989-1995, Dec. 2006. (Pubitemid 46002295)
43. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Neurodynamic progarmming and zero-sum games for constrained control systems," IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1243-1252, Jul. 2008.
44. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games," Automatica, vol. 47, no. 1, pp. 207-214, Jan. 2011.
45. K. G. Vamvoudakis and F. L. Lewis, "Online solution of nonlinear two-player zero-sum games using synchronous policy iteration," Int. J. Robust Nonlinear Control, vol. 22, no. 13, pp. 1460-1483, 2011.
46. T. Dierks and S. Jagannathan, "Optimal control of affine nonlinear continuous-time systems using an online Hamilton-Jacobi-Isaacs formulation," in Proc. IEEE Conf. Decision Control, Atlanta, GA, USA, Dec. 2010, pp. 3048-3053.
47. H. Wu and B. Luo, "Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H∞ control," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 12, pp. 1884-1895, Dec. 2012.
48. D. Varbie and F. L. Lewis, "Adaptive dynamic programming for online solution of a zero-sum differential game," J. Control Theory Appl., vol. 9, no. 3, pp. 353-360, 2011.
49. H. Wu and B. Luo, "Simultaneous policy update algorithms for learning the solution of linear continuous-time H∞ state feedback control," Inf. Sci., vol. 222, no. 10, pp. 472-485, Feb. 2013.