Experience Replay for Optimal Control of Nonzero-Sum Game Systems with Unknown Dynamics

IEEE Transactions on Cybernetics

Volumn 46, Issue 3, 2016, Pages 854-865

  Zhao, Dongbin ^a   Zhang, Qichao ^a   Wang, Ding ^a   Zhu, Yuanheng ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive dynamic programming (ADP); experience replay; nonzero sum (NZS) games; optimal control; unknown dynamics

Indexed keywords

CLOSED LOOP SYSTEMS; DYNAMICS; NETWORK LAYERS; NONLINEAR EQUATIONS; ONLINE SYSTEMS; SYSTEM STABILITY;

ADAPTIVE DYNAMIC PROGRAMMING; EXPERIENCE REPLAY; LYAPUNOV-BASED STABILITY ANALYSIS; NONZERO-SUM (NZS) GAMES; OPTIMAL CONTROLS; PERSISTENCE OF EXCITATION; THREE-LAYER NEURAL NETWORKS; UNIFORM ULTIMATE BOUNDEDNESS;

DYNAMIC PROGRAMMING;

EID: 84945951645     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2015.2488680     Document Type: Article

References (38)

1. F. L. Lewis and D. Liu, Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Hoboken, NJ, USA: Wiley, 2013.
2. Q. Wei and D. Liu, "An iterative optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, no. 6, pp. 236-244, Aug. 2012.
3. D. Zhao et al., "Full-range adaptive cruise control based on supervised adaptive dynamic programming," Neurocomputing, vol. 125, pp. 57-67, Feb. 2014.
4. D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
5. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
6. D. Zhao, X. Bai, F.-Y. Wang, J. Xu, and W. Yu, "DHP method for ramp metering of freeway traffic," IEEE Intell. Transp. Syst. Mag., vol. 12, no. 4, pp. 990-999, Dec. 2011.
7. Y. Zhu, D. Zhao, and D. Liu, "Convergence analysis and application of fuzzy-HDP for nonlinear discrete-time HJB systems," Neurocomputing, vol. 149, pp. 124-131, Feb. 2015.
8. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
9. A. Bidram, A. Davoudi, F. L. Lewis, and J. M. Guerrero, "Distributed cooperative secondary control of microgrids using feedback linearization," IEEE Trans. Power Syst., vol. 28, no. 3, pp. 3462-3470, Aug. 2013.
10. K. R. S. Kodagoda, W. S. Wijesoma, and E. K. Teoh, "Fuzzy speed and steering control of an AGV," IEEE Trans. Control Syst. Technol., vol. 10, no. 1, pp. 112-120, Jan. 2002.
11. P. Morris, Introduction to Game Theory. New York, NY, USA: Springer, 2012.
12. Z. Wang, S. Kokubo, J. Tanimoto, E. Fukuda, and K. Shigaki, "Insight into the so-called spatial reciprocity," Phys. Rev. E, vol. 88, no. 4, Oct. 2013, Art. ID 042145.
13. Z. Wang, C.-Y. Xia, S. Meloni, C.-S. Zhou, and Y. Moreno, "Impact of social punishment on cooperative behavior in complex networks," Sci. Rep., vol. 3, Oct. 2013, Art. ID 3055.
14. Z. Wang, A. Szolnoki, and M. Perc, "Optimal interdependence between networks for the evolution of cooperation," Sci. Rep., vol. 3, Aug. 2013, Art. ID 2470.
15. Z. Wang, A. Szolnoki, and M. Perc, "Self-organization towards optimally interdependent networks by means of coevolution," New J. Phys., vol. 16, no. 3, 2014, Art. ID 033041.
16. A. W. Starr and Y.-C. Ho, "Nonzero-sum differential games," J. Optim. Theory Appl., vol. 3, no. 3, pp. 184-206, 1969.
17. J. C. Engwerda and A. J. M. Weeren, The Open-Loop Nash Equilibrium in LQ-Games Revisited, Center Econ. Res., Tilburg Univ., Tilburg, The Netherlands, 1995.
18. M. Johnson, R. Kamalapurkar, S. Bhasin, and W. E. Dixon, "Approximate N-player nonzero-sum game solution for an uncertain continuous nonlinear system," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 8, pp. 1645-1658, Aug. 2015.
19. J. H. Case, "Toward a theory of many player differential games," SIAM J. Control, vol. 7, no. 2, pp. 179-197, 1969.
20. A. Friedman, Differential Games. Mineola, NY, USA: Courier Corporation, 2013.
21. K. G. Vamvoudakis and F. L. Lewis, "Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, no. 8, pp. 1556-1569, 2011.
22. H. Zhang, L. Cui, and Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using singlenetwork ADP," IEEE Trans. Cybern., vol. 43, no. 1, pp. 206-216, Feb. 2013.
23. D. Vrabie and F. Lewis, "Integral reinforcement learning for online computation of feedback Nash strategies of nonzero-sum differential games," in Proc. IEEE Conf. Decis. Control (CDC), Atlanta, GA, USA, 2010, pp. 3066-3071.
24. J. Y. Lee, J. B. Park, and Y. H. Choi, "Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 15, pp. 916-932, May 2015.
25. H. Li, D. Liu, and D. Wang, "Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 706-714, Jul. 2014.
26. D. Zhao, Z. Xia, and D. Wang, "Model-free optimal control for affine nonlinear systems with convergence analysis," IEEE Trans. Autom. Sci. Eng., vol. 12, no. 4, pp. 1461-1468, Oct. 2014.
27. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
28. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
29. H. Xu, S. Jagannathan, and F. L. Lewis, "Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses," Automatica, vol. 48, no. 6, pp. 1017-1030, 2012.
30. S. Adam, L. Busoniu, and R. Babuska, "Experience replay for real-time reinforcement learning control," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 42, no. 2, pp. 201-212, Mar. 2012.
31. G. Chowdhary and E. Johnson, "Concurrent learning for convergence in adaptive control without persistency of excitation," in Proc. IEEE. Conf. Decis. Control (CDC), Atlanta, GA, USA, 2010, pp. 3674-3679.
32. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 10, pp. 1513-1525, Oct. 2013.
33. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems," Automatica, vol. 50, no. 1, pp. 193-202, 2014.
34. S. Yasini, M. B. N. Sitani, and A. Kirampor, "Reinforcement learning and neural networks for multi-agent nonzero-sum games of nonlinear constrained-input systems," Int. J. Mach. Learn. Cybern., pp. 1-14, Oct. 2014.
35. R. Kamalapurkar, J. R. Klotz, and W. E. Dixon, "Concurrent learningbased approximate feedback-Nash equilibrium solution of N-player nonzero-sum differential games," IEEE/CAA J. Autom. Sinica, vol. 1, no. 3, pp. 239-247, Jul. 2014.
36. T. Basar et al., Dynamic Noncooperative Game Theory, vol. 200. Philadelphia, PA, USA: SIAM, 1995.
37. B. A. Finlayson, The Method of Weighted Residuals and Variational Principles. vol. 73. Philadelphia, PA, USA: SIAM, 2013.
38. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.