Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Volumn 44, Issue 8, 2014, Pages 1015-1027

  Liu, Derong ^a   Li, Hongliang ^a   Wang, Ding ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive dynamic programming (ADP); approximate dynamic programming; multiplayer nonzero sum games; neural networks; neuro dynamic programming; policy iteration

Indexed keywords

BANACH SPACES; CLOSED LOOP SYSTEMS; CONTINUOUS TIME SYSTEMS; DYNAMIC PROGRAMMING; DYNAMICAL SYSTEMS; E-LEARNING; ITERATIVE METHODS; LEARNING SYSTEMS; NEURAL NETWORKS; ONLINE SYSTEMS;

ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE DYNAMIC PROGRAMMING; MULTIPLAYERS; NEURO DYNAMIC PROGRAMMING; POLICY ITERATION;

LEARNING ALGORITHMS;

EID: 84904706555     PISSN: 21682216     EISSN: 21682232     Source Type: Journal    
DOI: 10.1109/TSMC.2013.2295351     Document Type: Article

References (57)

1. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand Reinhold, 1992, ch. 13.
2. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
3. J. Si, A. G. Barto, W. B. Powell, and D. C. Wunsch, Eds., Handbook of Learning and Approximate Dynamic Programming. New York, NY, USA: IEEE Press/Wiley, 2004.
4. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
5. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Jul. 2009.
6. S. Mohagheghi, G. K. Venayagamoorthy, and R. G. Harley, "Optimal wide area controller and state predictor for a power system," IEEE Trans. Power Syst., vol. 22, no. 2, pp. 693-705, May 2007. (Pubitemid 46746232)
7. D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
8. D. Liu, Y. Zhang, and H. Zhang, "A self-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw. (Special Issue on Adaptive Learning Systems in Comminucation Networks), vol. 16, no. 5, pp. 1219-1228, Sep. 2005. (Pubitemid 41444623)
9. F. L. Lewis and V. L. Syrmos, Optimal Control. New York, NY, USA: Wiley, 1995.
10. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997. (Pubitemid 127763331)
11. J. Si and Y. T. Wang, "On-line learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001. (Pubitemid 32371483)
12. M. Fairbank, E. Alonso, and D. Prokhorov, "Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1671-1676, Oct. 2012.
13. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
14. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
15. F. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear sytems with ∈-error bound," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1854-1862, Dec. 2011.
16. H. Li and D. Liu, "Optimal control for discrete-time affine nonlinear systems using general value iteration," IET Control Theory Applicat., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.
17. Q. Wei and D. Liu, "An iterative ∈-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, no. 6, pp. 236-244, Aug. 2012.
18. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
19. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
20. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach," Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
21. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
22. D. Wang, D. Liu, D. Zhao, Y. Huang, and D. Zhang, "A neural-networkbased iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints," Neural Comput. Applicat., vol. 22, no. 2, pp. 219-227, Feb. 2013.
23. D. Liu and Q. Wei, "Finite-approximation-error based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
24. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
25. F. L. Lewis and K. G. Vamvoudakis, "Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 14-25, Feb. 2011.
26. Q. Yang and S. Jagannathan, "Reinforcement learning controller design for affine nonlinear discrete-time systems using online approximators," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 2, pp. 377-390, Apr. 2012.
27. S. Adam, L. Busoniu, and R. Babuska, "Experience replay for real-time reinforcement learning control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 2, pp. 201-212, Mar. 2012.
28. I. Grondman, M. Vaandrager, L. Busoniu, R. Babuska, and E. Schuitema, "Efficient model learning methods for actor-critic control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 3, pp. 591-602, Jun. 2012.
29. R. A. Howard, Dynamic Programming and Markov Processes. Cambridge, MA, USA: MIT Press, 1960.
30. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002. (Pubitemid 35289398)
31. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, 779-791, May 2005. (Pubitemid 40352391)
32. T. Cheng, F. L. Lewis, and M. Abu-Khalaf, "A neural network solution for fixed-final time optimal control of nonlinear systems," Automatica, vol. 43, no. 3, pp. 482-490, Mar. 2007. (Pubitemid 46209051)
33. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-time linear systems based on policy iteration," Automatica, vol. 45, no. 2, pp. 477-484, Feb. 2009.
34. D. Vrabie and F. L. Lewis, "Neural network approach to continuoustime direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, Apr. 2009.
35. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
36. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
37. S. Bhasin, M. Johnson, and W. E. Dixon, "A model-free robust policy iteration algorithm for optimal control of nonlinear systems," in Proc. 49th IEEE Conf. Decision Control, Dec. 2010, pp. 3060-3065.
38. S. Tijs, Introduction to Game Theory. Delhi, India: Hindustan Book Agency, 2003.
39. T. Basar and G. J. Olsder, Dynamic Noncooperative Game Theory, 2nd ed. Philadelphia, PA: SIAM, 1999.
40. T. Basar and P. Bernhard, H∞ Optimal Conrol and Related Minimax Design Problems: A Dynamic Game Approach, 2nd ed. Boston, MA: Birkhäuser, 1995.
41. A. Al-Tamimi, M. Abu-Khalaf, and F. L. Lewis, "Adaptive critic designs for discretet-time zero-sum games with application to H∞ control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 1, pp. 240-247, Feb. 2007. (Pubitemid 46358495)
42. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Model-free Q-learning designs for linear discretet-time zero-sum games with application to H∞ control," Automatica, vol. 43, no. 3, pp. 473-481, Mar. 2007. (Pubitemid 46209050)
43. S. Mehraeen, T. Dierks, S. Jagannathan, and M. L. Crow, "Zero-sum two-player game theoretic formulation of affine nonlinear discrete-time systems using neural networks," in Proc. Int. Joint Conf. Neural Netw., Jul. 2010, pp. 1-8.
44. D. Liu, H. Li, and D. Wang, "H∞ control of unknown discrete-time nonlinear systems with control constraints using adaptive dynamic programming," in Proc. Int. Joint Conf. Neural Netw., Jun. 2012, pp. 3056-3061.
45. D. Varbie and F. L. Lewis, "Adaptive dynamic programming algorithm for finding online the equilibrium solution of the two-player zero-sum differential game," in Proc. Int. Joint Conf. Neural Netw., Jul. 2010, pp. 1-8.
46. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Policy iterations and the Hamilton-Jacobi-Isaacs equation for H∞ state feedback control with input saturation," IEEE Trans. Autom. Control, vol. 51, no. 12, pp. 1989-1995, Dec. 2006. (Pubitemid 46002295)
47. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Neurodynamic programming and zero-sum games for constrained control systems," IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1243-1252, Jul. 2008.
48. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games," Automatica, vol. 47, no. 1, pp. 207-214, Jan. 2011.
49. K. G. Vamvoudakis and F. L. Lewis, "Online solution of nonlinear two-player zero-sum games using synchronous policy iteration," Int. J. Robust Nonlinear Control, vol. 22, no. 13, pp. 1460-1483, 2012
50. T. Dierks and S. Jagannathan, "Optimal control of affine nonlinear continuous-time systems using an online Hamilton-Jacobi-Isaacs formulation," in Proc. IEEE Conf. Decision Control, Dec. 2010, pp. 3048-3053.
51. H. Wu and B. Luo, "Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H∞ control," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 12, pp. 1884-1895, Dec. 2012.
52. M. Johnson, S. Bhasin, and W. E. Dixon, "Nonlinear two-player zerosum game approximate solution using a policy iteration algorithm," in Proc. Conf. Decision Control Eur. Control Conf., Dec. 2011, pp. 142-147.
53. G. Freiling, G. Jank, and H. Abou-Kandil, "On global existence of solutions to coupled matrix Riccati equations in closed-loop Nash games," IEEE Trans. Autom. Control, vol. 41, no. 2, pp. 264-269, Feb. 1996. (Pubitemid 126768300)
54. D. Vrabie and F. L. Lewis, "Integral reinforcement learning for online computation of feedback Nash strategies of nonzero-sum differential games," in Proc. 49th IEEE Conf. Decision Control, Dec. 2010, pp. 3066-3071.
55. K. G. Vamvoudakis and F. L. Lewis, "Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, no. 8, pp. 1556-1569, Aug. 2011.
56. E. Zeidler, Nonlinear Functional Analysis vol. 1: Fixed Point Theorems. New York, NY, USA: Springer-Verlag, 1985.
57. V. Nevisti and J. A. Primbs, "Constrained nonlinear optimal control: A converse HJB approach," California Institute of Technology, Pasadena, CA, USA, Tech. Rep. TR96-021, 1996.