Approximate N-Player Nonzero-Sum Game Solution for an Uncertain Continuous Nonlinear System

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 8, 2015, Pages 1645-1658

  Johnson, Marcus ^a   Kamalapurkar, Rushikesh ^a   Bhasin, Shubhendu ^b   Dixon, Warren E ^a  

^a UNIVERSITY OF FLORIDA   (United States)

^b INDIAN INSTITUTE OF TECHNOLOGY DELHI   (India)

Author keywords

Actor critic (AC) methods; adaptive control; adaptive dynamic programming; differential games; optimal control

Indexed keywords

CONTINUOUS TIME SYSTEMS; GAME THEORY; LEAST SQUARES APPROXIMATIONS; NEURAL NETWORKS; OPTIMIZATION;

ACTOR CRITIC; ADAPTIVE CONTROL; ADAPTIVE DYNAMIC PROGRAMMING; DIFFERENTIAL GAMES; OPTIMAL CONTROLS;

DYNAMIC PROGRAMMING;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; NONLINEAR SYSTEM; REGRESSION ANALYSIS; SIGNAL PROCESSING; UNCERTAINTY;

ALGORITHMS; LEAST-SQUARES ANALYSIS; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; SIGNAL PROCESSING, COMPUTER-ASSISTED; UNCERTAINTY;

EID: 84937390462     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2014.2350835     Document Type: Article

References (59)

1. R. Isaacs, Differential Games: A Mathematical Theory With Applications to Warfare and Pursuit, Control and Optimization (Dover Books on Mathematics). New York, NY, USA: Dover, 1999.
2. S. Tijs, Introduction to Game Theory. Gurgaon, India: Hindustan Book Agency, 2003.
3. T. Basar and G. J. Olsder, Dynamic Noncooperative Game Theory (Classics in Applied Mathematics), 2nd ed. Philadelphia, PA, USA: SIAM, 1999.
4. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
5. A. W. Starr and Y. C. Ho, "Further properties of nonzero-sum differential games," J. Optim. Theory Appl., vol. 3, no. 4, pp. 207-219, 1969.
6. J. C. Engwerda and A. J. T. M. Weeren, "The open-loop Nash equilibrium in LQ-games revisited," Papers/Center Economic Research, Tilburg Univ., Tilburg, Netherlands, Tech. Rep., 1995, vol. 9551.
7. J. H. Case, "Toward a theory of many player differential games," SIAM J. Control, vol. 7, no. 2, pp. 179-197, 1969.
8. A. Friedman, Differential Games. New York, NY, USA: Wiley, 1971.
9. T. Başar and P. Bernhard, H8-Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach. Boston, MA, USA: Birkhäuser, 1995.
10. P. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand Reinhold, 1992.
11. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
12. D. V. Prokhorov and D. C. Wunsch, II, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, May 1997.
13. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
14. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Model-free Q-learning designs for linear discrete-time zero-sum games with application to H8 control," Automatica, vol. 43, no. 3, pp. 473-481, 2007.
15. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
16. B. Widrow, N. Gupta, and S. Maitra, "Punish/reward: Learning with a critic in adaptive threshold systems," IEEE Trans. Syst. Man Cybern., vol. SMC-3, no. 5, pp. 455-465, Sep. 1973.
17. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuron-like adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst. Man Cybern., vol. SMC-13, no. 5, pp. 834-846, Sep./Oct. 1983.
18. R. S. Sutton, A. G. Barto, and R. J. Williams, "Reinforcement learning is direct adaptive optimal control," IEEE Control Syst. Mag., vol. 12, no. 2, pp. 19-22, Apr. 1992.
19. J. Campos and F. L. Lewis, "Adaptive critic neural network for feedforward compensation," in Proc. Amer. Control Conf., vol. 4. Jun. 1999, pp. 2813-2818.
20. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Adaptive critic designs for discrete-time zero-sum games with application to H8 control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 1, pp. 240-247, Feb. 2007.
21. S. N. Balakrishnan and V. Biega, "Adaptive-critic-based neural networks for aircraft optimal control," J. Guid., Control, Dyn., vol. 19, no. 4, pp. 893-898, 1996.
22. G. G. Lendaris, L. Schultz, and T. Shannon, "Adaptive critic design for intelligent steering and speed control of a 2-axle vehicle," in Proc. Int. Joint Conf. Neural Netw., Jul. 2000, pp. 73-78.
23. S. Ferrari and R. F. Stengel, "An adaptive critic global controller," in Proc. Amer. Control Conf., vol. 4. 2002, pp. 2665-2670.
24. D. Han and S. N. Balakrishnan, "State-constrained agile missile control with adaptive-critic-based neural networks," IEEE Trans. Control Syst. Technol., vol. 10, no. 4, pp. 481-489, Jul. 2002.
25. P. He and S. Jagannathan, "Reinforcement learning neural-networkbased controller for nonlinear discrete-time systems with input constraints," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 2, pp. 425-436, Apr. 2007.
26. L. C. Baird, III, "Advantage updating," Wright Lab., Wright-Patterson AFB, OH, Tech. Rep. WL-TR-93-1146, 1993.
27. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
28. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
29. R. W. Beard, G. N. Saridis, and J. T. Wen, "Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation," Automatica, vol. 33, no. 12, pp. 2159-2178, 1997.
30. D. Vrabie and F. Lewis, "Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, 2009.
31. K. G. Vamvoudakis and F. L. Lewis, "Online synchronous policy iteration method for optimal control," in Recent Advances in Intelligent Control Systems, W. Yu, Ed. New York, NY, USA: Springer-Verlag, 2009, pp. 357-374.
32. S. Bhasin, N. Sharma, P. Patre, and W. Dixon, "Asymptotic tracking by a reinforcement learning-based adaptive critic controller," J. Control Theory Appl., vol. 9, no. 3, pp. 400-409, 2011.
33. Q. Wei and H. Zhang, "A new approach to solve a class of continuoustime nonlinear quadratic zero-sum game using ADP," in Proc. IEEE Int. Conf. Netw. Sens. Control, Apr. 2008, pp. 507-512.
34. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games," Automatica, vol. 47, no. 1, pp. 207-214, 2010.
35. X. Zhang, H. Zhang, Y. Luo, and M. Dong, "Iteration algorithm for solving the optimal strategies of a class of nonaffine nonlinear quadratic zero-sum games," in Proc. IEEE Conf. Decision Control, May 2010, pp. 1359-1364.
36. A. Mellouk, Ed., Advances in Reinforcement Learning. Winchester, U.K.: InTech, 2011.
37. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, 2010.
38. K. G. Vamvoudakis and F. L. Lewis, "Online solution of nonlinear two-player zero-sum games using synchronous policy iteration," in Proc. IEEE Conf. Decision Control, Dec. 2010, pp. 3040-3047.
39. K. G. Vamvoudakis and F. L. Lewis, "Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, pp. 1556-1569, Mar. 2011.
40. M. L. Littman, "Value-function reinforcement learning in Markov games," Cognit. Syst. Res., vol. 2, no. 1, pp. 55-66, 2001.
41. H. Zhang, L. Cui, and Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using singlenetwork ADP," IEEE Trans. Cybern., vol. 43, no. 1, pp. 206-216, Feb. 2013.
42. M. Johnson, S. Bhasin, and W. E. Dixon, "Nonlinear two-player zerosum game approximate solution using a policy iteration algorithm," in Proc. IEEE Conf. Decision Control, Dec. 2011, pp. 142-147.
43. T. Basar and P. Bernhard, H8-Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach (Modern Birkhäuser Classics), 2nd ed. Boston, MA, USA: Birkhäuser, 2008.
44. S. Bhasin, "Reinforcement learning and optimal control methods for uncertain nonlinear systems," Ph.D. dissertation, Dept. Mech. Aerosp. Eng., Univ. Florida, Gainesville, FL, USA, 2011.
45. F. L. Lewis, R. Selmic, and J. Campos, Neuro-Fuzzy Control of Industrial Systems With Actuator Nonlinearities. Philadelphia, PA, USA: SIAM, 2002.
46. W. E. Dixon, A. Behal, D. M. Dawson, and S. P. Nagarkatti, Nonlinear Control of Engineering Systems: A Lyapunov-Based Approach. Boston, MA, USA: Birkhäuser, 2003.
47. M. Krstic, P. V. Kokotovic, and I. Kanellakopoulos, Nonlinear and Adaptive Control Design. New York, NY, USA: Wiley, 1995.
48. P. M. Patre, W. MacKunis, K. Kaiser, and W. E. Dixon, "Asymptotic tracking for uncertain dynamic systems via a multilayer neural network feedforward and RISE feedback control structure," IEEE Trans. Autom. Control, vol. 53, no. 9, pp. 2180-2185, Oct. 2008.
49. A. F. Filippov, Differential Equations with Discontinuous Righthand Sides. Norwell, MA, USA: Kluwer, 1988.
50. F. H. Clarke, Optimization and Nonsmooth Analysis. Philadelphia, PA, USA: SIAM, 1990.
51. B. E. Paden and S. S. Sastry, "A calculus for computing Filippov's differential inclusion with application to the variable structure control of robot manipulators," IEEE Trans. Circuits Syst., vol. 34, no. 1, pp. 73-82, Jan. 1987.
52. N. Fischer, R. Kamalapurkar, and W. E. Dixon, "LaSalle-Yoshizawa corollaries for nonsmooth systems," IEEE Trans. Autom. Control, vol. 58, no. 9, pp. 2333-2338, Sep. 2013.
53. S. Sastry and M. Bodson, Adaptive Control: Stability, Convergence, and Robustness. Upper Saddle River, NJ, USA: Prentice-Hall, 1989.
54. R. M. Johnstone, C. R. Johnson, R. R. Bitmead, and B. D. O. Anderson, "Exponential convergence of recursive least squares with exponential forgetting factor," in Proc. IEEE Conf. Decision Control, vol. 21. Dec. 1982, pp. 994-997.
55. P. Ioannou and J. Sun, Robust Adaptive Control. Upper Saddle River, NJ, USA: Prentice-Hall, 1996.
56. A. Loría and E. Panteley, "Uniform exponential stability of linear timevarying systems: Revisited," Syst. Control Lett., vol. 47, no. 1, pp. 13-24, 2002.
57. H. K. Khalil, Nonlinear Systems, 3rd ed. Upper Saddle River, NJ, USA: Prentice-Hall, 2002.
58. V. Nevistic and J. A. Primbs, "Constrained nonlinear optimal control: A converse HJB approach," Control and Dynamical Systems Group, California Inst. Technol., Pasadena, CA, USA, Tech. Rep. CaltechCDSTR: 1996.021, 1996.
59. S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon, "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 89-92, 2013.