Online adaptive policy learning algorithm for H∞ state feedback control of unknown affine nonlinear discrete-time systems

IEEE Transactions on Cybernetics

Volumn 44, Issue 12, 2014, Pages 2706-2718

  Zhang, Huaguang ^a,b   Qin, Chunbin ^a,c   Jiang, Bin ^d   Luo, Yanhong ^a  

^a NORTHEASTERN UNIVERSITY   (China)

^b STATE KEY LABORATORY OF SYNTHETICAL AUTOMATION FOR PROCESS INDUSTRIES   (China)

^c HENAN UNIVERSITY   (China)

^d NANJING UNIVERSITY OF AERONAUTICS AND ASTRONAUTICS   (China)

Author keywords

Adaptive dynamic programming; H control; neural networks; nonlinear discrete time system; zero sum game

Indexed keywords

ADAPTIVE CONTROL SYSTEMS; APPROXIMATION ALGORITHMS; DIGITAL CONTROL SYSTEMS; DYNAMIC PROGRAMMING; E-LEARNING; FEEDBACK CONTROL; LEARNING ALGORITHMS; NEURAL NETWORKS; NONLINEAR FEEDBACK; ONLINE SYSTEMS; SOCIAL NETWORKING (ONLINE); STATE FEEDBACK;

ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATION ERRORS; HAMILTON-JACOBI-ISAACS; NEURAL NETWORKS (NNS); NONLINEAR DISCRETE-TIME SYSTEMS; OPTIMAL VALUE FUNCTIONS; STABILITY ANALYSIS; ZERO-SUM GAME;

DISCRETE TIME CONTROL SYSTEMS;

ALGORITHM; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; FEEDBACK SYSTEM; NONLINEAR SYSTEM; PROCEDURES; THEORETICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; FEEDBACK; MODELS, THEORETICAL; NONLINEAR DYNAMICS; PATTERN RECOGNITION, AUTOMATED;

EID: 84912048350     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2313915     Document Type: Article

References (57)

1. K. Zhou and J. C. Doyle, Essentials of Robust Control. Upper Saddle River, NJ, USA: Prentice-Hall, 1997.
2. W. Lin and C. I. Byrnes, "H control of discrete-Time nonlinear systems " IEEE Trans. Autom. Control, vol. 41, no. 4, pp. 494-510, Apr. 1996.
3. T. Basar and P. Bernhard, H Optimal Conrol and Related Minimax Design Problems: A Dynamic Game Approach, 2nd ed. Cambridge, MA, USA: Birkhaüser, 1995.
4. A. J. van der Shaft, "L2 gain analysis of nonlinear systems and nonlinear state feedback H control " IEEE Trans. Autom. Control, vol. 37, no. 6, pp. 770-784, Jun. 1992.
5. J. C. Willems, "Dissipative dynamical systems Part 1: General theory " Arch. Ration. Mech. Anal., vol. 45, no. 5, pp. 321-351, 1972.
6. B. Jiang, Z. Mao, and P. Shi, "H-infinity filter design for a class of networked control systems via T-S fuzzy model approach " IEEE Trans. Fuzzy Syst., vol. 18, no. 1, pp. 201-208, Feb. 2010.
7. J. Huang and C. F. Lin, "Numerical approach to computing nonlinear H control laws " J. Guid. Control Dyn., vol. 18, no. 5, pp. 989-994, 1995.
8. J. Huang, "An algorithm to solve the discrete HJI equation arising in L2 gain optimization problem " Int. J. Control, vol. 72, no. 1, pp. 770-784, 1999.
9. H. Zhang and G. Feng, "Stability analysis and H controller design of discrete-Time fuzzy large-scale systems based on piecewise Lyapunov functions " IEEE Trans. Syst., Man, Cybern., B Cybern., vol. 38, no. 5, pp. 1390-1401, Oct. 2008.
10. M. Liu, S. Zhang, Z. Fan, and M. Qiu, "H state estimation for discretetime chaotic systems based on a unified model " IEEE Trans. Syst., Man, Cybern., B Cybern., vol. 42, no. 4, pp. 1053-1063, Aug. 2012.
11. K. G. Vamvoudakis and F. L. Lewis, "Online solution of nonlinear two-player zero-sum games using synchronous policy iteration " Int. J. Robust Nonlinear Control, vol. 22, no. 13, pp. 1460-1483, Sep. 2012.
12. R. W. Beard and T. W. McLain, "Successive Galerkin approximation algorithms for nonlinear optimal and robust control " Int. J. Control, vol. 71, no. 5, pp. 717-743, 1998.
13. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Policy iterations and the Hamilton-Jacobi-Isaacs equation for H state feedback control with input saturation " IEEE Trans. Autom. Control, vol. 51, no. 12, pp. 1989-1995, Dec. 2006.
14. A. Lanzon, Y. Feng, B. D. O. Anderson, and M. Rotkowitz, "Computing the positive stabilizing solution to algebraic Riccati equations with an indefinite quadratic term via a recursive method " IEEE Trans. Autom. Control, vol. 53, no. 10, pp. 2280-2291, Nov. 2008.
15. P. J. Werbos, "Approximate dynamic programming for real-Time control and neural modeling " in Handbook of Intelligent Control, D. A. White and D.A. Sofge, Eds. New York, NY, USA: Van Nostrand, 1992, ch. 13, pp. 493-525.
16. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
17. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs " IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
18. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. New York, NY, USA: Wiley, 2007.
19. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control " IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Sep. 2009.
20. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction " IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
21. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-Time affine nonlinear systems with control constrains " IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
22. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-Time nonlinear HJB solution using approximate dynamic programming: Convergence proof " IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
23. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-Time optimal tracking control scheme for a class of discrete-Time nonlinear systems via the greedy HDP iteration algorithm " IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
24. T. Dierks and S. Jagannathan, "Optimal control of affine nonlinear discrete-Time systems " in Proc. Mediterranean Conf. Control Automation, Thessaloniki, Greece, Jun. 2009, pp. 1390-1395.
25. T. Dierks and S. Jagannathan, "Optimal tracking control of affine nonlinear discrete-Time systems with unknown internal dynamics " in Proc. 48th IEEE CDC 2009 & 28th CCC, Shanghai, China, pp. 6750-6755.
26. F. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-Time nonlinear sytems with-error bound " IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24-36, Jan. 2011.
27. X. Zhang, H. Zhang, Q. Sun, and Y. Luo, "Adaptive dynamic programming-based optimal control of unknown nonaffine discretetime systems with proof of convergence " Neurocomputing, vol. 91, pp. 48-55, Aug. 2012.
28. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-Time nonlinear systems using adaptive dynamic programming approach " Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
29. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-Time systems with unknown internal dynamics by using timebased policy update " IEEE Trans. Neural Netw. Learn. Syst., vol. 22, no. 7, pp. 1118-1129, Jul. 2012.
30. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming " IEEE Trans. Syst., Man, Cybern., C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
31. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach " Automatica, vol. 41, no. 5, pp. 779-791, May 2005.
32. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-Time linear systems based on policy iteration " Automatica, vol. 45, no. 2, pp. 477-484, Feb. 2009.
33. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-Time infinite horizon optimal control problem " Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
34. S. Bhasin et al., "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems " Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
35. H. Xu and S. Jagannathan, "Stochastic optimal controller design for uncertain nonlinear networked control system via neuro dynamic programming " IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 3, pp. 471-484, Mar. 2013.
36. H. Wu and B. Luo, "Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H control " IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 12, pp. 1884-1895, Dec. 2012.
37. T. Dierks and S. Jagannathan, "Optimal control of affine nonlinear continuous-Time systems using an online Hamilton-Jacobi-Isaacs formulation " in Proc. IEEE CDC, Atlanta, GA, USA, Dec. 2010, pp. 3048-3053.
38. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games " Automatica, vol. 47, no. 1, pp. 207-214, Jan. 2011.
39. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Neuro dynamic progarmming and zero-sum games for constrained control systems " IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1243-1252, Jul. 2008.
40. Y. Jiang and Z. Jiang, "Approximate dynamic programming for optimal stationary control with control-dependent noise " IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2392-2398, Dec. 2011.
41. D. Varbie and F. L. Lewis, "Adaptive dynamic programming for online solution of a zero-sum differential game " J. Contr. Theory Appl., vol. 9, no. 3, pp. 353-360, Aug. 2011.
42. H. Wu and B. Luo, "Simultaneous policy update algorithms for learning the solution of linear continuous-Time H state feedback control " Inform. Sci., vol. 222, pp. 472-485, Feb. 2013.
43. A. Al-Tamimi, M. Abu-Khalaf, and F. L. Lewis, "Adaptive critic designs for discrete-Time zero-sum games with application to H control " IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 37, no. 1, pp. 240-247, Feb. 2007.
44. Q. Wei, H. Zhang, and L. Cui, "Data-based optimal control for discretetime zero-sum games of 2-D systems using adaptive critic designs " ACTA Autom. Sinica, vol. 35, no. 6, pp. 682-692, Jun. 2009.
45. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Model-free Q-learning designs for linear discrete-Time zero-sum games with application to Hinfinity control " Automatica, vol. 43, no. 3, pp. 682-481, Mar. 2010.
46. K. H. Kim and F. L. Lewis, "Model-free H control design for unknown linear discrete-Time systems via Q-learning with LMI " Automatica, vol. 46, no. 8, pp. 1320-1326, 2010.
47. L. Cui, H. Zhang, X. Zhang, and Y. Luo, "Data-based adaptive critic design for discrete-Time zero-sum games using output feedback " in Proc. IEEE Symp. Adaptive Dynamic Programming Reinforcement Learning, Paris, France, Apr. 2011, pp. 190-195.
48. S. Mehraeen, T. Dierks, and S. Jagannathan, "Zero-sum two-player game theoretic formulation of affine nonlinear discrete-Time systems using neural networks " in Proc. Int. Joint Conf. Neural Netw., Barcelona, Spain, Jul. 2010, pp. 1-8.
49. D. Liu, H. Li, and D. Wang, "H control of unknown discrete-Time nonlinear systems with control constraints using adaptive dynamic programming " in Proc. IEEE World Congr. Comput. Intell., Brisbane, QLD, Australia, Jun. 2012, pp. 3056-3061.
50. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-Time nonlinear systems via iterative adaptive dynamic programming algorithm " Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
51. S. Mehraeen, T. Dierks, S. Jagannathan, and M. L. Crow, "Zero-sum two-player game theoretic formulation of affine nonlinear discrete-Time systems using neural networks " IEEE Trans. Cybern., vol. 43, no. 6, pp. 1641-1655, Dec. 2013.
52. M. Bardi and I. Capuzzo-Dolcetta, Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations. Cambridge, MA, USA: Birkhaüser, 1997.
53. J. Ball and W. Helton, "Viscosity solutions of Hamilton-Jacobi equations arising in nonlinear H-control " J. Math. Syst., Estimat., Control, vol. 6, no. 1, pp. 1-22, 1996.
54. S. Jagannathan, Neuarl Network Control of Nonlinear Discrete-Time Systems, Boca Raton, FL, USA: CRC Press, 2006.
55. G. Tao, "Adaptive and learning systems for signal processing, communications and control series " in Adaptive Control Design and Analysis. New York, NY, USA: Wiley, 2003.
56. H. K. Khalil, Nonlinear System. Englewood Cliffs, NJ, USA: Prentice-Hall, 2002.
57. F. L. Lewis, S. Jagannathan, and A. Yesildirek, Neural Network Control of Robot Manipulators and Nonlinear Systems. London, U.K.: Taylor & Francis, 1999.