Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H\infty control

IEEE Transactions on Neural Networks and Learning Systems

Volumn 23, Issue 12, 2012, Pages 1884-1895

  Wu, Huai Ning ^a   Luo, Biao ^a  

^a BEIHANG UNIVERSITY   (China)

Author keywords

H infty state feedback control; Hamilton Jacobi Isaacs equation; neural network; online; simultaneous policy update algorithm

Indexed keywords

HAMILTON-JACOBI-ISAACS; HAMILTON-JACOBI-ISAACS EQUATIONS; LEAST SQUARE METHODS; NONLINEAR PARTIAL DIFFERENTIAL EQUATIONS; ONLINE; REINFORCEMENT LEARNING TECHNIQUES; UNKNOWN ENVIRONMENTS; UPDATE ALGORITHMS;

ALGORITHMS; BANACH SPACES; LEAST SQUARES APPROXIMATIONS; NEURAL NETWORKS; NEWTON-RAPHSON METHOD; PARTIAL DIFFERENTIAL EQUATIONS; REINFORCEMENT LEARNING; STATE FEEDBACK;

NONLINEAR EQUATIONS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; HUMAN; NONLINEAR SYSTEM; PROCEDURES;

ALGORITHMS; HUMANS; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; PATTERN RECOGNITION, AUTOMATED;

EID: 84876909440     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2217349     Document Type: Article

References (49)

1. T. Ba̧sar and P. Bernhard, H∞ Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach, 2nd ed. Boston, MA: Birkhäuser, 1995.
2. A. J. van der Schaft, L2-Gain and Passivity Techniques in Nonlinear Control. Berlin, Germany: Springer-Verlag, 1996.
3. K. Zhou, J. C. Doyle, and K. Glover, Robust and Optimal Control. Upper Saddle River, NJ: Prentice-Hall, 1996.
4. A. Isidori and W. Kang, "H∞ control via measurement feedback for general nonlinear systems," IEEE Trans. Autom. Control, vol. 40, no. 3, pp. 466-472, Mar. 1995.
5. G. Bianchini, R. Genesio, A. Parenti, and A. Tesi, "Global H∞ controllers for a class of nonlinear systems," IEEE Trans. Autom. Control, vol. 49, no. 2, pp. 244-249, Feb. 2004.
6. A. J. van der Schaft, "L2-gain analysis of nonlinear systems and nonlinear state-feedback H∞ control," IEEE Trans. Autom. Control, vol. 37, no. 6, pp. 770-784, Jun. 1992.
7. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
8. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
9. Q. M. Yang, J. B. Vance, and S. Jagannathan, "Control of nonaffine nonlinear discrete-time systems using reinforcement-learning-based linearly parameterized neural networks," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 994-1001, Aug. 2008.
10. D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-time linear systems based on policy iteration," Automatica, vol. 45, no. 2, pp. 477-484, 2009.
11. D. Vrabie and F. L. Lewis, "Neural network approach to continuoustime direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, 2009.
12. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
13. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, 2010.
14. F. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear systems with ε-error bound," IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24-36, Jan. 2011.
15. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using timebased policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
16. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
17. Y. Jiang and Z. P. Jiang, "Approximate dynamic programming for optimal stationary control with control-dependent noise," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2392-2398, Dec. 2011.
18. H. Zhang, R. Song, Q. Wei, and T. Zhang, "Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1851-1862, Dec. 2011.
19. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
20. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Sep. 2009.
21. F. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
22. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. Hoboken, NJ: Wiley, 2007.
23. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
24. X. Xu, C. Liu, S. X. Yang, and D. Hu, "Hierarchical approximate policy iteration with binary-tree state space decomposition," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1863-1877, Dec. 2011.
25. M. Fairbank, E. Alonso, and D. Prokhorov, "Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 10, pp. 1671-1676, Oct. 2012.
26. Y. Feng, B. Anderson, and M. Rotkowitz, "A game theoretic algorithm to compute local stabilizing solutions to HJBI equations in nonlinear H∞ control," Automatica, vol. 45, no. 4, pp. 881-888, 2009.
27. A. Lanzon, Y. Feng, B. D. O. Anderson, and M. Rotkowitz, "Computing the positive stabilizing solution to algebraic Riccati equations with an indefinite quadratic term via a recursive method," IEEE Trans. Autom. Control, vol. 53, no. 10, pp. 2280-2291, Nov. 2008.
28. J. Huang and C. Lin, "Numerical approach to computing nonlinear H∞ control laws," AIAA J. Guidance, Control, Dynamics, vol. 18, no. 5, pp. 989-994, 1995.
29. G. N. Saridis and C. G. Lee, "An approximation theory of optimal control for trainable manipulators," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 9, no. 3, pp. 152-159, Mar. 1979.
30. R. Beard, G. N. Saridis, and J. Wen, "Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation," Automatica, vol. 33, no. 12, pp. 2159-2177, 1997.
31. R. Beard, G. N. Saridis, and J. Wen, "Approximate solutions to the timeinvariant Hamilton-Jacobi-Bellman equation," J. Optim. Theory Appl., vol. 96, no. 3, pp. 589-626, 1998.
32. R. W. Beard and T. W. Mclain, "Successive Galerkin approximation algorithms for nonlinear optimal and robust control," Int. J. Control, vol. 71, no. 5, pp. 717-743, 1998.
33. K. G. Vamvoudakis and F. L. Lewis, "Online solution of nonlinear two-player zero-sum games using synchronous policy iteration," Int. J. Robust Nonlinear Control, vol. 22, no. 13, pp. 1460-1483, 2011.
34. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Policy iterations on the Hamilton-Jacobi-Isaacs equation for H∞ state feedback control with input saturation," IEEE Trans. Autom. Control, vol. 51, no. 12, pp. 1989-1995, Dec. 2006.
35. M. Abu-Khalaf, F. L. Lewis, and J. Huang, "Neurodynamic programming and zero-sum games for constrained control systems," IEEE Trans. Neural Netw., vol. 19, no. 7, pp. 1243-1252, Jul. 2008.
36. M. Abu-Khalaf, J. Huang, and F. L. Lewis, Nonlinear H2/H-Infinity Constrained Feedback Control: A Practical Design Approach Using Neural Networks. New York: Springer-Verlag, 2006.
37. D. Vrabie and F. L. Lewis, "Adaptive dynamic programming for online solution of a zero-sum differential game," J. Control Theory Appl., vol. 9, no. 3, pp. 353-360, 2011.
38. E. Zeidler, Nonlinear Functional Analysis: Fixed Point Theorems, vol. 1. New York: Springer-Verlag, 1985.
39. L. Kantorovitch, "The method of successive approximation for functional equations," Acta Math., vol. 71, no. 1, pp. 63-97, 1939.
40. R. A. Tapia, "The Kantorovich theorem for Newton's method," Amer. Math. Monthly, vol. 78, no. 4, pp. 389-392, 1971.
41. L. B. Rall, "A note on the convergence of Newton's method," SIAM J. Numer. Anal., vol. 11, no. 1, pp. 34-36, 1974.
42. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 13, no. 5, pp. 834-846, May 1983.
43. V. Konda, "On actor-critic algorithms," Ph.D dissertation, Dept. Electr. Eng. & Comput. Sci., Massachusetts Inst. Technology, Cambridge, 2002.
44. V. Konda and J. N. Tsitsiklis, "On actor-critic algorithms," SIAM J. Control Optim., vol. 42, no. 4, pp. 1143-1166, 2003.
45. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
46. B. A. Finlayson, The Method of Weighted Residuals and Variational Principles. New York: Academic, 1972.
47. B. Stevens and F. L. Lewis, Aircraft Control and Simulation, 2nd ed. New York: Wiley, 2003.
48. K. G. Vamvoudakis, "Online learning algorithms for differential dynamic games and optimal control," Ph.D. dissertation, Faculty of Graduate School, Univ. Texas at Arlington, Arlington, 2011.
49. V. Nevisti'C and J. A. Primbs, "Constrained nonlinear optimal control: A converse HJB approach," Dept. Control & Dynamical Syst., California Inst. Technology, Pasadena, Tech. Rep. TR96-021, 1996.