Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems

IEEE Transactions on Cybernetics

Volumn 44, Issue 12, 2014, Pages 2834-2847

  Liu, Derong ^a   Wang, Ding ^a   Wang, Fei Yue ^a   Li, Hongliang ^a   Yang, Xiong ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive critic designs; Adaptive approximate dynamic programming (ADP); Hamilton Jacobi Bellman (HJB) equation; Neural networks; Optimal robust guaranteed cost control; Uncertain nonlinear systems

Indexed keywords

CLOSED LOOP SYSTEMS; CONTINUOUS TIME SYSTEMS; COST EFFECTIVENESS; COST FUNCTIONS; COSTS; DYNAMIC PROGRAMMING; NEURAL NETWORKS; NONLINEAR EQUATIONS; NONLINEAR SYSTEMS; ONLINE SYSTEMS; OPTIMAL CONTROL SYSTEMS; SOCIAL NETWORKING (ONLINE); UNCERTAIN SYSTEMS;

ADAPTIVE CRITIC DESIGNS; ADAPTIVE/APPROXIMATE DYNAMIC PROGRAMMING; HAMILTON-JACOBI-BELLMAN EQUATIONS; ROBUST GUARANTEED COST CONTROL; UNCERTAIN NONLINEAR SYSTEMS;

ADAPTIVE CONTROL SYSTEMS;

ALGORITHM; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; FEEDBACK SYSTEM; NONLINEAR SYSTEM; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; FEEDBACK; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS;

EID: 84912073419     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2357896     Document Type: Article

References (70)

1. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand Reinhold, 1992, ch. 13.
2. P. J. Werbos, "ADP: The key direction for future research in intelligent control and understanding brain intelligence," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 898-900, Aug. 2008.
3. P. J. Werbos, "Intelligence in the brain: A theory of how it works and how to build it," Neural Netw., vol. 22, no. 3, pp. 200-212, Apr. 2009.
4. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
5. D. P. Bertsekas, M. L. Homer, D. A. Logan, S. D. Patek, and N. R. Sandell, "Missile defense and interceptor allocation by neurodynamic programming," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 30, no. 1, pp. 42-51, Jan. 2000.
6. J. Si and Y. T. Wang, "On-line learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001.
7. S. Shervais, T. T. Shannon, and G. G. Lendaris, "Intelligent supply chain management using adaptive critic learning," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 33, no. 2, pp. 235-244, Mar. 2003.
8. J. Varghese and S. Mukhopadhyay, "Automated web navigation using multiagent adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 33, no. 3, pp. 412-417, May 2003.
9. F. Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
10. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Jul. 2009.
11. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst. Mag., vol. 32, no. 6, pp. 76-105, Dec. 2012.
12. H. Zhang, X. Zhang, Y. Luo, and J. Yang, "An overview of research on adaptive dynamic programming," Acta Autom. Sinica, vol. 39, no. 4, pp. 303-311, Apr. 2013.
13. D. Liu, H. Li, and D. Wang, "Data-based self-learning optimal control: Research progress and prospects," Acta Autom. Sinica, vol. 39, no. 11, pp. 1858-1870, Nov. 2013.
14. P. Rakshit et al., "Realization of an adaptive memetic algorithm using differential evolution and Q-learning: A case study in multirobot path planning," IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 4, pp. 814-831, Jul. 2013.
15. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
16. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
17. F. Y. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear systems with ε-error bound," IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24-36, Jan. 2011.
18. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach," Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
19. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
20. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
21. D. Wang, D. Liu, D. Zhao, Y. Huang, and D. Zhang, "A neural-network-based iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints," Neural Comput. Appl., vol. 22, no. 2, pp. 219-227, Feb. 2013.
22. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
23. H. Li and D. Liu, "Optimal control for discrete-time affine nonlinear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.
24. A. Heydari and S. N. Balakrishnan, "Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145-157, Jan. 2013.
25. Z. Ni, H. He, and J. Wen, "Adaptive learning in tracking control based on the dual critic network design," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 6, pp. 913-928, Jun. 2013.
26. H. He, Z. Ni, and J. Fu, "A three-network architecture for on-line learning and optimization based on adaptive dynamic programming," Neurocomputing, vol. 78, no. 1, pp. 3-13, Feb. 2012.
27. Z. Ni, H. He, J. Wen, and X. Xu, "Goal representation heuristic dynamic programming on maze navigation," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 12, pp. 2038-2050, Dec. 2013.
28. X. Xu, C. Lian, L. Zuo, and H. He, "Kernel-based approximate dynamic programming for real-time online learning control: An experimental study," IEEE Trans. Control Syst. Technol., vol. 22, no. 1, pp. 146-156, Jan. 2014.
29. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
30. Q. Wei and D. Liu, "Data-driven neuro-optimal temperature control of water gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014.
31. D. Liu and Q. Wei, "Policy iterative adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
32. D. Wang and D. Liu, "Neuro-optimal control for a class of unknown nonlinear dynamic systems using SN-DHP technique," Neurocomputing, vol. 121, pp. 218-225, Dec. 2013.
33. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
34. Q. Wei and D. Liu, "An iterative ε-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, no. 6, pp. 236-244, Aug. 2012.
35. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
36. D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
37. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, 779-791, May 2005.
38. D. Vrabie and F. L. Lewis, "Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, Apr. 2009.
39. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
40. S. Bhasin et al., "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
41. ∞ control," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 12, pp. 1884-1895, Dec. 2012.
42. H. Zhang, L. Cui, and Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network ADP," IEEE Trans. Cybern., vol. 43, no. 1, pp. 206-216, Feb. 2013.
43. D. Liu, X. Yang, and H. Li, "Adaptive optimal control for a class of continuous-time affine nonlinear systems with unknown internal dynamics," Neural Comput. Appl., vol. 23, pp. 1843-1850, Dec. 2013.
44. D. Liu, Y. Huang, D. Wang, and Q. Wei, "Neural-network-observer-based optimal control for unknown nonlinear systems using adaptive dynamic programming," Int. J. Control, vol. 86, no. 9, pp. 1554-1566, 2013.
45. T. Dierks and S. Jagannathan, "Optimal control of affine nonlinear continuous-time systems," in Proc. Amer. Control Conf., Baltimore, MD, USA, Jun. 2010, pp. 1568-1573.
46. D. Nodland, H. Zargarzadeh, and S. Jagannathan, "Neural network-based optimal adaptive output feedback control of a helicopter UAV," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 7, pp. 1061-1073, Jul. 2013.
47. D. Liu, D. Wang, and H. Li, "Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 418-428, Feb. 2014.
48. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
49. D. Wang, D. Liu, and H. Li, "Policy iteration algorithm for online design of robust control for a class of continuous-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 2, pp. 627-632, Apr. 2014.
50. D. Wang, D. Liu, H. Li, and H. Ma, "Neural-network-based robust optimal control design for a class of uncertain nonlinear systems via adaptive dynamic programming," Inf. Sci., vol. 282, pp. 167-179, Oct. 2014.
51. X. Yang, D. Liu, and D. Wang, "Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints," Int. J. Control, vol. 87, no. 3, pp. 553-566, Mar. 2014.
52. H. Li, D. Liu, and D. Wang, "Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 706-714, Jul. 2014.
53. H. Modares, F. L. Lewis, and M. B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 10, pp. 1513-1525, Oct. 2013.
54. ∞ control design," IEEE Trans. Cybern., to be published.
55. D. M. Adhyaru, I. N. Kar, and M. Gopal, "Bounded robust control of nonlinear systems using neural network-based HJB solution," Neural Comput. Appl., vol. 20, no. 1, pp. 91-103, 2011.
56. D. M. Adhyaru, I. N. Kar, and M. Gopal, "Fixed final time optimal control approach for bounded robust controller design using Hamilton-Jacobi-Bellman solution," IET Control Theory Appl., vol. 3, no. 9, pp. 1183-1195, Sep. 2009.
57. S. Mehraeen and S. Jagannathan, "Decentralized optimal control of a class of interconnected nonlinear discrete-time systems by using online Hamilton-Jacobi-Bellman formulation," IEEE Trans. Neural Netw., vol. 22, no. 11, pp. 1757-1769, Nov. 2011.
58. H. Xu, S. Jagannathan, and F. L. Lewis, "Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses," Automatica, vol. 48, no. 6, pp. 1017-1030, Jun. 2012.
59. J. Liang, G. K. Venayagamoorthy, and R. G. Harley, "Wide-area measurement based dynamic stochastic optimal power flow control for smart grids with high variability and uncertainty," IEEE Trans. Smart Grid, vol. 3, no. 1, pp. 59-69, Mar. 2012.
60. W. M. Haddad, V. S. Chellaboina, and J. L. Fausz, "Robust nonlinear feedback control for uncertain linear systems with nonquadratic performance criteria," Syst. Control Lett., vol. 33, no. 5, pp. 327-338, 1998.
61. W. M. Haddad, V. Chellaboina, J. L. Fausz, and A. Leonessa, "Optimal non-linear robust control for nonlinear uncertain systems," Int. J. Control, vol. 73, no. 4, pp. 329-342, 2000.
62. W. M. Haddad and V. Chellaboina, Nonlinear Dynamical Systems and Control: A Lyapunov-Based Approach. Princeton, NJ, USA: Princeton Univ. Press, 2008.
63. F. Lin, R. D. Brand, and J. Sun, "Robust control of nonlinear systems: Compensating for uncertainty," Int. J. Control, vol. 56, no. 6, pp. 1453-1459, 1992.
64. S. S. L. Chang and T. K. C. Peng, "Adaptive guaranteed cost control of systems with uncertain parameters," IEEE Trans. Autom. Control, vol. 17, no. 4, pp. 474-483, Apr. 1972.
65. L. Yu, Q. L. Han, and M. X. Sun, "Optimal guaranteed cost control of linear uncertain systems with input constraints," Int. J. Control Autom. Syst., vol. 3, no. 3, pp. 397-402, Sep. 2005.
66. L. Yu and J. Chu, "An LMI approach to guaranteed cost control of linear uncertain time-delay systems," Automatica, vol. 35, no. 6, pp. 1155-1159, Jun. 1999.
67. R. W. Beard, G. N. Saridis, and J. T. Wen, "Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation," Automatica, vol. 33, no. 12, pp. 2159-2177, Dec. 1997.
68. M. Krstic and Z. H. Li, "Inverse optimal design of input-to-state stabilizing nonlinear controllers," IEEE Trans. Autom. Control, vol. 43, no. 3, pp. 336-350, Mar. 1998.
69. M. Krstic and P. Tsiotras, "Inverse optimal stabilization of a rigid spacecraft," IEEE Trans. Autom. Control, vol. 44, no. 5, pp. 1042-1049, May 1999.
70. F. L. Lewis, S. Jagannathan, and A. Yesildirek, Neural Network Control of Robot Manipulators and Nonlinear Systems. London, U.K.: Taylor & Francis, 1999.