Reinforcement-Learning-Based Robust Controller Design for Continuous-Time Uncertain Nonlinear Systems Subject to Input Constraints

IEEE Transactions on Cybernetics

Volumn 45, Issue 7, 2015, Pages 1372-1385

  Liu, Derong ^a   Yang, Xiong ^a   Wang, Ding ^a   Wei, Qinglai ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Approximate dynamic programming (ADP); neural networks (NNs); neuro dynamic programming; nonlinear systems; optimal control; reinforcement learning (RL); robust control

Indexed keywords

ADAPTIVE CONTROL SYSTEMS; ALGORITHMS; CONTINUOUS TIME SYSTEMS; DYNAMIC PROGRAMMING; NONLINEAR SYSTEMS; OPTIMAL CONTROL SYSTEMS; REINFORCEMENT LEARNING; ROBUST CONTROL; TELECOMMUNICATION NETWORKS;

APPROXIMATE DYNAMIC PROGRAMMING; CONSTRAINED OPTIMAL CONTROL PROBLEMS; NEURAL NETWORKS (NNS); NEURO DYNAMIC PROGRAMMING; OPTIMAL CONTROLS; REINFORCEMENT LEARNING METHOD; UNIFORM ULTIMATE BOUNDEDNESS; UNIFORMLY ULTIMATELY BOUNDED;

CONTROLLERS;

EID: 85027951427     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2015.2417170     Document Type: Article

References (56)

1. T. Hu and Z. Lin, Control Systems With Actuator Saturation: Analysis and Design. Boston, MA, USA: Birkhauser, 2001.
2. F. L. Lewis, J. Campos, and R. R. Selmic, Neuro-Fuzzy Control of Industrial Systems With Actuator Nonlinearities. Philadelphia, PA, USA: SIAM Press, 2002.
3. W. Gao and R. R. Selmic, "Neural network control of a class of nonlinear systems with actuator saturation," IEEE Trans. Neural Netw., vol. 17, no. 1, pp. 147-156, Jan. 2006.
4. H. Li and Y. Shi, "Robust distributed model predictive control of constrained continuous-time nonlinear systems: A robustness constraint approach," IEEE Trans. Autom. Control, vol. 59, no. 6, pp. 1673-1678, Jun. 2014.
5. A. E. Bryson and Y. C. Ho, Applied Optimal Control: Optimization, Estimation and Control. Boca Raton, FL, USA: CRC Press, 1975.
6. F. L. Lewis, D. Vrabie, and V. L. Syrmos, Optimal Control. Hoboken, NJ, USA: Wiley, 2012.
7. R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957.
8. P. J. Werbos, "Beyond regression: New tools for prediction and analysis in the behavioral sciences," Ph.D. dissertation, Dept. Appl. Math., Harvard University, Cambridge, MA, USA, 1974.
9. H. Zhang, D. Liu, Y. Luo, and D. Wang, Adaptive Dynamic Programming for Control: Algorithms and Stability. London, U.K.: Springer, 2013.
10. Q. Wei, F. Y. Wang, D. Liu, and X. Yang, "Finite-approximation-errorbased discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014.
11. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
12. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
13. Z. Ni, H. He, J. Wen, and X. Xu, "Goal representation heuristic dynamic programming on maze navigation," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 12, pp. 2038-2050, Dec. 2013.
14. Z. Ni, H. He, and J. Wen, "Adaptive learning in tracking control based on the dual critic network design," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 6, pp. 913-928, Jun. 2013.
15. H. Zhang, L. Cui, and Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using singlenetwork ADP," IEEE Trans. Cybern., vol. 43, no. 1, pp. 206-216, Feb. 2013.
16. H. Zhang, C. Qin, and Y. Luo, "Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 839-849, Jul. 2014.
17. Q. Wei, D. Liu, and X. Yang, "Infinite horizon self-learning optimal control of nonaffine discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 866-879, Apr. 2015.
18. S. Mehraeen, T. Dierks, S. Jagannathan, and M. L. Crow, "Zero-sum two-player game theoretic formulation of affine nonlinear discrete-time systems using neural networks," IEEE Trans. Cybern., vol. 43, no. 6, pp. 1641-1655, Dec. 2013.
19. J. Nascimento and W. B. Powell, "An optimal approximate dynamic programming algorithm for concave, scalar storage problems with vector-valued controls," IEEE Trans. Autom. Control, vol. 58, no. 12, pp. 2995-3010, Dec. 2013.
20. A. Heydari, "Revisiting approximate dynamic programming and its convergence," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2733-2743, Dec. 2014.
21. R. S. Sutton and A. G. Barto, Reinforcement Learning - An Introduction. Cambridge, MA, USA: MIT Press, 1998.
22. M. Kusy and R. Zajdel, "Application of reinforcement learning algorithms for the adaptive computation of the smoothing parameter for probabilistic neural network," IEEE Trans. Neural Netw. Learn. Syst., to be published.
23. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst. Mag., vol. 32, no. 6, pp. 76-105, Dec. 2012.
24. F. L. Lewis and K. G. Vamvoudakis, "Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 14-25, Feb. 2011.
25. X. Yang, D. Liu, and Q. Wei, "Near-optimal online control of uncertain nonlinear continuous-time systems based on concurrent learning," in Proc. Int. Joint Conf. Neural Netw., Beijing, China, Jul. 2014, pp. 231-238.
26. B. Luo, H. N. Wu, and T. Huang, "Off-policy reinforcement learning for H∞ control design," IEEE Trans. Cybern., vol. 45, no. 1, pp. 65-76, Jan. 2015.
27. X. Zhong, H. He, H. Zhang, and Z. Wang, "Optimal control for unknown discrete-time nonlinear Markov jump systems using adaptive dynamic programming," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 12, pp. 2141-2155, Dec. 2014.
28. Y. Liu, L. Tang, S. Tong, C. L. P. Chen, and D. Li, "Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 165-176, Jan. 2015.
29. J. Y. Lee, J. B. Park, and Y. H. Choi, "Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations," IEEE Trans. Neural Netw. Learn. Syst., to be published.
30. D. Wang, D. Liu, and H. Li, "Policy iteration algorithm for online design of robust control for a class of continuous-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 2, pp. 627-632, Apr. 2014.
31. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
32. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
33. S. Bhasin et al., "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
34. F. L. Lewis and D. Liu, Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Hoboken, NJ, USA: Wiley, 2013.
35. X. Yang, D. Liu, and Y. Huang, "Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints," IET Control Theory Appl., vol. 7, no. 17, pp. 2037-2047, Nov. 2013.
36. X. Yang, D. Liu, D. Wang, and Q. Wei, "Discrete-time online learning control for a class of unknown nonaffine nonlinear systems using reinforcement learning," Neural Netw., vol. 55, pp. 30-41, Jul. 2014.
37. M. Palanisamy, H. Modares, F. L. Lewis, and M. Aurangzeb, "Continuous-time Q-learning for infinite-horizon discounted cost linear quadratic regulator problems," IEEE Trans. Cybern., vol. 45, no. 2, pp. 165-176, Feb. 2015.
38. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, May 2005.
39. H. Modares, F. L. Lewis, and M. Naghibi-Sistani, "Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems," Automatica, vol. 50, no. 1, pp. 193-202, Jan. 2014.
40. H. Modares and F. L. Lewis, "Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning," Automatica, vol. 50, no. 7, pp. 1780-1792, Jul. 2014.
41. X. Yang, D. Liu, and D. Wang, "Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints," Int. J. Control, vol. 87, no. 3, pp. 553-566, 2014.
42. Y. Jiang and Z. P. Jiang, "Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics," Automatica, vol. 48, no. 10, pp. 2699-2704, Oct. 2012.
43. D. M. Adhyaru, I. N. Kar, and M. Gopal, "Bounded robust control of nonlinear systems using neural network-based HJB solution," Neural Comput. Appl., vol. 20, no. 1, pp. 91-103, Feb. 2011.
44. Y. Jiang and Z. P. Jiang, "Robust adaptive dynamic programming and feedback stabilization of nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 5, pp. 882-893, May 2014.
45. T. Dierks and S. Jagannathan, "Optimal control of affine nonlinear continuous-time systems," in Proc. Amer. Control Conf., Baltimore, MD, USA, Jun./Jul. 2010, pp. 1568-1573.
46. M. J. Corless and G. Leitmann, "Continuous state feedback guaranteeing uniform ultimate boundedness for uncertain dynamic systems," IEEE Trans. Autom. Control, vol. 26, no. 5, pp. 1139-1144, Oct. 1981.
47. F. Lin and R. D. Brandt, "An optimal control approach to robust control of robot manipulators," IEEE Trans. Robot. Autom., vol. 14, no. 1, pp. 69-77, Feb. 1998.
48. H. Khalil, Nonlinear Systems, 3rd ed. Upper Saddle River, NJ, USA: Prentice-Hall, 2002.
49. W. Rudin, Functional Analysis, 2nd ed. Singapore: McGraw-Hill, 1991.
50. R. Beard, G. Saridis, and J. Wen, "Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation," Automatica, vol. 33, no. 12, pp. 2159-2177, Dec. 1997.
51. W. Rudin, Principles of Mathematical Analysis. 3rd ed. Singapore: McGraw-Hill, 1976.
52. D. Nodland, H. Zargarzadeh, and S. Jagannathan, "Neural networkbased optimal adaptive output feedback control of a helicopter UAV," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 7, pp. 1061-1073, Jul. 2013.
53. F. L. Lewis, S. Jagannathan, and A. Yesildirek, Neural Network Control of Robot Manipulators and Nonlinear Systems. London, U.K.: Taylor and Francis, 1999.
54. K. Hornic and M. Stinchombe, "Multilayer feedforward neural networks are universal approximators," Neural Netw., vol. 2, no. 5, pp. 359-366, 1989.
55. B. A. Finlayson, The Method of Weighted Residuals and Variational Principles. New York, NY, USA: Academic Press, 1972.
56. R. Padhi, N. Unnikrishnan, X. Wang, and S. N. Balakrishnan, "A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems," Neural Netw., vol. 19, no. 10, pp. 1648-1660, Dec. 2006.