Actor-critic-based optimal tracking for partially unknown nonlinear discrete-time systems

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 1, 2015, Pages 140-151

  Kiumarsi, Bahare ^a   Lewis, Frank L ^a  

^a UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

Author keywords

Actor critic algorithm; discrete time (DT) nonlinear optimal tracking; input constraints; neural network (NN); reinforcement learning (RL)

Indexed keywords

DIGITAL CONTROL SYSTEMS; DISCRETE TIME CONTROL SYSTEMS; DYNAMIC PROGRAMMING; DYNAMICS; LEARNING ALGORITHMS; NAVIGATION; OPTIMIZATION; SOCIAL NETWORKING (ONLINE);

ACTOR-CRITIC ALGORITHM; ADAPTIVE OPTIMAL CONTROL; INPUT CONSTRAINTS; NEURAL NETWORK (NN); NON-LINEAR TRACKING PROBLEMS; NONLINEAR DISCRETE-TIME SYSTEMS; NONLINEAR OPTIMAL; TRACKING CONTROL PROBLEM;

REINFORCEMENT LEARNING;

EID: 84919687575     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2014.2358227     Document Type: Article

References (47)

1. W. B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. New York, NY, USA: Wiley, 2009.
2. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
3. J. Si, A. G. Barto, W. B. Powell, and D. Wunch, Handbook of Learning and Approximate Dynamic Programming. New York, NY, USA: Wiley, 2004.
4. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
5. F. L. Lewis, D. Vrabie, and V. Syrmos, Optimal Control, 3rd ed. New York, NY, USA: Wiley, 2012.
6. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Aug. 2009.
7. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012.
8. F. L. Lewis and D. Liu, Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Hoboken, NJ, USA: Wiley, 2013.
9. M. Fairbank, S. Li, X. Fu, E. Alonso, and D. Wunsch, "An adaptive recurrent neural-network controller using a stabilization matrix and predictive inputs to solve a tracking problem under disturbances," Neural Netw., vol. 49, pp. 74-86, Jan. 2014.
10. D. Vrabie and F. Lewis, "Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, Apr. 2009.
11. D. Liu, X. Yang, and H. Li, "Adaptive optimal control for a class of continuous-time affine nonlinear systems with unknown internal dynamics," Neural Comput. Appl., vol. 23, nos. 7-8, pp. 1843-1850, Dec. 2013.
12. S. Bhasin, R. Kamalapurkar, M. Johnson, K. Vamvoudakis, F. L. Lewis, and W. Dixon, "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
13. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
14. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
15. Y. Luo and H. Zhang, "Approximate optimal control for a class of nonlinear discrete-time systems with saturating actuators," Prog. Natural Sci., vol. 18, no. 8, pp. 1023-1029, 2008.
16. T. Dierks and S. Jagannathan, "Online optimal control of nonlinear discrete-time systems using approximate dynamic programming," J. Control Theory Appl., vol. 9, no. 3, pp. 361-369, 2011.
17. J. Si and Y.-T. Wang, "Online learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001.
18. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control. Cambridge, MA, USA: MIT Press, 1991.
19. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Reinhold, 1992.
20. P. J. Werbos, "Neural networks for control and system identification," in Proc. 28th IEEE CDC, Dec. 1989, pp. 260-265.
21. H. Zhang, D. Liu, Y. Luo, and D. Wang, Adaptive Dynamic Programming for Control: Algorithms and Stability. London, U.K.: Springer-Verlag, 2012.
22. Q. Wei and D. Liu, "Data-driven neuro-optimal temperature control of water-gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014.
23. Q. Wei and D. Liu, "A novel iterative θ-adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., to be published.
24. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
25. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
26. R. Song, W. Xiao, H. Zhang, and C. Sun, "Adaptive dynamic programming for a class of complex-valued nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 9, pp. 1733-1739, Sep. 2014.
27. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
28. D. Liu, D. Wang, and H. Li, "Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 418-428, Feb. 2014.
29. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
30. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 10, pp. 1513-1525, Oct. 2013.
31. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems," Automatica, vol. 50, no. 1, pp. 193-202, 2014.
32. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
33. T. Dierks and S. Jagannathan, "Optimal tracking control of affine nonlinear discrete-time systems with unknown internal dynamics," in Proc. 48th IEEE CDC, Dec. 2009, pp. 6750-6755.
34. Y. Huang and D. Liu, "Neural-network-based optimal tracking control scheme for a class of unknown discrete-time nonlinear systems using iterative ADP algorithm," Neurocomputing, vol. 125, pp. 46-56, Feb. 2014.
35. H. Zhang, L. Cui, X. Zhang, and X. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
36. Q. Wei and D. Liu, "Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification," IEEE Trans. Autom. Sci. Eng., to be published.
37. B. Kiumarsi, F. L. Lewis, H. Modares, A. Karimpour, and M.-B. Naghibi-Sistani, "Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics," Automatica, vol. 50, no. 4, pp. 1167-1175, 2014.
38. B. Kiumarsi-Khomartash, F. L. Lewis, M.-B. Naghibi-Sistani, and A. Karimpour, "Optimal tracking control for linear discrete-time systems using reinforcement learning," in Proc. IEEE 52nd Annu. CDC, Florence, Italy, Dec. 2013, pp. 3845-3850.
39. A. Isidori, Nonlinear Control Systems, 3rd ed. London, U.K.: Springer-Verlag, 1995.
40. S. E. Lyshevski, "Optimal control of nonlinear continuous-time systems: Design of bounded controllers via generalized nonquadratic functionals," in Proc. IEEE ACC, Jun. 1998, pp. 205-209.
41. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
42. M. Fairbank, "Value-gradient learning," Ph.D. dissertation, Dept. Comput. Sci., City Univ. London, London, U.K., 2014.
43. F. L. Lewis, S. Jagannathan, and A. Yesildirek, Neural Network Control of Robot Manipulators and Nonlinear Systems. New York, NY, USA: Taylor & Francis, 1999.
44. K. G. Vamvodakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem," Automatica, vol. 46, no. 5, pp. 878-888, 2010.
45. P. J. Werbos. (1998). "Stable adaptive control using new critic designs." [online]. Available: http://arxiv.org/abs/adap-org/9810001
46. L. Baird, "Residual algorithms: Reinforcement learning with function approximation," in Proc. 12th Int. Conf. Mach. Learn., 1995, pp. 30-37.
47. X. Luo and J. Si, "Stability of direct heuristic dynamic programming for nonlinear tracking control using PID neural network," in Proc. IJCNN, Dallas, TX, USA, Aug. 2013, pp. 1-7.