Model-Free Optimal Control for Affine Nonlinear Systems with Convergence Analysis

IEEE Transactions on Automation Science and Engineering

Volumn 12, Issue 4, 2015, Pages 1461-1468

  Zhao, Dongbin ^a   Xia, Zhongpu ^a   Wang, Ding ^a  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

Author keywords

Action dependent heuristic dynamic programming; adaptive dynamic programming; model free optimal control; neural networks; policy iteration

Indexed keywords

HEURISTIC PROGRAMMING; ITERATIVE METHODS; NETWORK LAYERS; NEURAL NETWORKS; NONLINEAR SYSTEMS; OPTIMIZATION;

ACTION DEPENDENT HEURISTIC DYNAMIC PROGRAMMING; AFFINE NONLINEAR SYSTEMS; BELLMAN EQUATIONS; CONVERGENCE ANALYSIS; OPTIMAL CONTROL POLICY; SELF-LEARNING CONTROL; SIMULATION EXAMPLE; THREE-LAYER PERCEPTRON;

DYNAMIC PROGRAMMING;

EID: 84960449514     PISSN: 15455955     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASE.2014.2348991     Document Type: Article

References (25)

1. F. L. Lewis, D. Vrabie, V. L. Syrmos, Optimal Control, 3rd ed. New York, NY, USA: Wiley, 2012.
2. A. Al-Tamimi, F. L. Lewis, M. Abu-Khalaf, "Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control," Automatica, vol. 43, no. 3, pp. 473-481, 2007.
3. D. Liu, D. Wang, D. Zhao, Q. Wei, N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
4. D. Wang, D. Liu, Q. Wei, D. Zhao, N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, 2012.
5. F. Wang, H. Zhang, D. Liu, "Adaptive dynamic programming: an introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, 2009.
6. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, Adaptative Approaches, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand Reinhold, 1992, pp. 493-525.
7. H. Zhang, D. Liu, Y. Luo, D. Wang, Adaptive Dynamic Programming for Control: Algorithms and Stability. London, U.K.: Springer, 2013.
8. A. Al-Tamimi, F. L. Lewis, M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug., 2008.
9. D. Zhao, X. Bai, F.-Y. Wang, J. Xu, W. Yu, "DHP method for ramp metering of freeway traffic," IEEE Trans. Intell. Transp. Syst., vol. 12, no. 4, pp. 990-999, Dec. 2011.
10. G. G. Yen and P. G. DeLima, "Improving the performance of globalized dual heuristic programming for fault tolerant control through an online learning supervisor," IEEE Trans. Autom. Sci. Eng., vol. 2, no. 2, pp. 121-131, Apr. 2005.
11. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
12. D. Vrabie and F. Lewis, "Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems," Neural Netw., vol. 22, no. 3, pp. 237-246, 2009.
13. H. Zhang, L. Cui, Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using singlenetwork ADP," IEEE Trans. Syst., Man, Cybern. B, vol. 43, no. 1, pp. 206-216, Feb. 2013.
14. H. Li and D. Liu, "Optimal control for discrete-time affine non-linear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, 2012.
15. H. Zhang, Y. Luo, D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
16. S. Adam, L. Busoniu, R. Babuska, "Experience replay for real-time reinforcement learning control," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 42, no. 2, pp. 201-212, Mar. 2012.
17. J. Si and Y.-T. Wang, "Online learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001.
18. R. S. Sutton and A. G. Barto, Reinforcement learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998, no. 1.
19. D. Zhao, Z. Hu, Z. Xia, C. Alippi, Y. Zhu, D. Wang, "Full-range adaptive cruise control based on supervised adaptive dynamic programming," Neurocomputing, vol. 125, pp. 57-67, 2014.
20. D. Zhao, B. Wang, D. Liu, "A supervised Actor-Critic approach for adaptive cruise control," Soft Comput., vol. 17, no. 11, pp. 2089-2099, 2013.
21. D. Zhao and Z. Xia, "Adaptive optimal control for the uncertain driving habit problem in adaptive cruise control system," in Proc. IEEE Int. Conf. Veh. Electron. Safety, 2013, pp. 159-164.
22. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res., vol. 4, pp. 1107-1149, 2003.
23. F. L. Lewis, D. Vrabie, K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst. Mag., vol. 32, no. 6, pp. 76-105, Dec. 2012.
24. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
25. C. J. C. H. Watkins, "Learning from delayed rewards," Ph.D. dissertation, Univ. Cambridge, Cambridge, U.K., 1989.