Discrete-Time Local Value Iteration Adaptive Dynamic Programming: Convergence Analysis

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Volumn 48, Issue 6, 2018, Pages 875-891

  Wei, Qinglai ^a   Lewis, Frank L ^b,c   Liu, Derong ^d   Song, Ruizhuo ^d   Lin, Hanquan ^a  

^a INSTITUTE OF AUTOMATION   (China)

^b UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

^c NORTHEASTERN UNIVERSITY   (China)

^d UNIVERSITY OF SCIENCE AND TECHNOLOGY BEIJING   (China)

Author keywords

Adaptive critic designs; adaptive dynamic programming (ADP); approximate dynamic programming; local iteration; neural networks; neuro dynamic programming; nonlinear systems; optimal control

Indexed keywords

ADAPTIVE CONTROL SYSTEMS; ITERATIVE METHODS; LEARNING ALGORITHMS; NEURAL NETWORKS; NONLINEAR SYSTEMS;

ADAPTIVE CRITIC DESIGNS; ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE DYNAMIC PROGRAMMING; LOCAL ITERATION; NEURO DYNAMIC PROGRAMMING; OPTIMAL CONTROLS;

DYNAMIC PROGRAMMING;

EID: 85046690719     PISSN: 21682216     EISSN: 21682232     Source Type: Journal    
DOI: 10.1109/TSMC.2016.2623766     Document Type: Article

References (54)

1. Z. Li, S. Xiao, S. S. Ge, and H. Su, "Constrained multilegged robot system modeling and fuzzy control with uncertain kinematics and dynamics incorporating foot force optimization," IEEE Trans. Syst., Man, Cybern., Syst., vol. 46, no. 1, pp. 1-15, Jan. 2016.
2. J. Vales-Alonso et al., "SAETA: A smart coaching assistant for professional volleyball training," IEEE Trans. Syst., Man, Cybern., Syst., vol. 45, no. 8, pp. 1138-1150, Aug. 2015.
3. X. Li, B. Shou, and D. Ralescu, "Train rescheduling with stochastic recovery time: A new track-backup approach," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 9, pp. 1216-1233, Sep. 2014.
4. P. J. Werbos, "Advanced forecasting methods for global crisis warning and models of intelligence," in General Systems Yearbook, vol. 22. 1977, pp. 25-38.
5. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA, USA: MIT Press, 1991, pp. 67-95.
6. B. Huang, Y. Li, H. Zhang, and Q. Sun, "Distributed optimal comulti-microgrids energy management for energy Internet," IEEE/CAA J. Automatica Sinica, vol. 3, no. 4, pp. 357-364, Oct. 2016.
7. Q. Wei, D. Liu, Y. Liu, and R. Song, "Optimal constrained self-learning battery sequential management in microgrid via adaptive dynamic programming," IEEE/CAA J. Automatica Sinica, to be published.
8. Y.-J. Liu, Y. Gao, S. Tong, and Y. Li, "Fuzzy approximation-based adaptive backstepping optimal control for a class of nonlinear discretetime systems with dead-zone," IEEE Trans. Fuzzy Syst., vol. 24, no. 1, pp. 16-28, Feb. 2016.
9. Y.-J. Liu and S. Tong, "Optimal control-based adaptive NN design for a class of nonlinear discrete-time block-triangular systems," IEEE Trans. Cybern., vol. 46, no. 11, pp. 2670-2680, Nov. 2016.
10. Q. Wei, R. Song, and P. Yan, "Data-driven zero-sum neuro-optimal control for a class of continuous-time unknown nonlinear systems with disturbance using ADP," IEEE Trans. Neural Netw. Learn. Syst., vol. 27, no. 2, pp. 444-458, Feb. 2016.
11. Q. Wei, F. L. Lewis, Q. Sun, P. Yan, and R. Song, "Discrete-time deterministic Q-learning: A novel convergence analysis," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2542923.
12. Q. Wei, D. Liu, Q. Lin, and R. Song, "Discrete-time optimal control via local policy iteration adaptive dynamic programming," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2586082.
13. Q. Wei and D. Liu, "A novel iterative ?-Adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1176-1190, Oct. 2014.
14. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012.
15. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
16. R. Song, F. L. Lewis, Q. Wei, and H. Zhang, "Off-policy actor-critic structure for optimal control of unknown systems with disturbances," IEEE Trans. Cybern., vol. 46, no. 5, pp. 1041-1050, May 2016.
17. H. Zhang, J. Zhang, G.-H. Yang, and Y. Luo, "Leader-based optimal coordination control for the consensus problem of multiagent differential games via fuzzy adaptive dynamic programming," IEEE Trans. Fuzzy Syst., vol. 23, no. 1, pp. 152-163, Feb. 2015.
18. R. Song, W. Xiao, H. Zhang, and C. Sun, "Adaptive dynamic programming for a class of complex-valued nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 9, pp. 1733-1739, Sep. 2014.
19. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
20. H. Zhang, C. Qin, and Y. Luo, "Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 839-849, Jul. 2014.
21. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
22. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Sci., 1996.
23. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
24. B. Lincoln and A. Rantzer, "Relaxing dynamic programming," IEEE Trans. Autom. Control, vol. 51, no. 8, pp. 1249-1260, Aug. 2006.
25. A. Rantzer, "Relaxed dynamic programming in switching systems," IEE Proc. Control Theory Appl. vol. 153, no. 5, pp. 567-574, Sep. 2006.
26. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
27. Q. Wei, F.-Y. Wang, D. Liu, and X. Yang, "Finite-Approximation-errorbased discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014.
28. D. Liu and Q. Wei, "Finite-Approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
29. H. Xu and S. Jagannathan, "Neural network-based finite horizon stochastic optimal control design for nonlinear networked control systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 3, pp. 472-485, Mar. 2015.
30. A. Heydari and S. N. Balakrishnan, "Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145-157, Jan. 2013.
31. Q. Zhao, H. Xu, and S. Jagannathan, "Neural network-based finitehorizon optimal control of uncertain affine nonlinear discrete-time systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 3, pp. 486-499, Mar. 2015.
32. Q. Wei, D. Liu, and X. Yang, "Infinite horizon self-learning optimal control of nonaffine discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 866-879, Apr. 2015.
33. Q. Wei and D. Liu, "Data-driven neuro-optimal temperature control of water-gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014.
34. Q. Wei, D. Liu, Q. Lin, and R. Song, "Adaptive dynamic programming for discrete-time zero-sum games," IEEE Trans. Neural Netw. Learn. Syst., to be published.
35. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
36. B. Kiumarsi and F. L. Lewis, "Actor-critic-based optimal tracking for partially unknown nonlinear discrete-time systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 140-151, Jan. 2015.
37. H. Zhang, T. Feng, G.-H. Yang, and H. Liang, "Distributed cooperative optimal control for multiagent systems on directed graphs: An inverse optimal approach," IEEE Trans. Cybern., vol. 45, no. 7, pp. 1315-1326, Jul. 2015.
38. R. Song et al., "Multiple actor-critic structures for continuous-time optimal control using input-output data," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 851-865, Apr. 2015.
39. R. Song, F. L. Lewis, and Q. Wei, "Off-policy integral reinforcement learning method to solve nonlinear continuous-time multiplayer nonzerosum games," IEEE Trans. Neural Netw. Learn. Syst., to be published, doi: 10.1109/TNNLS.2016.2582849.
40. Z. Ni, H. He, D. Zhao, X. Xu, and D. V. Prokhorov, "GrDHP: A general utility function representation for dual heuristic dynamic programming," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 3, pp. 614-627, Mar. 2015.
41. Q. Wei and D. Liu, "A novel policy iteration based deterministic Q-learning for discrete-time nonlinear systems," Sci. China Inf. Sci., vol. 58, no. 12, pp. 1-15, Dec. 2015.
42. R. Song, W. Xiao, and C. Sun, "A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture," Sci. China Inf. Sci., vol. 57, no. 6, pp. 1-10. Jun. 2014.
43. Y.-J. Liu, L. Tang, S. Tong, C. L. P. Chen, and D.-J. Li, "Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 165-176, Jan. 2015.
44. D. P. Bertsekas, Dynamic Programming and Optimal Control, 3rd ed. Belmont, MA, USA: Athena Sci., 2007.
45. Y. Jiang and Z.-P. Jiang, "Global adaptive dynamic programming for continuous-time nonlinear systems," IEEE Trans. Autom. Control, vol. 60, no. 11, pp. 2917-2929, Nov. 2015.
46. Q.Wei, D. Liu, and G. Shi, "A novel dual iterative Q-learning method for optimal battery management in smart residential environments," IEEE Trans. Ind. Electron., vol. 62, no. 4, pp. 2509-2518, Apr. 2015.
47. Q. Wei, D. Liu, G. Shi, and Y. Liu, "Multibattery optimal coordination control for home energy management systems via distributed iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 62, no. 7, pp. 4203-4214, Jul. 2015.
48. Q. Wei and D. Liu, "Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1020-1036, Oct. 2014.
49. Q. Wei and D. Liu, "Numerical adaptive learning control scheme for discrete-time non-linear systems," IET Control Theory Appl., vol. 7, no. 11, pp. 1472-1486, Jul. 2013.
50. Q. Wei, D. Liu, and H. Lin, "Value iteration adaptive dynamic programming for optimal control of discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 46, no. 3, pp. 840-853, Mar. 2016.
51. D. Liu, Q. Wei, and P. Yan, "Generalized policy iteration adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Syst., Man, Cybern., Syst., vol. 45, no. 12, pp. 1577-1591, Dec. 2015.
52. J. Si and Y.-T. Wang, "Online learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001.
53. T. Çimen and S. P. Banks, "Global optimal feedback control for general nonlinear systems with nonquadratic performance criteria," Syst. Control Lett., vol. 53, no. 5, pp. 327-346, Dec. 2004.
54. B. Luo, H.-N. Wu, T. Huang, and D. Liu, "Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design," Auomatica, vol. 50, no. 12, pp. 3281-3290, Dec. 2014.