Discrete-time deterministic Q-learning: A novel convergence analysis

IEEE Transactions on Cybernetics

Volumn 47, Issue 5, 2017, Pages 1224-1237

  Wei, Qinglai ^a   Lewis, Frank L ^b,c   Sun, Qiuye ^d   Yan, Pengfei ^a   Song, Ruizhuo ^e  

^a INSTITUTE OF AUTOMATION   (China)

^b UNIVERSITY OF TEXAS AT ARLINGTON   (United States)

^c NORTHEASTERN UNIVERSITY   (China)

^d NORTHEASTERN UNIVERSITY   (China)

^e UNIVERSITY OF SCIENCE AND TECHNOLOGY BEIJING   (China)

Author keywords

Adaptive critic designs; adaptive dynamic programming (ADP); approximate dynamic programming; neural networks (NNs); neuro dynamic programming; optimal control; Q learning

Indexed keywords

ALGORITHMS; ITERATIVE METHODS;

CONTROL SPACES; CONVERGENCE ANALYSIS; CONVERGENCE CRITERION; CONVERGENCE PROPERTIES; ITERATIVE CONTROL; LEARNING RATES; Q-LEARNING ALGORITHMS; UPPER AND LOWER BOUNDS;

LEARNING ALGORITHMS;

EID: 84963604827     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2016.2542923     Document Type: Article

References (62)

1. P. J. Werbos, "Advanced forecasting methods for global crisis warning, and models of intelligence," Gen. Syst. Yearbook, vol. 22, pp. 25-38, 1977
2. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA, USA: MIT Press, 1991, pp. 67-95
3. X. Zhong, and H. He, "An event-triggered ADP control approach for continuous-time system with unknown internal states," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2523878
4. R. Song, W. Xiao, H. Zhang, and C. Sun, "Adaptive dynamic programming for a class of complex-valued nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 9, pp. 1733-1739, Sep. 2014
5. Q. Wei, R. Song, and P. Yan, "Data-driven zero-sum neuro-optimal control for a class of continuous-time unknown nonlinear systems with disturbance using ADP," IEEE Trans. Neural Netw. Learn. Syst., vol. 27, no. 2, pp. 444-458, Feb. 2016
6. Z. Ni, H. He, D. Zhao, X. Xu, and D. V. Prokhorov, "GrDHP: A general utility function representation for dual heuristic dynamic programming," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 3, pp. 614-627, Mar. 2015
7. Q. Wei, and D. Liu, "Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1020-1036, Oct. 2014
8. Z. Ni, H. He, J. Wen, and X. Xu, "Goal representation heuristic dynamic programming on maze navigation," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 12, pp. 2038-2050, Dec. 2013
9. D. P. Bertsekas, "Value, and policy iterations in optimal control, and adaptive dynamic programming," IEEE Trans. Neural Netw. Learn. Syst., to be published, doi: 10.1109/TNNLS.2015.2503980
10. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning, and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012
11. D. Vrabie, K. G. Vamvoudakis, and F. L. Lewis, Optimal Adaptive Control, and Differential Games by Reinforcement Learning Principles. London, U.K.: Inst. Eng. Technol., 2013
12. A. Heydari, "Revisiting approximate dynamic programming, and its convergence," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2733-2743, Dec. 2014
13. T. Dierks, and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using timebased policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012
14. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Integral reinforcement learning, and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems," Automatica, vol. 50, no. 1, pp. 193-202, Jan. 2014
15. H. Modares, and F. L. Lewis, "Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning," IEEE Trans. Autom. Control, vol. 59, no. 11, pp. 3051-3056, Nov. 2014
16. Q. Wei, F. Y. Wang, D. Liu, and X. Yang, "Finite-approximation-errorbased discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014
17. K. G. Vamvoudakis, M. F. Miranda, and J. P. Hespanha, "Asymptotically stable adaptive-optimal control algorithm with saturating actuators, and relaxed persistence of excitation," IEEE Trans. Neural Netw. Learn. Syst., to be published, doi: 10.1109/TNNLS.2015.2487972
18. K. G. Vamvoudakis, and F. L. Lewis, "Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, no. 8, pp. 1556-1569, Aug. 2011
19. H. Zhang, L. Cui, and Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using singlenetwork ADP," IEEE Trans. Cybern., vol. 43, no. 1, pp. 206-216, Feb. 2013
20. R. Song, et al., "Multiple actor-critic structures for continuous-time optimal control using input-output data," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 851-865, Apr. 2015
21. H. Modares, and F. L. Lewis, "Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning," Automatica, vol. 50, no. 7, pp. 1780-1792, Jul. 2014
22. D. Liu, and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014
23. B. Lincoln, and A. Rantzer, "Relaxing dynamic programming," IEEE Trans. Autom. Control, vol. 51, no. 8, pp. 1249-1260, Aug. 2006
24. Q. Wei, D. Liu, G. Shi, and Y. Liu, "Multibattery optimal coordination control for home energy management systems via distributed iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 62, no. 7, pp. 4203-4214, Jul. 2015
25. Q. Wei, D. Liu, and G. Shi, "A novel dual iterative Q-learning method for optimal battery management in smart residential environments," IEEE Trans. Ind. Electron., vol. 62, no. 4, pp. 2509-2518, Apr. 2015
26. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008
27. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008
28. J. M. Lee, and J. H. Lee, "Approximate dynamic programming-based approaches for input-output data-driven control of nonlinear processes," Automatica, vol. 41, no. 7, pp. 1281-1288, Jul. 2005
29. L. Busoniu, R. Babuska, B. D. Schutter, and D. Ernst, Reinforcement Learning, and Dynamic Programming Using Function Approximators. Boca Raton, FL, USA: CRC Press, 2010
30. H. Modares, F. L. Lewis, and M. B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration, and neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 10, pp. 1513-1525, Oct. 2013
31. Q. Wei, and D. Liu, "A novel iterative -adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1176-1190, Oct. 2014
32. Q. Wei, and D. Liu, "Data-driven neuro-optimal temperature control of water-gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014
33. R. Song, F. L. Lewis, Q. Wei, and H. Zhang, "Off-policy actor-critic structure for optimal control of unknown systems with disturbances," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2015.2421338
34. C. Watkins, "Learning from delayed rewards," Ph.D. dissertation, Cambridge Univ., Cambridge, U.K., 1989
35. C. Watkins, and P. Dayan, "Q-learning," Mach. Learn., vol. 8, nos. 3-4, pp. 279-292, May 1992
36. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control," Automatica, vol. 43, no. 3, pp. 473-481, Mar. 2007
37. J.-H. Kim, and F. L. Lewis, "Model-free H control design for unknown linear discrete-time systems via Q-learning with LMI," Automatica, vol. 46, no. 8, pp. 1320-1326, Aug. 2010
38. A. Konar, I. G. Chakraborty, S. J. Singh, L. C. Jain, and A. K. Nagar, "A deterministic improved Q-learning for path planning of a mobile robot," IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 5, pp. 1141-1153, Sep. 2013
39. D. V. Prokhorov, and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997
40. M. Rahimiyan, and H. R. Mashhadi, "An adaptive Q-learning algorithm developed for agent-based computational modeling of electricity market," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 40, no. 5, pp. 547-556, Sep. 2010
41. L. A. Prashanth, and S. Bhatnagar, "Reinforcement learning with function approximation for traffic signal control," IEEE Trans. Intell. Transp. Syst., vol. 12, no. 2, pp. 412-421, Jun. 2011
42. S. Kar, J. M. F. Moura, and H. V. Poor, "QD-learning: A collaborative distributed strategy for multi-agent reinforcement learning through consensus + innovations," IEEE Trans. Signal Process., vol. 61, no. 7, pp. 1848-1862, Jul. 2013
43. T. Huang, and D. Liu, "A self-learning scheme for residential energy system control, and management," Neural Comput. Appl., vol. 22, no. 2, pp. 259-269, Feb. 2013
44. Q. Sun, J. Zhou, J. M. Guerrero, and H. Zhang, "Hybrid threephase/single-phase microgrid architecture with power management capabilities," IEEE Trans. Power Electron., vol. 30, no. 10, pp. 5964-5977, Oct. 2015
45. Q. Sun, R. Han, H. Zhang, J. Zhou, and J. M. Guerrero, "A multiagent-based consensus algorithm for distributed coordinated control of distributed generators in the energy Internet," IEEE Trans. Smart Grid, vol. 6, no. 6, pp. 3006-3019, Nov. 2015, doi: 10.1109/TSG.2015.2412779
46. Q. Sun, Y. Zhang, H. He, D. Ma, and H. Zhang, "A novel energy function-based stability evaluation, and nonlinear control for energy Internet," IEEE Trans. Smart Grid, to be published, doi: 10.1109/TSG.2015.2497691
47. J. Ni, M. Liu, L. Ren, and S. X. Yang, "A multiagent Q-learningbased optimal allocation approach for urban water resource management system," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 1, pp. 204-214, Jan. 2014
48. R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957
49. Y.-J. Liu, and S. C. Tong, "Adaptive neural network tracking control of uncertain nonlinear discrete-time systems with nonaffine dead-zone input," IEEE Trans. Cybern., vol. 45, no. 3, pp. 497-505, Mar. 2015
50. Y. J. Liu, Y. Gao, S. C. Tong, and C. L. P. Chen, "A unified approach to adaptive neural control for nonlinear discrete-time systems with nonlinear dead-zone input," IEEE Trans. Neural Netw. Learn. Syst., vol. 27, no. 1, pp. 139-150, Jan. 2016
51. Y. J. Liu, L. Tang, S. C. Tong, C. L. P. Chen, and D. J. Li, "Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 165-176, Jan. 2015
52. J. Yu, M. Tan, J. Chen, and J. Zhang, "A survey on CPG-inspired control models, and system implementation," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 441-456, Mar. 2014
53. R. C. Dorf, and R. H. Bishop, Modern Control System. New York, NY, USA: Prentice Hall, 2011
54. R. W. Beard, "Improving the closed-loop performance of nonlinear systems," Ph.D. dissertation, Dept. Electr. Eng., Rensselaer Polytech. Inst., Troy, NY, USA, 1995
55. Q. Zhao, H. Xu, and S. Jagannathan, "Finite-horizon near optimal adaptive control of uncertain linear discrete-time systems," Optimal Control Appl. Methods, vol. 36, no. 6, pp. 853-872, 2015, doi: 10.1002/oca.2143
56. H. Xu, S. Jagannathan, and F. L. Lewis, "Stochastic optimal output feedback design for unknown linear discrete-time system zero-sum games under communication constraints," Asian J. Control, vol. 16, no. 5, pp. 1263-1276, Sep. 2014
57. H. Xu, S. Jagannathan, and F. L. Lewis, "Stochastic optimal control of unknown linear networked control systems in the presence of random delays, and packet losses," Automatica, vol. 48, no. 6, pp. 1017-1030, Jun. 2012
58. Y. Jiang, and Z. P. Jiang, "Global adaptive dynamic programming for continuous-time nonlinear systems," IEEE Trans. Autom. Control, vol. 60, no. 11, pp. 2917-2929, Nov. 2015
59. T. Bian, Y. Jiang, and Z.-P. Jiang, "Adaptive dynamic programming, and optimal control of nonlinear nonaffine systems," Automatica, vol. 50, no. 10, pp. 2624-2632, Oct. 2014
60. H. Zhang, T. Feng, G. H. Yang, and H. Liang, "Distributed cooperative optimal control for multiagent systems on directed graphs: An inverse optimal approach," IEEE Trans. Cybern., vol. 45, no. 7, pp. 1315-1326, Jul. 2015
61. Q. Wei, D. Liu, and H. Lin, "Value iteration adaptive dynamic programming for optimal control of discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 46, no. 3, pp. 840-853, Mar. 2016
62. Q. Wei, D. Liu, and X. Yang, "Infinite horizon self-learning optimal control of nonaffine discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 866-879, Apr. 2015