Infinite Horizon Self-Learning Optimal Control of Nonaffine Discrete-Time Nonlinear Systems

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 4, 2015, Pages 866-879

  Wei, Qinglai ^a   Liu, Derong ^a   Yang, Xiong ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive critic designs; adaptive dynamic programming (ADP); approximate dynamic programming; generalized policy iteration; neural networks (NNs); neurodynamic programming; nonlinear systems; optimal control; reinforcement learning

Indexed keywords

ADAPTIVE CONTROL SYSTEMS; ALGORITHMS; DISCRETE TIME CONTROL SYSTEMS; ITERATIVE METHODS; NONLINEAR SYSTEMS; REINFORCEMENT LEARNING;

ADAPTIVE CRITIC DESIGNS; ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE DYNAMIC PROGRAMMING; NEURAL NETWORKS (NNS); NEURO DYNAMIC PROGRAMMING; OPTIMAL CONTROLS; POLICY ITERATION;

DYNAMIC PROGRAMMING;

EID: 85027953921     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2015.2401334     Document Type: Article

References (72)

1. P. J. Werbos, "Advanced forecasting methods for global crisis warning and models of intelligence," General Syst. Yearbook, vol. 22, pp. 25-38, 1977.
2. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA, USA: MIT Press, 1991, pp. 67-95.
3. M. Fairbank, E. Alonso, and D. Prokhorov, "An equivalence between adaptive dynamic programming with a critic and backpropagation through time," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 12, pp. 2088-2100, Dec. 2013.
4. A. Heydari and S. N. Balakrishnan, "Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145-157, Jan. 2013.
5. K.-S. Hwang, Y.-J. Chen, and C.-J. Wu, "Fusion of multiple behaviors using layered reinforcement learning," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 42, no. 4, pp. 999-1004, Jul. 2012.
6. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
7. D. Liu, Y. Zhang, and H. Zhang, "A self-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw., vol. 16, no. 5, pp. 1219-1228, Sep. 2005.
8. S. Shervais, T. T. Shannon, and G. G. Lendaris, "Intelligent supply chain management using adaptive critic learning," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 33, no. 2, pp. 235-244, Mar. 2003.
9. X. Xu, Z. Hou, C. Lian, and H. He, "Online learning control using adaptive critic designs with sparse kernel machines," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 762-775, May 2013.
10. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
11. J. Liang, D. D. Molina, G. K. Venayagamoorthy, and R. G. Harley, "Two-level dynamic stochastic optimal power flow control for power systems with intermittent renewable generation," IEEE Trans. Power Syst., vol. 28, no. 3, pp. 2670-2678, Aug. 2013.
12. Z. Ni, H. He, J. Wen, and X. Xu, "Goal representation heuristic dynamic programming on maze navigation," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 12, pp. 2038-2050, Dec. 2013.
13. Z. Ni, H. He, and J. Wen, "Adaptive learning in tracking control based on the dual critic network design," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 6, pp. 913-928, Jun. 2013.
14. Q. Wei and D. Liu, "Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1020-1036, Oct. 2014.
15. J. Varghese and S. Mukhopadhyay, "Automated web navigation using multiagent adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 33, no. 3, pp. 412-417, May 2003.
16. Q. Wei and D. Liu, "Stable iterative adaptive dynamic programming algorithm with approximation errors for discrete-time nonlinear systems,"Neural Comput. Appl., vol. 24, no. 6, pp. 1355-1367, May 2014.
17. Q. Wei, F.-Y. Wang, D. Liu, and X. Yang, "Finite-approximation-error-based discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014.
18. Y. Jiang and Z.-P. Jiang, "Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics,"Automatica, vol. 48, no. 10, pp. 2699-2704, Oct. 2012.
19. Y. Jiang and Z.-P. Jiang, "Robust adaptive dynamic programming with an application to power systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 7, pp. 1150-1156, Jul. 2013.
20. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach," Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
21. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
22. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
23. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
24. X. Xu, C. Lian, L. Zuo, and H. He, "Kernel-based approximate dynamic programming for real-time online learning control: An experimental study," IEEE Trans. Control Syst. Technol., vol. 22, no. 1, pp. 146-156, Jan. 2014.
25. D. Molina, G. K. Venayagamoorthy, J. Liang, and R. G. Harley, "Intelligent local area signals based damping of power system oscillations using virtual generators and approximate dynamic programming,"IEEE Trans. Smart Grid, vol. 4, no. 1, pp. 498-508, Jan. 2013.
26. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
27. H. Xu and S. Jagannathan, "Stochastic optimal controller design for uncertain nonlinear networked control system via neuro dynamic programming,"IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 3, pp. 471-484, Mar. 2013.
28. R. Enns and J. Si, "Helicopter trimming and tracking control using direct neural dynamic programming," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 929-939, Jul. 2003.
29. B. Kiumarsi, F. L. Lewis, H. Modares, A. Karimpour, and M.-B. Naghibi-Sistani, "Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics,"Automatica, vol. 50, no. 4, pp. 1167-1175, Apr. 2014.
30. B. Xu, C. Yang, and Z. Shi, "Reinforcement learning output feedback NN control using deterministic learning technique," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 635-641, Mar. 2014.
31. H. Li, D. Liu, and D. Wang, "Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 706-714, Jul. 2014.
32. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
33. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern. A, Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
34. D. Liu, D. Wang, and H. Li, "Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 418-428, Feb. 2014.
35. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems,"Automatica, vol. 50, no. 1, pp. 193-202, Jan. 2014.
36. Q. Wei and D. Liu, "An iterative ∈-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, pp. 236-244, Aug. 2012.
37. Q. Wei, H. Zhang, and J. Dai, "Model-free multiobjective approximate dynamic programming for discrete-time nonlinear systems with general performance index functions," Neurocomputing, vol. 72, nos. 7-9, pp. 1839-1848, Mar. 2009.
38. Q. Wei and D. Liu, "A novel iterative θ-adaptive dynamic programming for discrete-time nonlinear systems," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 4, pp. 1176-1190, Oct. 2014.
39. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games," Automatica, vol. 47, no. 1, pp. 207-214, Jan. 2011.
40. Q. Wei and D. Liu, "Data-driven neuro-optimal temperature control of water-gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014.
41. Q. Wei and D. Liu, "Neural-network-based adaptive optimal tracking control scheme for discrete-time nonlinear systems with approximation errors," Neurocomputing, vol. 149, pp. 106-115, Feb. 2015.
42. D. Liu, D. Wang, F.-Y. Wang, H. Li, and X. Yang, "Neural-networkbased online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2834-2847, Dec. 2014.
43. X. Yang, D. Liu, and Y. Huang, "Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints," IET Control Theory Appl., vol. 7, no. 17, pp. 2037-2047, Nov. 2013.
44. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012.
45. D. Vrabie, K. G. Vamvoudakis, and F. L. Lewis, Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles. London, U.K.: The Institution of Engineering and Technology, 2013.
46. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp. 140-153, May 2002.
47. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, May 2005.
48. H. Zhang and F. L. Lewis, "Adaptive cooperative tracking control of higher-order nonlinear systems with unknown dynamics," Automatica, vol. 48, no. 7, pp. 1432-1439, Jul. 2012.
49. K. G. Vamvoudakis and F. L. Lewis, "Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton-Jacobi equations," Automatica, vol. 47, no. 8, pp. 1556-1569, Aug. 2011.
50. K. G. Vamvoudakis, F. L. Lewis, and G. R. Hudas, "Multi-agent differential graphical games: Online adaptive learning solution for synchronization with optimality," Automatica, vol. 48, no. 8, pp. 1598-1611, Aug. 2012.
51. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
52. R. Song, W. Xiao, H. Zhang, and C. Sun, "Adaptive dynamic programming for a class of complex-valued nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 9, pp. 1733-1739, Sep. 2014.
53. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
54. B. Lincoln and A. Rantzer, "Relaxing dynamic programming," IEEE Trans. Autom. Control, vol. 51, no. 8, pp. 1249-1260, Aug. 2006.
55. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
56. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
57. E. Zhou, "Optimal stopping under partial observation: Near-value iteration,"IEEE Trans. Autom. Control, vol. 58, no. 2, pp. 500-506, Feb. 2013.
58. S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon, "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems,"Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
59. H. Modares, F. L. Lewis, and M.-B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 10, pp. 1513-1525, Oct. 2013.
60. Q. Wei and D. Liu, "Numerical adaptive learning control scheme for discrete-time non-linear systems," IET Control Theory Appl., vol. 7, no. 11, pp. 1472-1486, Jul. 2013.
61. K. G. Vamvoudakis and F. L. Lewis, "Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem,"Automatica, vol. 46, no. 5, pp. 878-888, May 2010.
62. Q. Wei, D. Liu, G. Shi, and Y. Liu, "Multi-battery optimal coordination control for home energy management systems via distributed iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., to be published.
63. Q.Wei, D. Liu, and G. Shi, "A novel dual iterative Q-learning method for optimal battery management in smart residential environments," IEEE Trans. Ind. Electron., to be published.
64. B. Kiumarsi and F. L. Lewis, "Actor-critic-based optimal tracking for partially unknown nonlinear discrete-time systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 1, pp. 140-151, Jan. 2014.
65. D. Wang and D. Liu, "Neuro-optimal control for a class of unknown nonlinear dynamic systems using SN-DHP technique," Neurocomputing, vol. 121, pp. 218-225, Dec. 2013.
66. H. Li and D. Liu, "Optimal control for discrete-time affine non-linear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.
67. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
68. J. Y. Lee, J. B. Park, and Y. H. Choi, "Integral Q -learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems," Automatica, vol. 48, no. 11, pp. 2850-2859, Nov. 2012.
69. th Medit. Conf. Control Autom., Thessaloniki, Greece, Jun. 2009, pp. 1402-1409.
70. nd ed. Boston, MA, USA: Addison-Wesley, 1974.
71. J. Si and Y.-T. Wang, "Online learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001.
72. R. W. Beard, "Improving the closed-loop performance of nonlinear systems," Ph.D. dissertation, Dept. Elect. Eng., Rensselaer Polytech. Inst., Troy, NY, USA, 1995.