Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems

IEEE Transactions on Neural Networks and Learning Systems

Volumn 25, Issue 3, 2014, Pages 621-634

  Liu, Derong ^a   Wei, Qinglai ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive critic designs; adaptive dynamic programming (ADP); approximate dynamic programming; discrete time policy iteration; neural networks; neurodynamic programming; nonlinear systems; optimal control; reinforcement learning

Indexed keywords

ADAPTIVE CRITIC DESIGNS; ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE DYNAMIC PROGRAMMING; NEURO DYNAMIC PROGRAMMING; OPTIMAL CONTROLS; POLICY ITERATION;

ALGORITHMS; CONTROL; CONTROL THEORY; DYNAMIC PROGRAMMING; NEURAL NETWORKS; NONLINEAR SYSTEMS; NUMERICAL METHODS; OPTIMAL CONTROL SYSTEMS; REINFORCEMENT LEARNING; STABILITY;

ITERATIVE METHODS;

EID: 84897594646     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2281663     Document Type: Article

References (50)

1. R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957.
2. P. J. Werbos, "Advanced forecasting methods for global crisis warning and models of intelligence," General Syst. Yearbook, vol. 22, pp. 25-38, Jan. 1977.
3. P. J. Werbos, "A menu of designs for reinforcement learning over time," in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Ed., Cambridge, MA, USA: MIT Press, 1991, pp. 67-95.
4. T. Dierks and S. Jagannathan, "Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using timebased policy update," IEEE Trans. Neurl Netw. Learn. Syst., vol. 23, no. 7, pp. 1118-1129, Jul. 2012.
5. M. Fairbank, E. Alonso, and D. Prokhorov, "Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 10, pp. 1671-1676, Oct. 2012.
6. Y. Jiang and Z. P. Jiang, "Robust adaptive dynamic programming with an application to power systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 7, pp. 1150-1156, Jul. 2013.
7. D. Liu, Y. Zhang, and H. Zhang, "A self-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw., vol. 16, no. 5, pp. 1219-1228, Sep. 2005. (Pubitemid 41444623)
8. Z. Ni, H. He, and J. Wen, "Adaptive learning in tracking control based on the dual critic network design," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 6, pp. 913-928, Jun. 2013.
9. D. Nodland, H. Zargarzadeh, and S. Jagannathan, "Neural networkbased optimal adaptive output feedback control of a helicopter UAV," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 7, pp. 1061-1073, Jul. 2013.
10. D. Wang, D. Liu, D. Zhao, Y. Huang, and D. Zhang, "A neuralnetwork-based iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints," Neural Comput. Appl., vol. 22, no. 2, pp. 219-227, Feb. 2013.
11. H. Xu, S. Jagannathan, and F. L. Lewis, "Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses," Automatica, vol. 48, no. 6, pp. 1017-1030, Jun. 2012.
12. H. Xu and S. Jagannathan, "Stochastic optimal controller design for uncertain nonlinear networked control system via neuro dynamic programming," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 3, pp. 471-484, Mar. 2013.
13. J. Liang, G. K. Venayagamoorthy, and R. G. Harley, "Wide-area measurement based dynamic stochastic optimal power flow control for smart grids with high variability and uncertainty," IEEE Trans. Smart Grid, vol. 3, no. 1, pp. 59-69, Mar. 2012.
14. D. V. Prokhorov and D. C. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997. (Pubitemid 127763331)
15. X. Xu, Z. Hou, C. Lian, and H. He, "Online learning control using adaptive critic designs with sparse kernel machines," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 762-775, May 2013.
16. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
17. J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, "Adaptive dynamic programming," IEEE Trans. Syst., Man, Cybern., Part C, Appl, Rev., vol. 32, no. 2, pp. 140-153, May 2002. (Pubitemid 35289398)
18. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
19. Q. Wei and D. Liu, "An iterative optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state," Neural Netw., vol. 32, pp. 236-244, Aug. 2012.
20. Y. Jiang and Z. P. Jiang, "Approximate dynamic programming for optimal stationary control with control-dependent noise," IEEE Trans. Neurl Netw., vol. 22, no. 12, pp. 2392-2398, Dec. 2011.
21. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Ed., New York, NY, USA: Van Nostrand Reinhold, 1992, ch. 13.
22. R. Enns and J. Si, "Helicopter trimming and tracking control using direct neural dynamic programming," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 929-939, Aug. 2003.
23. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
24. J. Si and Y.-T. Wang, "On-line learning control by association and reinforcement," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264-276, Mar. 2001. (Pubitemid 32371483)
25. M. Geist and O. Pietquin, "Algorithmic survey of parametric value function approximation," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 6, pp. 845-867, Jun. 2013.
26. K. S. Hwang and C. Y. Lo, "Policy improvement by a model-free Dyna architecture," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 776-788, May 2013.
27. C. Watkins, "Learning from delayed rewards," Ph.D. dissertation, Dept. Comput. Sci., Cambridge Univ., Cambridge, U.K., 1989.
28. T. Huang and D. Liu, "A self-learning scheme for residential energy system control and management," Neural Comput. Appl., vol. 22, no. 2, pp. 259-269, Feb. 2013.
29. H. Li and D. Liu, "Optimal control for discrete-time affine nonlinear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.
30. D. Liu and Q. Wei, "Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
31. H. Modares, F. L. Lewis, and M. B. Naghibi-Sistani, "Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks," IEEE Trans. Neural Netw. Learn. Syst., Aug. 2013, to be published.
32. F. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear systems with error bound," J. Control Theory Appl., vol. 22, no. 1, pp. 24-36, Jan. 2011.
33. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach," Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
34. Q. Wei, H. Zhang, and J. Dai, "Model-free multiobjective approximate dynamic programming for discrete-time nonlinear systems with general performance index functions," Neurocomputing, vol. 72, nos. 7-9, pp. 1839-1848, 2009.
35. H. Zhang, R. Song, Q. Wei, and T. Zhang, "Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1851-1862, Dec. 2011.
36. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Jul. 2009.
37. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012.
38. R. Beard, "Improving the closed-loop performance of nonlinear systems," Ph.D. dissertation, Rensselaer Polytechnic Inst., Troy, NY, USA, 1995.
39. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern., Part B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
40. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern., Part B, Cybern., vol. 38, no. 4, pp. 937-942, Jul. 2008.
41. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
42. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discretetime nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
43. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, May 2005. (Pubitemid 40352391)
44. H. Zhang, Q. Wei, and D. Liu, "An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games," Automatica, vol. 47, no. 1, pp. 207-214, Jan. 2011.
45. K. G. Vamvoudakis, F. L. Lewis, and G. R. Hudas, "Multi-agent differential graphical games: Online adaptive learning solution for synchronization with optimality," Automatica, vol. 48, no. 8, pp. 1598-1611, 2012.
46. S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon, "A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems," Automatica, vol. 49, no. 1, pp. 82-92, Jan. 2013.
47. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
48. S. K. Gupta, Numerical Methods for Engineerings. New Delhi, India: New Age Int. Company, 1995.
49. A. Heydari and S. N. Balakrishnan, "Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145-157, Jan. 2013.
50. J. R. Wertz, Spacecraft Attitude Determination and Control. Amsterdam, The Netherlands: Kluwer, 1978, pp. 521-570.