Error bounds of adaptive dynamic programming algorithms for solving undiscounted optimal control problems

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 6, 2015, Pages 1323-1334

  Liu, Derong ^a   Li, Hongliang ^a   Wang, Ding ^a  

^a INSTITUTE OF AUTOMATION   (China)

Author keywords

Adaptive critic designs; Adaptive dynamic programming (ADP); Approximate dynamic programming; Neural networks; Neurodynamic programming; Nonlinear systems; Optimal control

Indexed keywords

ADAPTIVE CONTROL SYSTEMS; ALGORITHMS; DISCRETE TIME CONTROL SYSTEMS; ERROR ANALYSIS; ERRORS; ITERATIVE METHODS; NEURAL NETWORKS; NONLINEAR SYSTEMS; OPTIMAL CONTROL SYSTEMS;

ADAPTIVE CRITIC DESIGNS; ADAPTIVE DYNAMIC PROGRAMMING; APPROXIMATE DYNAMIC PROGRAMMING; NEURO DYNAMIC PROGRAMMING; OPTIMAL CONTROLS;

DYNAMIC PROGRAMMING;

ALGORITHM; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; NONLINEAR SYSTEM; PROBLEM SOLVING; TIME FACTOR;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; PROBLEM SOLVING; TIME FACTORS;

EID: 84930194751     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2015.2402203     Document Type: Article

References (62)

1. D. P. Bertsekas, Dynamic Programming and Optimal Control, 2nd ed. Belmont, MA, USA: Athena Scientific, 2012.
2. R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957.
3. F.-Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
4. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand, 1992, ch. 13.
5. J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Eds., Handbook of Learning and Approximate Dynamic Programming. New York, NY, USA: IEEE Press, 2004.
6. F. L. Lewis and D. Liu, Eds., Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Hoboken, NJ, USA: Wiley, 2013.
7. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
8. D. Liu, Y. Zhang, and H. Zhang, "A self-learning call admission control scheme for CDMA cellular networks," IEEE Trans. Neural Netw., vol. 16, no. 5, pp. 1219-1228, Sep. 2005.
9. D. Liu, H. Javaherian, O. Kovalenko, and T. Huang, "Adaptive critic learning techniques for engine torque and air-fuel ratio control," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 988-993, Aug. 2008.
10. T. Huang and D. Liu, "A self-learning scheme for residential energy system control and management," Neural Comput. Appl., vol. 22, no. 2, pp. 259-269, Feb. 2013.
11. Q. Wei and D. Liu, "Data-driven neuro-optimal temperature control of water-gas shift reaction using stable iterative adaptive dynamic programming," IEEE Trans. Ind. Electron., vol. 61, no. 11, pp. 6399-6408, Nov. 2014.
12. D. Liu, H. Li, and D. Wang, "Neural-network-based zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm," Neurocomputing, vol. 110, pp. 92-100, Jun. 2013.
13. H. Li, D. Liu, and D. Wang, "Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics," IEEE Trans. Autom. Sci. Eng., vol. 11, no. 3, pp. 706-714, Jul. 2014.
14. D. Liu, H. Li, and D. Wang, "Online synchronous approximate optimal learning algorithm for multi-player non-zero-sum games with unknown dynamics," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1015-1027, Aug. 2014.
15. D. Liu, D. Wang, and H. Li, "Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 418-428, Feb. 2014.
16. H. Zhang, J. Zhang, G.-H. Yang, and Y. Luo, "Leader-based optimal coordination control for the consensus problem of multiagent differential games via fuzzy adaptive dynamic programming," IEEE Trans. Fuzzy Syst., vol. 23, no. 1, pp. 152-163, Feb. 2015.
17. F. L. Lewis and D. Vrabie, "Reinforcement learning and adaptive dynamic programming for feedback control," IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32-50, Jul. 2009.
18. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Circuits Syst. Mag., vol. 32, no. 6, pp. 76-105, Dec. 2012.
19. R. S. Sutton and A. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
20. M. L. Puterman and M. C. Shin, "Modifed policy iteration algorithms for discounted Markov decision problems," Manage. Sci., vol. 24, no. 11, pp. 1127-1137, 1978.
21. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
22. T. Dierks, B. T. Thumati, and S. Jagannathan, "Optimal control of unknown affine nonlinear discrete-time systems using offline-trained neural networks with proof of convergence," Neural Netw., vol. 22, nos. 5-6, pp. 851-860, Jul./Aug. 2009.
23. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
24. D. Liu, D. Wang, and X. Yang, "An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs," Inf. Sci., vol. 220, pp. 331-342, Jan. 2013.
25. F.-Y. Wang, N. Jin, D. Liu, and Q. Wei, "Adaptive dynamic programming for finite-horizon optimal control of discrete-time nonlinear systems with ε-error bound," IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24-36, Jan. 2011.
26. A. Heydari and S. N. Balakrishnan, "Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145-157, Jan. 2013.
27. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
28. D. Wang, D. Liu, and Q. Wei, "Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach," Neurocomputing, vol. 78, no. 1, pp. 14-22, Feb. 2012.
29. H. Zhang, R. Song, Q. Wei, and T. Zhang, "Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1851-1862, Dec. 2011.
30. D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, "Neural-network-based optimal control for a class of unknown discrete-time nonlinear systems using globalized dual heuristic programming," IEEE Trans. Autom. Sci. Eng., vol. 9, no. 3, pp. 628-634, Jul. 2012.
31. D. Wang, D. Liu, Q. Wei, D. Zhao, and N. Jin, "Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming," Automatica, vol. 48, no. 8, pp. 1825-1832, Aug. 2012.
32. H. Zhang, D. Liu, Y. Luo, and D. Wang, Adaptive Dynamic Programming for Control: Algorithms and Stability. London, U.K.: Springer-Verlag, 2013.
33. R. A. Howard, Dynamic Programming and Markov Processes. Cambridge, MA, USA: MIT Press, 1960.
34. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, May 2005.
35. D. Liu, D. Wang, F.-Y. Wang, H. Li, and X. Yang, "Neural-networkbased online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2834-2847, Dec. 2014.
36. Z. Chen and S. Jagannathan, "Generalized Hamilton-Jacobi-Bellman formulation-based neural network control of affine nonlinear discretetime systems," IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 90-106, Jan. 2008.
37. D. Liu and Q. Wei, "Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 3, pp. 621-634, Mar. 2014.
38. J. N. Tsitsiklis, "On the convergence of optimistic policy iteration," J. Mach. Learn. Res., vol. 3, pp. 59-72, Jul. 2002.
39. D. P. Bertsekas, "Weighted sup-norm contractions in dynamic programming: A review and some new applications," Dept. Elect. Eng. Comput. Sci., Massachusetts Inst. Technol., Cambridge, MA, USA, Tech. Rep. LIDS-P-2884, May 2012.
40. D. P. Bertsekas, "Approximate policy iteration: A survey and some new methods," J. Control Theory Appl., vol. 9, no. 3, pp. 310-335, 2011.
41. L. Busoniu, D. Ernst, B. De Schutter, and R. Babuska, "Approximate reinforcement learning: An overview," in Proc. IEEE Symp. Adapt. Dyn. Program. Reinforcement Learn., Paris, France, Apr. 2011, pp. 1-8.
42. B. V. Roy, "Performance loss bounds for approximate value iteration with state aggregation," Math. Oper. Res., vol. 31 no. 2, pp. 234-244, 2006.
43. R. Munos, "Error bounds for approximate value iteration," in Proc. Nat. Conf. Artif. Intell., Pittsburgh, PA, USA, Jul. 2005, pp. 1006-1011.
44. R. Munos, "Performance bounds in L p-norm for approximate value iteration," SIAM J. Control Optim., vol. 46, no. 2, pp. 541-561, 2007.
45. R. Munos, "Error bounds for approximate policy iteration," in Proc. 20th Int. Conf. Mach. Learn., Washington, DC, USA, Aug. 2003, pp. 560-567.
46. T. J. Perkins and D. Precup, "A convergent form of approximate policy iteration," in Advances in Neural Information Processing Systems. Vancouver, BC, Canada: MIT Press, 2002.
47. B. Li and J. Si, "Approximate robust policy iteration using multilayer perceptron neural networks for discounted infinite-horizon Markov decision processes with uncertain correlated transition matrices," IEEE Trans. Neural Netw., vol. 21, no. 8, pp. 1270-1280, Aug. 2010.
48. C. Thiery and B. Scherrer, "Performance bound for approximate optimistic policy iteration," INRIA, Rocquencourt, France, Tech. Rep. INRIA-00480952, 2010.
49. B. Scherrer, V. Gabillon, M. Ghavamzadeh, and M. Geist, "Approximate modified policy iteration," in Proc. 29th Int. Conf. Mach. Learn., Edinburgh, Scotland, Jun./Jul. 2012, pp. 1207-1214.
50. R. J. Leake and R.-W. Liu, "Construction of suboptimal control sequences," SIAM J. Control, vol. 5, no. 1, pp. 54-63, 1967.
51. A. Rantzer, "Relaxed dynamic programming in switching systems," IEE Proc. Control Theory Appl., vol. 153, no. 5, pp. 567-574, Sep. 2006.
52. B. Lincoln and A. Rantzer, "Relaxing dynamic programming," IEEE Trans. Autom. Control, vol. 51, no. 8, pp. 1249-1260, Aug. 2006.
53. L. Grune and A. Rantzer, "On the infinite horizon performance of receding horizon controllers," IEEE Trans. Autom. Control, vol. 53, no. 9, pp. 2100-2111, Oct. 2008.
54. D. Liu and Q. Wei, "Finite-approximation-error based optimal control approach for discrete-time nonlinear systems," IEEE Trans. Cybern., vol. 43, no. 2, pp. 779-789, Apr. 2013.
55. Q. Wei, F.-Y. Wang, D. Liu, and X. Yang, "Finite-approximationerror- based discrete-time iterative adaptive dynamic programming," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2820-2833, Dec. 2014.
56. D. P. Bertsekas, Abstract Dynamic Programming. Belmont, MA, USA: Athena Scientific, 2013.
57. A. Almudevar and E. F. de Arruda, "Optimal approximation schedules for a class of iterative algorithms, with an application to multigrid value iteration," IEEE Trans. Autom. Control, vol. 57, no. 12, pp. 3132-3146, Dec. 2012.
58. X. Xu, D. Hu, and X. Lu, "Kernel-based least squares policy iteration for reinforcement learning," IEEE Trans. Neural Netw., vol. 18, no. 4, pp. 973-992, Jul. 2007.
59. S. Mahadevan and M. Maggioni, "Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes," J. Mach. Learn. Res., vol. 8, no. 10, pp. 2169-2231, 2007.
60. X. Xu, Z. Huang, D. Graves, and W. Pedrycz, "A clustering-based graph Laplacian framework for value function approximation in reinforcement learning," IEEE Trans. Cybern., vol. 44, no. 12, pp. 2613-2625, Dec. 2014.
61. Z. Huang, X. Xu, and L. Zuo, "Reinforcement learning with automatic basis construction based on isometric feature mapping," Inf. Sci., vol. 286, pp. 209-227, Dec. 2014.
62. H. Li and D. Liu, "Optimal control for discrete-time affine non-linear systems using general value iteration," IET Control Theory Appl., vol. 6, no. 18, pp. 2725-2736, Dec. 2012.