An equivalence between adaptive dynamic programming with a critic and backpropagation through time

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 12, 2013, Pages 2088-2100

  Fairbank, Michael ^a   Alonso, Eduardo ^a   Prokhorov, Danil ^b  

^a CITY UNIVERSITY   (United Kingdom)

^b Toyota Motor North America Inc   (United States)

Author keywords

Adaptive dynamic programming (ADP); backpropagation through time; dual heuristic programming (DHP); neural networks; value gradient learning

Indexed keywords

ADAPTIVE DYNAMIC PROGRAMMING; BACK-PROPAGATION THROUGH TIME; CONTINUOUS STATE SPACE; DUAL HEURISTIC PROGRAMMING; GUARANTEED CONVERGENCE; NONLINEAR FUNCTIONS; SMOOTHNESS CONDITIONS; VALUE-GRADIENT LEARNING;

DYNAMIC PROGRAMMING; HEURISTIC PROGRAMMING; NEURAL NETWORKS;

ALGORITHMS;

EID: 84887996993     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2271778     Document Type: Article

References (29)

1. F.-Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
2. R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton Univ. Press, 1957.
3. P. J. Werbos, "Approximate dynamic programming for real-time control and neural modeling," in Handbook of Intelligent Control, D. White and D. Sofge, Eds. New York, NY, USA: Van Nostrand Reinhold, 1992, ch. 13, pp. 493-525.
4. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997. (Pubitemid 127763331)
5. S. Ferrari and R. F. Stengel, "Model-based adaptive critic designs," in Handbook of Learning and Approximate Dynamic Programming, J. Si, A. Barto, W. Powell, and D. Wunsch, Eds. New York, NY, USA: Wiley, 2004, pp. 65-96.
6. M. Fairbank, E. Alonso, and D. Prokhorov, "Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 10, pp. 1671-1678, Oct. 2012.
7. M. Fairbank, "Reinforcement learning by value gradients," (2008) [Online]. Available: http://arxiv.org/abs/0803.3539
8. M. Fairbank and E. Alonso, "Value-gradient learning," in Proc. IEEE IJCNN, Jun. 2012, pp. 3062-3069.
9. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, no. 1, pp. 9-44, 1988.
10. G. K. Venayagamoorthy and D. C. Wunsch, "Dual heuristic programming excitation neurocontrol for generators in a multimachine power system," IEEE Trans. Ind. Appl., vol. 39, no. 2, pp. 382-394, Mar./Apr. 2003.
11. G. G. Lendaris and C. Paintz, "Training strategies for critic and action neural networks in dual heuristic programming method," in Proc. Int. Conf. Neural Netw., Jun. 1997, pp. 712-717.
12. L. S. Pontryagin, V. G. Boltayanskii, R. V. Gamkrelidze, and E. F. Mishchenko, The Mathematical Theory of Optimal Processes, vol. 4. New York, NY, USA: Wiley, 1962.
13. M. Fairbank and E. Alonso, "The local optimality of reinforcement learning by value gradients, and its relationship to policy gradient learning," (2011) [Online]. Available: http://arxiv.org/abs/1101.0428
14. M. Fairbank and E. Alonso, "A comparison of learning speed and ability to cope without exploration between DHP and TD(0)," in Proc. IEEE IJCNN, Jun. 2012, pp. 1478-1485.
15. P. J. Werbos, "Neural networks, system identification, and control in the chemical process industries," in Handbook of Intelligent Control, D. White and D. Sofge, Eds. New York, NY, USA: Van Nostrand Reinhold, 1992, ch. 10, pp. 283-356.
16. R. A. Howard, Dynamic Programming and Markov Processes. Cambridge, MA, USA: MIT Press, 1960, ch. 4, pp. 42-43.
17. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
18. A. Heydari and S. N. Balakrishnan, "Finite-horizon input-constrained nonlinear optimal control using single network adaptive critics," in Proc. ACC, Jun./Jul. 2011, pp. 3047-3052.
19. D. V. Prokhorov and D. C. Wunsch, "Convergence of critic-based training," in Proc. IEEE Int. Conf. Syst. Man, Cybern. Comput. Cybern. Simul., Oct. 1997, pp. 3057-3060.
20. P. J. Werbos, "Stable adaptive control using new critic designs," Proc. SPIE, vol. 3728, p. 510, Mar. 1999, ArXiv:adap-org/9810001
21. L. C. Baird, "Residual algorithms: Reinforcement learning with function approximation," in Proc. Int. Conf. Mach. Learn., 1995, pp. 30-37.
22. P. J. Werbos, "Consistency of HDP applied to a simple reinforcement learning problem," Neural Netw., vol. 3, pp. 179-189, Jan. 1990.
23. P. J. Werbos, "Backpropagation through time: What it does and how to do it," Proc. IEEE, vol. 78, no. 10, pp. 1550-1560, Oct. 1990.
24. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
25. E. Barnard, "Temporal-difference methods and Markov models," IEEE Trans. Syst., Man, Cybern., vol. 23, no. 2, pp. 357-365, Mar./Apr. 1993.
26. M. Fairbank, D. Prokhorov, and E. Alonso, "Approximating optimal control with value gradient learning," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, F. Lewis and D. Liu, Eds. New York, NY, USA: Wiley, 2012, Sections 7.3.4 and 7.4.3.
27. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Netw., San Francisco, CA, USA, Apr. 1993, pp. 586-591.
28. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. 13, no. 5, pp. 834-846, Sep./Oct. 1983. (Pubitemid 14138646)
29. R. V. Florian, "Correct equations for the dynamics of the cart-pole system," Center for Cognit., Neural Studies (Coneural), Cluj-Napoca, Romania, Tech. Rep., 2007.