Clipping in neurocontrol by adaptive dynamic programming

IEEE Transactions on Neural Networks and Learning Systems

Volumn 25, Issue 10, 2014, Pages 1909-1920

  Fairbank, Michael ^a   Prokhorov, Danil ^b   Alonso, Eduardo ^a  

^a CITY UNIVERSITY   (United Kingdom)

^b Toyota Motor North America Inc   (United States)

Author keywords

Backpropagation through time (BPTT); clipping; dual heuristic programming (DHP); neurocontrol; value gradient learning

Indexed keywords

BACKPROPAGATION THROUGH TIME; CLIPPING; DUAL HEURISTIC PROGRAMMING; NEURO CONTROL; VALUE-GRADIENT LEARNING;

ALGORITHM; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; HUMAN; NONLINEAR SYSTEM; REINFORCEMENT;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMANS; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; REINFORCEMENT (PSYCHOLOGY);

EID: 84907821545     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2014.2297991     Document Type: Article

References (35)

1. F.-Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39-47, May 2009.
2. P. J. Werbos, "Neurocontrol: Where it is going and why it is crucial," in Artifical Neural Networks II (ICANN-2), North Holland, Amsterdam, I. Aleksander, Ed. New York, NY, USA: Taylor, 1992, pp. 61-68.
3. M. Hagan and H. Demuth, "Neural networks for control," in Proc. Amer. Control Conf., vol. 3. 1999, pp. 1642-1656.
4. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
5. F. L. Lewis, D. Vrabie, and K. G. Vamvoudakis, "Reinforcement learning and feedback control: Using natural decision methods to design optimal adaptive controllers," IEEE Control Syst., vol. 32, no. 6, pp. 76-105, Dec. 2012.
6. P. J. Werbos, "Approximating dynamic programming for real-time control and neural modeling." in Handbook of Intelligent Control, D. A. White and D. A. Sofge, Eds. New York, NY, USA: Van Nostrand, 1992, pp. 493-525, ch. 13.
7. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997-1007, Sep. 1997.
8. P. J. Werbos, "Backpropagation through time: What it does and how to do it," Proc. IEEE, vol. 78, no. 10, pp. 1550-1560, Oct. 1990.
9. M. Fairbank and E. Alonso, "Value-gradient learning," in Proc. IEEE IJCNN, Jun. 2012, pp. 3062-3069.
10. M. Fairbank, D. Prokhorov, and E. Alonso, "Approximating optimal control with value gradient learning," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, F. Lewis and D. Liu, Eds. New York, NY, USA: Wiley, 2012, pp. 142-161.
11. M. Fairbank. (2008). Reinforcement Learning by Value Gradients [Online]. Available: http://arxiv.org/abs/0803.3539
12. M. Fairbank, E. Alonso, and D. Prokhorov, "An equivalence between adaptive dynamic programming with a critic and backpropagation through time," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 12, pp. 2088-2100, Dec. 2013.
13. D. Prokhorov, "Backpropagation through time and derivative adaptive critics: A common framework for comparison," in Handbook of Learning and Approximate Dynamic Programming, J. Si, A. Barto, W. Powell, and D. Wunsch, Eds. New York, NY, USA: Wiley, 2004.
14. M. Fairbank and E. Alonso. (2011). The Local Optimality of Reinforcement Learning by Value Gradients, and its Relationship to Policy Gradient Learning [Online]. Available: http://arxiv.org/abs/1101.0428
15. R. Munos, "Policy gradient in continuous time," J. Mach. Learn. Res., vol. 7, pp. 413-427, Jan. 2006.
16. M. Fairbank, E. Alonso, and D. Prokhorov, "Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 10, pp. 1671-1678, Oct. 2012.
17. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
18. A. Al-Tamimi, M. Abu-Khalaf, and F. L. Lewis, "Adaptive critic designs for discrete-time zero-sum games with application to h∞ control," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 37, no. 1, pp. 240-247, Feb. 2007.
19. A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, "Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 943-949, Aug. 2008.
20. H. Zhang, Y. Luo, and D. Liu, "Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints," IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490-1503, Sep. 2009.
21. H. Zhang, Q. Wei, and Y. Luo, "A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 38, no. 4, pp. 937-942, Aug. 2008.
22. R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Mach. Learn., vol. 8, nos. 3-4, pp. 229-356, 1992.
23. J. Peters and S. Schaal, "Policy gradient methods for robotics," in Proc. IEEE/RSJ Int. Conf. IROS, Oct. 2006, pp. 2219-2225.
24. C. J. C. H. Watkins, "Learning from delayed rewards," Ph.D. dissertation, Dept. Comput. Sci., Cambridge Univ., Cambridge, U.K., 1989.
25. S. Ferrari and R. F. Stengel, "Model-based adaptive critic designs," in Handbook of Learning and Approximate Dynamic Programming, J. Si, A. Barto, W. Powell, and D. Wunsch, Eds. New York, NY, USA: Wiley, 2004, pp. 65-96.
26. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, no. 1, pp. 9-44, 1988.
27. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
28. P. J. Werbos, "Beyond regression: New tools for prediction and analysis in the behavioral sciences," Ph.D. dissertation, Dept. Comput. Sci., Harvard Univ., Cambridge, MA, USA, 1974.
29. D. Rumelhart, G. Hinton, and R. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, no. 6088, pp. 533-536, Oct. 1986.
30. E. Barnard, "Temporal-difference methods and Markov models," IEEE Trans. Syst., Man, Cybern., vol. 23, no. 2, pp. 357-365, Mar./Apr. 1993.
31. C. M. Bishop, Neural Networks for Pattern Recognition. Oxford, U.K.: Oxford Univ. Press, 1995.
32. M. Fairbank and E. Alonso, "A comparison of learning speed and ability to cope without exploration between DHP and TD(0)," in Proc. IEEE Int. Joint Conf. Neural Netw., Jun. 2012, pp. 1478-1485.
33. A. G. Barto, R. S. Sutton, and C. W. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. 13, no. 5, pp. 834-846, Sep./Oct. 1983.
34. G. G. Lendaris and C. Paintz, "Training strategies for critic and action neural networks in dual heuristic programming method," in Proc. Int. Conf. Neural Netw., Jun. 1997, pp. 712-717.
35. R. V. Florian, "Correct equations for the dynamics of the cart-pole system," Center for Cognitive and Neural Studies (Coneural), Univ. Bucharest, Cluj-Napoca, Romania, Tech. Rep., 2007.