Least squares solutions of the HJB equation with neural network value-function approximators

IEEE Transactions on Neural Networks

Volumn 18, Issue 4, 2007, Pages 1031-1041

  Tassa, Yuval ^a   Erez, Tom ^b  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^b WASHINGTON UNIVERSITY IN ST LOUIS   (United States)

Author keywords

Differential neural networks (NNs); Dynamic programming; Feedforward neural networks; Hamilton Jacoby Bellman (HJB) equation; Optimal control; Viscosity solution

Indexed keywords

FIRST- AND SECOND-ORDER DIFFERENTIAL BACKPROPAGATION; HAMILTON-JACOBI-BELLMAN (HLB) RESIDUAL; INVERTED-PENDULUM SYSTEM; VALUE FUNCTION COMPLEXITY;

BACKPROPAGATION; CONTROL NONLINEARITIES; CONVERGENCE OF NUMERICAL METHODS; DYNAMIC PROGRAMMING; LEAST SQUARES APPROXIMATIONS; OPTIMAL CONTROL SYSTEMS; RANDOM PROCESSES;

FEEDFORWARD NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; COMPUTER SYSTEM; DECISION SUPPORT SYSTEM; FEEDBACK SYSTEM; REGRESSION ANALYSIS; THEORETICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; COMPUTER SYSTEMS; DECISION SUPPORT TECHNIQUES; FEEDBACK; LEAST-SQUARES ANALYSIS; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER);

EID: 34547095501     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2007.899249     Document Type: Article

References (35)

1. M. Crandall and P. Lions, "Viscosity solutions of Hamilton-Jacobi equations," Trans. Amer. Math. Soc., vol. 277, 1983.
2. J. A. Boyan and A. W. Moore, "Generalization in reinforcement learning: Safely approximating the value function," in Advances in Neural Information Processing Systems 7, G. Tesauro, D. S. Touretzky, and T. K. Leen, Eds. Cambridge, MA: MIT Press, 1995, pp. 369-376.
3. K.-I. Funahashi, "On the approximate realization of continuous mappings by neural networks," Neural Netw., vol. 2, pp. 183-192, 1989.
4. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
5. R. Coulom, "Reinforcement learning using neural networks, with applications to motor control," Ph.D. dissertation, Institut National Polytechnique de Grenoble, Grenoble, France, 2002.
6. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995.
7. D. H. Jacobson and D. Q. Mayne, Differential Dynamic Programming. New York: Elsevier, 1970.
8. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, pp. 9-44, 1988.
9. W. Schultz, P. Dayan, and P. R. Montague, "A neural substrate of prediction and reward," Science, vol. 275, pp. 1593-1599, 1997.
10. M. G. Lagoudakis, R. E. Parr, and M. L. Littman, "Least-squares methods in reinforcement learning for control," in Proc. 2nd Hellenic Conf. Artif. Intell., 2002, vol. 2308, pp. 249-260.
11. R. Munos and A. W. Moore, "Variable resolution discretization for high-accuracy solutions of optimal control problems," in Proc. Int. Joint Conf. Artif. Intell., 1999, pp. 1348-1355.
12. K. Doya, "Temporal difference learning in continuous time and space," in Advances in Neural Information Processing Systems, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996, vol. 8.
13. G. Tesauro, "Practical issues in temporal difference learning," in Advances in Neural Information Processing Systems, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, vol. 4, pp. 259-266.
14. R. Munos, L. Baird, and A. Moore, "Gradient descent approaches to neural net-based solutions of the Hamilton-Jacobi-Bellman equation," in Proc. Int. Joint Conf. Neural Netw., 1999, pp. 1316-1323.
15. S. Thrun and A. Schwartz, "Issues in using function approximation for reinforcement learning," in Proc. 1993 Connectionist Models Summer School, M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and A. Weigend, Eds., 1993, pp. 255-263.
16. R. Beard and T. McLain, "Successive galerkin approximation algorithms for nonlinear optimal and robust control, Proc. Int. J. Control: Special Issue Breakthroughs Control Nonlinear Syst., vol. 71, no. 5, pp. 717-743, 1998.
17. J. A. Sethian, Level Set Methods and Fast Marching Methods. Cambridge, U.K.: Cambridge Univ. Press, 1999.
18. K. S. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks," IEEE Trans. Neural Netw. vol. 1, no. 1, pp. 4-27, Mar. 1990.
19. C. J. Goh, "On the nonlinear optimal regulator problem," Automatica vol. 29, no. 3, pp. 751-756, 1993.
20. P. Simard, B. Victorri, Y. LeCun, and J. Denker, "Tangent prop - A formalism for specifying selected invariances in an adaptive network," in Neural Information Processing Systems, J. M. R. Lippman and S. J. Hanson, Eds. San Mateo, CA: Morgan Kaufmann, 1992, vol. 4.
21. J. W. Lee and J. H. Oh, "Hybrid learning of mapping and its Jacobian in multilayer neural networks," Neural Comput., vol. 9, pp. 937-958, 1997.
22. R. Masuoka, "Neural networks learning differential data," IEICE Trans. Inf. Syst., vol. E83-D, no. 6, pp. 1291-1300, 2000.
23. W. Fleming and H. Soner, Controlled Markov Processes and Viscosity Solutions. New York: Springer-Verlag, 1993.
24. G. Saridis and C. S. Lee, "An approximation theory of optimal control for trainable manipulators," IEEE Trans. Syst., Man, Cybern., vol. SMC-9, no. 3, pp. 152-159, Mar. 1979.
25. F. Pineda, "Generalization of back-propagation to recurrent neural networks," Phys. Rev. Lett., vol. 19, no. 59, pp. 2229-2232, 1987.
26. M. S. Bazarraa, H. D. Sherali, and C. M. Shetty, Nonlinear Programming: Theory and Algorithms. New York: Wiley, 1993.
27. D. H. Nguyen and B. Widrow, "Improving the learning speed of 2-layer neural network by choosing initial values of the adaptive weights," in Proc. 1st IEEE Int. Joint Conf. Neural Netw., 1990, vol. 3, pp. 21-26.
28. J. H. Halton, "On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals," Numerische Mathematik, vol. 2, pp. 84-90, 1960.
29. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, 2000.
30. S. E. Lyshevski and A. U. Meyer, "Control system analysis and design upon the Lyapunov method," in Proc. Amer. Control Conf., Jun. 1995, pp. 3219-3223.
31. D. Kleinman, "On an iterative technique for Riccati equation computations," IEEE Trans. Autom. Control, vol. 13, no. 1, pp. 114-115, Feb. 1968.
32. A. Moore and C. Atkeson, "The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces," Mach. Learn., vol. 21, pp. 1-36, 1995.
33. N. N. Schraudolph, "Local gain adaptation in stochastic gradient descent ISDIA, Lugano, Switzerland, Tech. Rep. IDSIA-09-99, 1999, p. 8.
34. M. Lazaro, I. Santamaria, F. Perez-Cruz, and A. Artes-Rodriguez, "Support vector regression for the simultaneous learning of a multivariate function and its derivatives," Neurocomput., vol. 69, pp. 42-61, 2005.
35. B. A. Pearlmutter, "Fast exact multiplication by the Hessian," Neural Comput., vol. 6, no. 1, pp. 147-160, 1994.