Study of reinforcement learning in the continuous case by the means of viscosity solutions

Machine Learning

Volumn 40, Issue 3, 2000, Pages 265-299

  Munos, Rémi ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; COMPUTER SIMULATION; CONVERGENCE OF NUMERICAL METHODS; DECISION THEORY; DIFFERENTIAL EQUATIONS; DYNAMIC PROGRAMMING; FINITE DIFFERENCE METHOD; FINITE ELEMENT METHOD; FUNCTION EVALUATION; MARKOV PROCESSES; PROBLEM SOLVING; THEOREM PROVING;

HAMILTON-JACOBI-BELLMAN EQUATION; MARKOV DECISION PROCESS; REINFORCEMENT LEARNING; STOCHASTIC STATE DYNAMICS; VISCOSITY SOLUTIONS;

LEARNING ALGORITHMS;

EID: 0034274415     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1007686309208     Document Type: Article

References (50)

1. Akian, M. (1990). Méthodes multigrilles en contrôle stochastique. Ph.D. Thesis, University Paris IX Dauphine.
2. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In Machine Learning: Proceedings of the Twelfth International Conference.
3. Barles, G. (1994). Solutions de viscosité des équations de Hamilton-Jacobi, Springer-Verlag. Mathématiques et Applications, Vol. 17.
4. Barles, G. & Perthame, B. (1988). Exit time problems in optimal control and vanishing viscosity solutions of hamilton-jacobi equations. SIAM Control Optimization, 26, 1133-1148.
5. Barles, G. & Perthame, B. (1990). Comparison principle for dirichlet-type hamilton-jacobi equations and singular perturbations of degenerated elliptic equations. Applied Mathematics and Optimization, 21, 21-44.
6. Barles, G. & Souganidis, P. (1991). Convergence of approximation schemes for fully nonlinear second order equations. Asymptotic Analysis, 4, 271-283.
7. Barto, A. G. (1990). Connectionist learning for control: An overview. In W. T. Miller, R. S. Sutton, & P. J. Werbos (Eds.), Neural Networks for Control (pp. 5-58). Cambridge, Massachussetts: MIT Press.
8. Barto, A. G., Bradtke, S. J., & Singh, S. P. (1991). Real-time learning and control using asynchronous dynamic programming. Tech. Rep. 91-57, Computer Science Department, University of Massachusetts.
9. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man and Sybernetics, 13, 835-846.
10. Bellman, R. (1957). Dynamic Programming. Princeton Univ. Press.
11. Bersini, H. & Gorrini, V. (1997). A simplification of the back-propagation-through-time algorithm for optimal neurocontrol. IEEE Transaction on Neural Networks, 8, 437-441.
12. Bertsekas, D. P. (1987). Dynamic Programming: Deterministic and Stochastic Models. Prentice Hall.
13. Bertsekas, D. P. & Tsitsiklis, J. (1996). Neuro-Dynamic Programming. Athena Scientific.
14. Boyan, J. & Moore, A. (1995). Generalization in reinforcement learning: Safely approximating the value function. Advances in Neural Information Processing Systems, 7, 369-376.
15. Crandall, M., Ishii, H., & Lions, P. (1992). User's guide to viscosity solutions of second order partial differential equations. Bulletin of the American Mathematical Society, 27(1), 1-67.
16. Crandall, M. & Lions, P. (1983). Viscosity solutions of hamilton-jacobi equations. Trans. of the American Mathematical Society, 277, 1-42.
17. Doya, K. (1996). Temporal difference learning in continuous time and space. Advances in Neural Information Processing Systems, 8, 1073-1079.
18. Dupuis, P. & James, M. R. (1998). Rates of convergence for approximation schemes in optimal control. SIAM Journal Control and Optimization, 360(2).
19. Fleming, W. H. & Soner, H. M. (1993). Controlled Markov Processes and Viscosity Solutions. Springer-Verlag. Applications of Mathematics.
20. Glorennec, P. & Jouffe, L. (1997). Fuzzy q-learning. In Sixth International Conference on Fuzzy Systems.
21. Gordon, G. (1995). Stable function approximation in dynamic programming. In International Conference on Machine Learning.
22. Griebel, M. (1998). Adaptive sparse grid multilevel methods for elliptic pdes based on finite differences. In Proceedings Large Scale Scientific Computations. Notes on Numerical Fluid Mechanics: Computing, submitted.
23. Gullapalli, V. (1992). Reinforcement Learning and its application to control. Ph.D. Thesis, University of Massachusetts, Amherst.
24. Harmon, M. E., Baird, L. C., & Klopf, A. H. (1996). Reinforcement learning applied to a differential game. Adaptive Behavior, 4, 3-28.
25. Kaelbling, L. P., Littman, M., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of AI Research, 4, 237-285.
26. Kushner, H. J. (1990). Numerical methods for stochastic control problems in continuous time. SIAM J. Control and Optimization, 28, 999-1048.
27. Kushner, H. J. & Dupuis, P. (1992). Numerical Methods for Stochastic Control Problems in Continuous Time. Springer-Verlag. Applications of Mathematics.
28. Lin, L.-J. (1993). Reinforcement learning for robots using neural networks. Ph.D. Thesis, Carnegie Mellon University, Pittsburg, Pennsylvania.
29. Mahadevan, S. & Connell, J. (1992). Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence, 5, 311-365.
30. Meuleau, N. (1996). Le dilemme Exploration/Exploitation dans les systèmes d'apprentissage par renforcement. Ph.D. Thesis, Université de Caen.
31. Moore, A. W. (1991). Variable resolution dynamic programming: Efficiently learning action maps in multivariate real-valued state-spaces. In Machine Learning: Poceedings of the Eight International Workshop (pp. 333-337).
32. Moore, A. W. & Atkeson, C. (1995). The parti-game algorithm for variable resolution reinforcement learning in multidimensional state space. Machine Learning Journal, 21.
33. Munos, R. (1996). A convergent reinforcement learning algorithm in the continuous case: The finite-element reinforcement learning. In International Conference on Machine Learning.
34. Munos, R. (1997a). Apprentissage par Renforcement, étude du cas continu. Ph.D. Thesis, Ecole des Hautes Etudes en Sciences Sociales.
35. Munos, R. (1997b). A convergent reinforcement learning algorithm in the continuous case based on a finite difference method. In International Joint Conference on Artificial Intelligence.
36. Munos, R. (1997c). Finite-element methods with local triangulation refinement for continuous reinforcement learning problems. In European Conference on Machine Learning.
37. Munos, R. (1998). A general convergence theorem for reinforcement learning in the continuous case. In European Conference on Machine Learning.
38. Munos, R., Baird, L., & Moore, A. (1999). Gradient descent approaches to neural-net-based solutions of the hamilton-jacobi-bellman equation. In International Joint Conference on Neural Networks.
39. Munos, R. & Bourgine, P. (1997). Reinforcement learning for continuous stochastic control problems. Advances in Neural Information Processing Systems, 10.
40. Munos, R. & Moore, A. (1998). Barycentric interpolators for continuous space and time reinforcement learning. Advances in Neural Information Processing Systems, 11, 1024-1030.
41. Munos, R. & Moore, A. (1999). Variable resolution discretization for high-accuracy solutions of optimal control problems. In International Joint Conference on Artificial Intelligence, 1348-1355.
42. Nowé, A. (1995). Fuzzy reinforcement learning an overview. Advances in Fuzzy Theory and Technology.
43. Pareigis, S. (1996). Multi-grid methods for reinforcement learning in controlled diffusion processes. Advances in Neural Information Processing Systems, 9.
44. Pareigis, S. (1997). Adaptive choice of grid and time in reinforcement learning. Advances in Neural Information Processing Systems, 10.
45. Pontryagin, L., Boltyanskii, V., Gamkriledze, R., & Mischenko, E. (1962). The Mathematical Theory of Optimal Processes. New York: Interscience.
46. Puterman, M. L. (1994). Markov Decision Processes, Discrete Stochastic Dynamic Programming. A Wiley-Interscience Publication.
47. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Reinforcement learning with soft state aggregation. Advances in Neural Information Processing Systems, 6, 359-368.
48. Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in Neural Information Processing Systems, 8, 1038-1044.
49. Sutton, R. & Whitehead, S. (1993). Online learning with random representations. In International Conference on Machine Learning.
50. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229-256.