Performance bounds in Lp-norm for approximate value iteration

SIAM Journal on Control and Optimization

Volumn 46, Issue 2, 2007, Pages 541-561

  Munos, Rémi ^a  

^a UNIV LILLE   (France)

Author keywords

Dynamic programming; Error analysis; Function approximation; Markov decision processes; Optimal control; Reinforcement learning; Statistical learning

Indexed keywords

DYNAMIC PROGRAMMING; ERROR ANALYSIS; ITERATIVE METHODS; OPTIMAL CONTROL SYSTEMS; REINFORCEMENT LEARNING;

FUNCTION APPROXIMATION; STATISTICAL LEARNING;

MARKOV PROCESSES;

EID: 40949107944     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/040614384     Document Type: Article

References (38)

1. A. ANTOS, CS. SZEPESVARI, AND R. MUNOS, Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path, in Proceedings of the Conference on Learning Theory, Springer-Verlag, New York, 2006, pp. 574-588.
2. C. G. ATKESON, A. W. MOORE, AND S. A. SCHAAL, Locally weighted learning, AI Rev., 11 (1997), pp. 11-73.
3. C. G. ATKESON. A. W. MOORE, AND S. A. SCHAAL, Locally weighted learning for control, AI Rev., 11 (1997), pp. 75-113.
4. R. BELLMAN, Dynamic Programming, Princeton University Press, Princeton, NJ, 1957.
5. R. E. BELLMAN AND S. E. DREYFUS, Functional approximation and dynamic programming, Math. Tables Aids Comput., 13 (1959), pp. 247-251.
6. D. P. BERTSEKAS, Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, NJ, 1987.
7. D. P. BERTSEKAS AND J. TSITSIKLIS, Neuro-Dynamic Programming, Athena Scientific, Nashua, NH, 1996.
8. P. BOUGEROL AND N. PICARD, Strict stationarity of generalized autoregressive processes, Ann. Probab., 20 (1992), pp. 1714-1730.
9. G. M. DAVIES, S. MALLAT, AND M. AVELLANEDA, Adaptive greedy approximations, J. Constr. Approx., 13 (1997), pp. 57-98.
10. D. P. DE FARIAS AND B. VAN ROY, The linear programming approach to approximate dynamic programming, Oper. Res., 51 (2003), pp. 850-865.
11. R. DEVORE, Nonlinear approximation, Acta Numer., 7 (1998), pp. 51-150.
12. G. GORDON, Stable function approximation in dynamic programming, in Proceedings of the International Conference on Machine Learning, Morgan Kaufmann, 1995, pp. 261-268.
13. G. J. GORDON, Approximate Solutions to Markov Decision Processes, Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA, 1999.
14. C. GUESTRIN, D. KOLLER, AND R. PARR, Max-norm projections for factored MDPs, in Proceedings of the International Joint Conference on Artificial Intelligence, Lawrence Erlbaum, 2001, pp. 673-682.
15. T. HASTIE, R. TIBSHIRANI, AND J. FRIEDMAN, The Elements of Statistical Learning, Springer Ser. Statist., Springer-Verlag, New York, 2001.
16. O. HERNÁNDEZ-LERMA AND J. B. LASSERRE, Discrete-Time Markov Control Processes, Basic Optimality Criteria, Springer-Verlag, New York, 1996.
17. O. HERNÁNDEZ-LERMA AND J. B. LASSERRE, Further Topics on Discrete-Time Markov Control Processes, Springer-Verlag, New York, 1999.
18. O. HERNÁNDEZ-LERMA, R. MONTES- DE-OCA, AND R. CAVAZOS-CANEDA, Recurrence conditions for Markov decision processes with Borel state space: A survey, Ann. Oper. Res., 28 (1991), pp. 29-46.
19. A. HORDIJK AND F. SPIEKSMA, On ergodicity and recurrence properties of a Markov chain with an application to an open Jackson network, Adv. Appl. Probab., 24 (1992), pp. 343-376.
20. S. M. KAKADE, On the Sample Complexity of Reinforcement Learning, Ph.D. thesis, University College London, 2003.
21. S. KAKADE AND J. LANGFORD, Approximately optimal approximate reinforcement learning, in Proceedings of the 19th International Conference on Machine Learning, Morgan Kaufmann, 2002, pp. 267-274.
22. D. KOLLER AND R. PARR, Policy iteration for factored MDPs, in Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann, 2000, pp. 326-334.
23. M. LAGOUDAKIS AND R. PARR, Least-squares policy iteration, J. Mach. Learn. Res., 4 (2003), pp. 1107-1149.
24. S. MALLAT, A Wavelet Tour of Signal Processing, Academic Press, San Diego, CA, 1997.
25. S. P. MEYN, Stability, performance evaluation, and optimization, in Handbook of Markov Decision Processes: Methods and Applications, Kluwer Academic, Boston, MA, 2002, pp. 305-346.
26. R. MUNOS, Error bounds for approximate policy iteration, in Proceedings of the 19th International Conference on Machine Learning, AAAI Press, 2003, pp. 560-567.
27. R. MUNOS AND CS. SZEPESVÁRI, Finite-Time Bounds for Sampling-Based Fitted Value Iteration, Technical report, INRIA, 2006; available online from http://hal.inria.fr/inria-00120882.
28. D. POLLARD, Convergence of Stochastic Processes, Springer-Verlag, New York, 1984.
29. M. L. PUTERMAN, Markov Decision Processes, Discrete Stochastic Dynamic Programming, Wiley-Interscience, New York, 1994.
30. E. RASMUSSEN AND C. K. I. WILLIAMS, Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, 2005.
31. J. RUST, Numerical dynamic programming in economics, in Handbook of Computational Economics, Elsevier/North-Holland, Amsterdam, 1996, pp. 619-729.
32. A. L. SAMUEL, Some studies in machine learning using the game of checkers, IBM J. Res. Develop., 3 (1959), pp. 210-229; reprinted in Computers and Thought, E. A. Feigenbaum and J. Feldman, eds., McGraw-Hill, New York, 1963.
33. R. S. SUTTON AND A. G. BARTO, Reinforcement Learning: An Introduction, Bradford Book, MIT Press, Cambridge, MA, 1998.
34. CS. SZEPESVARI AND R. MUNOS, Finite time bounds for sampling based fitted value iteration,in Proceedings of the International Conference on Machine Learning, ACM, New York, 2005, pp. 881-886.
35. J. N. TSITSIKLIS AND B. VAN ROY, An analysis of temporal difference learning with function approximation, IEEE Trans. Automat. Control, 42 (1997), pp. 674-690.
36. V. VAPNIK, Statistical Learning Theory, John Wiley & Sons, New York, 1998.
37. V. VAPNIK, S. E. GOLOWICH, AND A. SMOLA, Support vector method for function approximation, regression estimation and signal processing, in Advances in Neural Information Processing Systems, 1997, pp. 281-287.
38. R. J. WILLIAMS AND L. C. BAIRD, Tight Performance Bounds on Greedy Policies Based on Imperfect Value Functions, Technical report NU-CCS-93-14, Northeastern University, Boston, MA, 1993.