Finite-time bounds for fitted value iteration

Journal of Machine Learning Research

Volumn 9, Issue , 2008, Pages 815-857

  Munos, Rémi ^a   Szepesvári, Csaba ^b  

^a INRIA   (France)

^b UNIVERSITY OF ALBERTA   (Canada)

Author keywords

Discounted markovian decision processes; Fitted value iteration; Generative model; Optimal control; Pollard's inequality; Regression; Reinforcement learning; Statistical learning theory; Supervised learning

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; ELECTRIC NETWORK ANALYSIS; MODAL ANALYSIS; RISK ASSESSMENT; ROBOT LEARNING; STATE SPACE METHODS;

CONVERGENCE RATES; FITTED VALUE ITERATION; GENERATIVE MODELING; HIGH PROBABILITY; INFINITE STATES; MARKOVIAN DECISION PROCESSES;

MULTITASKING;

EID: 44649189852     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (66)

1. M. Anthony and P.L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, Cambridge, UK, 1999.
2. A. Antos, Cs. Szepesvári, and R. Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. In G. Lugosi and H.U. Simon, editors, The Nineteenth Annual Conference on Learning Theory, COLT 2006, Proceedings, volume 4005 of LNCS/LNAI, pages 574-588, Berlin, Heidelberg, June 2006. Springer-Verlag. (Pittsburgh, PA, USA, June 22-25, 2006.).
3. A. Antos, Cs. Szepesvári, and R. Munos. Value-iteration based fitted policy iteration: learning with a single trajectory. In 2007 IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2007), pages 330-337. IEEE, April 2007. (Honolulu, Hawaii, Apr 1-5, 2007.).
4. A. Antos, Cs. Szepesvári, and R. Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71:89-129, 2008.
5. Leemon C. Baird. Residual algorithms: Reinforcement learning with function approximation. In Armand Prieditis and Stuart Russell, editors, Proceedings of the Twelfth International Conference on Machine Learning, pages 30-37, San Francisco, CA, 1995. Morgan Kaufmann.
6. R.E. Bellman and S.E. Dreyfus. Functional approximation and dynamic programming. Math. Tables and other Aids Comp., 13:247-251, 1959.
7. D. P. Bertsekas and S.E. Shreve. Stochastic Optimal Control (The Discrete Time Case). Academic Press, New York, 1978.
8. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.
9. P. Bougerol and N. Picard. Strict stationarity of generalized autoregressive processes. Annals of Probability, 20:1714-1730, 1992.
10. E.W. Cheney. Introduction to Approximation Theory. McGraw-Hill, London, New York, 1966.
11. C.S. Chow and J.N. Tsitsiklis. The complexity of dynamic programming. Journal of Complexity, 5:466-488, 1989.
12. C.S. Chow and J.N. Tsitsiklis. An optimal multigrid algorithm for continuous state discrete time stochastic control. IEEE Transactions on Automatic Control, 36(8):898-914, 1991.
13. N. Cristianini and J. Shawe-Taylor. An introduction to support vector machines (and other kernel-based learning methods). Cambridge University Press, 2000.
14. R.H. Crites and A.G. Barto. Improving elevator performance using reinforcement learning. In Advances in Neural Information Processing Systems 9, 1997.
15. R. DeVore. Nonlinear Approximation. Acta Numerica, 1997.
16. T. G. Dietterich and X. Wang. Batch value function approximation via support vectors. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, Cambridge, MA, 2002. MIT Press.
17. D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
18. E. Even-Dar, S. Mannor, and Y. Mansour. PAC bounds for multi-armed bandit and Markov decision processes. In Fifteenth Annual Conference on Computational Learning Theory (COLT), pages 255-270, 2002.
19. G.J. Gordon. Stable function approximation in dynamic programming. In Armand Prieditis and Stuart Russell, editors, Proceedings of the Twelfth International Conference on Machine Learning, pages 261-268, San Francisco, CA, 1995. Morgan Kaufmann.
20. A. Gosavi. A reinforcement learning algorithm based on policy iteration for average reward: Empirical results with yield management and convergence analysis. Machine Learning, 55:5-29, 2004.
21. U. Grendander. Abstract Inference. Wiley, New York, 1981.
22. L. Györfi, M. Kohler, A. Krzyżak, and H. Walk. A distribution-free theory of nonparametric regression. Springer-Verlag, New York, 2002.
23. M. Haugh. Duality theory and simulation in financial engineering. In Proceedings of the Winter Simulation Conference, pages 327-334, 2003.
24. D. Haussler. Sphere packing numbers for subsets of the boolean n-cube with bounded Vapnik-Chervonenkis dimension. Journal of Combinatorial Theory, Series A, 69(2):217-232, 1995.
25. W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.
26. T. Jung and T. Uthmann. Experiments in value function approximation with sparse support vector regression. In ECML, pages 180-191, 2004.
27. S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
28. S.M. Kakade. On the sample complexity of reinforcement learning. PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.
29. M. Kearns, Y. Mansour, and A.Y. Ng. A sparse sampling algorithm for near-optimal planning in large Markovian decision processes. In Proceedings of IJCAI'99, pages 1324-1331, 1999.
30. G. Kimeldorf and G. Wahba. Some results on Tchebycheffian spline functions. J. Math. Anal. Applic., 33:82-95, 1971.
31. M. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of Machine Learning Research, 4:1107-1149, 2003.
32. W.S. Lee, P.L. Bartlett, and R.C. Williamson. Efficient agnostic learning of neural networks with bounded fan-in. IEEE Transactions on Information Theory, 42(6):2118-2132, 1996.
33. F. A. Longstaff and E. S. Shwartz. Valuing american options by simulation: A simple least-squares approach. Rev. Financial Studies, 14(1): 113-147, 2001.
34. S. Mahadevan, N. Marchalleck, T. Das, and A. Gosavi. Self-improving factory simulation using continuous-time average-reward reinforcement learning. In Proceedings of the 14th International Conference on Machine Learning (IMLC '97), 1997.
35. T.L. Morin. Computational advances in dynamic programming. In Dynamic Programming and its Applications, pages 53-90. Academic Press, 1978.
36. R. Munos. Error bounds for approximate policy iteration. In 19th International Conference on Machine Learning, pages 560-567, 2003.
37. R. Munos. Error bounds for approximate value iteration. American Conference on Artificial Intelligence, 2005.
38. S.A. Murphy. A generalization error for Q-learning. Journal of Machine Learning Research, 6: 1073-1097, 2005.
39. A.Y. Ng and M. Jordan. PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence, pages 406-415, 2000.
40. P. Niyogi and F. Girosi. Generalization bounds for function approximation from scattered noisy data. Advances in Computational Mathematics, 10:51-80, 1999.
41. D. Ormoneit and S. Sen. Kernel-based reinforcement learning. Machine Learning, 49:161-178, 2002.
42. M.L. Puterman. Markov Decision Processes - Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, 1994.
43. D. Reetz. Approximate solutions of a discounted Markovian decision problem. Bonner Mathematischer Schriften, 98: Dynamische Optimierungen:77-92, 1977.
44. M. Riedmiller. Neural fitted Q iteration - first experiences with a data efficient neural reinforcement learning method. In 16th European Conference on Machine Learning, pages 317-328, 2005.
45. J. Rust. Numerical dyanmic programming in economics. In H. Amman, D. Kendrick, and J. Rust, editors, Handbook of Computational Economics. Elsevier, North Holland, 1996a.
46. J. Rust. Using randomization to break the curse of dimensionality. Econometrica, 65:487-516, 1996b.
47. A.L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal on Research and Development, pages 210-229, 1959.
48. Reprinted in Computers and Thought, E.A. Feigenbaum and J. Feldman, editors, McGraw-Hill, New York, 1963.
49. A.L. Samuel. Some studies in machine learning using the game of checkers, II - recent progress. IBM Journal on Research and Development, pages 601-617, 1967.
50. N. Sauer. On the density of families of sets. Journal of Combinatorial Theory Series A, 13:145-147, 1972.
51. B. Schölkopf and A.J. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.
52. S.P. Singh and D.P. Bertsekas. Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Advances in Neural Information Processing Systems 9, 1997.
53. S.P. Singh, T. Jaakkola, and M.I. Jordan. Reinforcement learning with soft state aggregation. In Proceedings of Neural Information Processing Systems 7, pages 361-368. MIT Press, 1995.
54. C.J. Stone. Optimal rates of convergence for nonparametric estimators. Annals of Statistics, 8: 1348-1360, 1980.
55. C.J. Stone. Optimal global rates of convergence for nonparametric regression. Annals of Statistics, 10:1040-1053, 1982.
56. Cs. Szepesvári. Efficient approximate planning in continuous space Markovian decision problems. AI Communications, 13:163-176, 2001.
57. Cs. Szepesvári. Efficient approximate planning in continuous space Markovian decision problems. Journal of European Artificial Intelligence Research, 2000. accepted.
58. Cs. Szepesvári and R. Munos. Finite time bounds for sampling based fitted value iteration. In ICML'2005, pages 881-886, 2005.
59. Cs. Szepesvári and W.D. Smart. Interpolation-based Q-learning. In D. Schuurmans R. Greiner, editor, Proceedings of the International Conference on Machine Learning, pages 791-798, 2004.
60. G.J. Tesauro. Temporal difference learning and TD-Gammon. Communications of the ACM, 38: 58-67, March 1995.
61. J. N. Tsitsiklis and Van B. Roy. Regression methods for pricing complex American-style options. IEEE Transactions on Neural Networks, 12:694-703, 2001.
62. J. N. Tsitsiklis and B. Van Roy. Feature-based methods for large scale dynamic programming. Machine Learning, 22:59-94, 1996.
63. V. N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16:264-280, 1971.
64. X. Wang and T.G. Dietterich. Efficient value function approximation using regression trees. In Proceedings of the IJCAI Workshop on Statistical Machine Learning for Large-Scale Optimization, Stockholm, Sweden, 1999.
65. T. Zhang. Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2:527-550, 2002.
66. W. Zhang and T. G. Dietterich. A reinforcement learning approach to job-shop scheduling. In Proceedings of the International Joint Conference on Artificial Intellience, 1995.