Kernel-based reinforcement learning in average-cost problems

IEEE Transactions on Automatic Control

Volumn 47, Issue 10, 2002, Pages 1624-1636

  Ormoneit, Dirk ^a   Glynn, Peter ^b  

^a Marshall Wace Asset Management   (United Kingdom)

^b Stanford University   (United States)

Author keywords

Average cost problem; Dynamic programming; Kernel smoothing; Local averaging; Markov decision process (MDP); Perturbation theory; Policy iteration; Reinforcement learning; Temporal difference learning

Indexed keywords

APPROXIMATION THEORY; DECISION THEORY; DYNAMIC PROGRAMMING; ITERATIVE METHODS; LEARNING SYSTEMS; MARKOV PROCESSES;

REINFORCEMENT LEARNING;

OPTIMAL CONTROL SYSTEMS;

EID: 0036804005     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAC.2002.803530     Document Type: Article

References (30)

1. R.S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, pp. 9-44, 1988.
2. D.P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995.
3. J.N. Tsitsiklis and B. Van Roy, "Feature-based methods for large-scale dynamic programming," Mach. Learn., vol. 22, pp. 59-94, 1996.
4. D. Ormoneit and Ś. Sen, "Kernel-based reinforcement learning," Mach. Learn., vol. 49, pp. 161-178, 2002.
5. A. Arapostathis, V.S. Borkhar, E. Fernández-Gaucherand, M.K. Ghosh, and S.I. Marcus, "Discrete-time controlled Markov processes with average cost criterion: A survey," SIAM J. Control Optim., vol. 31, no. 2, pp. 282-344, 1993.
6. S.P. Meyn, "The policy iteration algorithm for average reward Markov decision processes with general state space," IEEE Trans. Automat. Contr., vol. 42, pp. 1382-1393, Oct. 1997.
7. J. Rust, "Using randomization to break the curse of dimensionality," Econometrica, vol. 65, no. 3, pp. 487-516, 1997.
8. G. Gordon, "Approximate solutions to Markov decision processes," Ph.D. dissertation, Comput. Sci. Dept., Carnegie Mellon Univ., Pittsburgh, PA, 1999.
9. G.J. Gordon, "Stable fitted reinforcement learning," in Advances in Neural Information Processing Systems, D. Touretzky, M. Mozer, and M. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996, vol. 8.
10. W.L. Baker, "Learning via stochastic approximation in function space," Ph.D. dissertation, Div. Appl. Sci., Harvard Univ., Cambridge, MA, 1997.
11. V.S. Borkar, "A learning algorithm for discrete-time stochastic control," Probab. Eng. Inform. Sci., vol. 14, pp. 243-258, 2000.
12. R. Munos and A. Moore, "Variable resolution discretization in optimal control," Mach. Learn., vol. 49, pp. 291-324, 2002.
13. J.N. Tsitsiklis and V.R. Konda, "Actor-critic algorithms," Tech. Rep., Lab. Inform. Decision Systems., Mass. Inst. Technol.., Cambridge, MA, 2001, Preprint.
14. S.P. Meyn and R.L. Tweedie, Markov Chains and Stochastic Stability. New York: Springer-Verlag, 1993.
15. E.A. Nadaraya, "On estimating regression," Theor. Probab. Appl., vol. 9, pp. 141-142, 1964.
16. G.S. Watson, "Smooth regression analysis," Sankhyã Series A, vol. 26, pp. 359-372, 1964.
17. C.J. Stone, "Optimal global rates of convergence for nonparametric regression," Ann. Stat., vol. 10, no. 4, pp. 1040-1053, 1982.
18. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1996.
19. H. Niederreiter, Random Number Generation and Quasi-Monte Carlo Methods. Philadelphia, PA: SIAM, 1992.
20. L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone, Classification and Regression Trees. Belmont, CA: Wadsworth, 1983.
21. C.J. Stone, "Consistent nonparametric regression," Ann. Stat., vol. 5, no. 4, pp. 595-645, 1977.
22. D. Ormoneit and P.W. Glynn, "Kernel-based reinforcement learning in average-cost problems: An application to optimal portfolio choice," in Advances in Neural Information Processing Systems 13. Cambridge, MA: MIT Press, 2001.
23. J. Fan and I. Gijbels, Local Polynomial Modeling and Its Applications. London, U.K.: Chapman & Hall, 1996.
24. D. Ormoneit and P.W. Glynn, "Kernel-based reinforcement learning in average-cost problems," Tech. Rep., Dept. Comput. Sci., Stanford Univ., Stanford, CA, 2001.
25. M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York: Wiley, 1994.
26. X.-R. Cat, "Perturbation realization, potentials, and sensitivity analysis of Markov processes," IEEE Trans. Automat. Contr., vol. 42, pp. 1382-1393, Oct. 1997.
27. P.W. Glynn and D. Ormoneit, "Hoeffding's inequality for uniformly ergodic Markov chains," Stat. Probab. Lett., vol. 56, pp. 143-146, 2002.
28. V.N. Vapnik The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
29. D. Ormoneit and T. Hastie, "Optimal kernel shapes for local linear regression," in Advances in Neural Information Processing Systems 12, S.A. Solla, T.K. Leen, and K-R. Müller, Eds. Cambridge, MA: MIT Press, 2000, pp. 540-546.
30. L. Devroye, "The uniform convergence of nearest neighbor regression function estimators and their application in optimization," IEEE Trans. Inform. Theory, vol. IT-24, pp. 142-151, Feb. 1978.