Adaptive-resolution reinforcement learning with polynomial exploration in deterministic domains

Machine Learning

Volumn 81, Issue 3, 2010, Pages 359-397

  Bernstein, Andrey ^a   Shimkin, Nahum ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Adaptive resolution; Efficient exploration; Kernel functions; Reinforcement learning

Indexed keywords

ADAPTIVE APPROXIMATION; ADAPTIVE RESOLUTION; APPROXIMATION SCHEME; COARSE TO FINE; COARSER RESOLUTION; CONTINUOUS STATE SPACE; DETERMINISTIC DOMAINS; EFFICIENT EXPLORATION; EXPLORATION TECHNIQUES; KERNEL FUNCTION; LEARNING RATES; MISTAKE BOUNDS; MODEL-BASED; ONLINE LEARNING; OPTIMAL VALUE FUNCTIONS; ORIGINAL ALGORITHMS; STATE SPACE; UNCERTAINTY INTERVALS;

ADAPTIVE ALGORITHMS; APPROXIMATION ALGORITHMS; COMPUTATIONAL COMPLEXITY; POLYNOMIAL APPROXIMATION; REINFORCEMENT LEARNING;

LEARNING ALGORITHMS;

EID: 78649716899     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-010-5186-7     Document Type: Article

References (32)

1. J. S. Albus 1975 A new approach to manipulator control: the cerebellar model articulation controller (CMAC) Journal of Dynamic Systems, Measurement and Control 97 220 227 0314.92007
2. A. Antos C. Szepesvári R. Munos 2008 Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path Machine Learning 71 1 89 129 10.1007/s10994-007-5038-2
3. Auer, P., & Ortner, R. (2006). Logarithmic online regret bounds for undiscounted reinforcement learning. In Proceedings of neural information processing systems conference (NIPS).
4. Bernstein, A. (2007). Adaptive state aggregation for reinforcement learning. Master's thesis, Technion-Israel Institute of Technology. URL: http://tx.technion.ac.il/~andreyb/MSc-Thesis-final.pdf.
5. Bernstein, A., & Shimkin, N. (2008). Adaptive aggregation for reinforcement learning with efficient exploration: deterministic domains. In Proceedings of the 21st annual conference on learning theory (COLT 2008).
6. Bertsekas, D. P. (2007). Dynamic programming and optimal control (3rd ed., vol. 2). Belmont: Athena Scientific.
7. Bonarini, A., Lazaric, A., & Restelli, M. (2005). LEAP: learning entities adaptive partitioning. In Proceedings of neural information processing systems conference (NIPS 2005), workshop on reinforcement learning benchmarks and bake-offs II, Whistler, Canada (pp. 41-47).
8. C. Boutilier T. Dean S. Hanks 1999 Decision-theoretic planning: structural assumptions and computational leverage Journal of Artificial Intelligence Research 11 1 94 0918.68110 1718251
9. R. I. Brafman M. Tennenholtz 2002 R-MAX-a general polynomial time algorithm for near-optimal reinforcement learning Journal of Machine Learning Research 3 213 231 10.1162/153244303765208377 1971337
10. Chapman, H. (2007). Global confidence bound algorithms for the exploration-exploitation tradeoff in reinforcement learning. Master's thesis, Technion-Israel Institute of Technology.
11. C.-S. Chow J. N. Tsitsiklis 1991 An optimal one-way multigrid algorithm for discrete-time stochastic control IEEE Transactions on Automatic Control 36 8 898 914 0752.93078 10.1109/9.133184 1116447 (Pubitemid 21674882)
12. K. Doya 2000 Reinforcement learning in continuous time and space Neural Computation 12 219 245 10.1162/089976600300015961
13. Jong, N., & Stone, P. (2006). Kernel-based models for reinforcement learning in continuous state spaces. In 23th international conference on machine learning (ICML 2006), workshop on kernel machines and reinforcement learning.
14. Kakade, S. M. (2003). On the sample complexity of reinforcement learning. PhD thesis, Gatsby Computational Neuroscience Unit, University College London, UK.
15. M. Kearns S. P. Singh 2002 Near-optimal reinforcement learning in polynomial time Machine Learning 49 209 232 1014.68071 10.1023/A:1017984413808 (Pubitemid 34325687)
16. V. R. Konda J. N. Tsitsiklis 2003 On actor-critic algorithms SIAM Journal on Control and Optimization 42 4 1143 1166 1049.93095 10.1137/S0363012901385691 2044789
17. Loth, M., Davy, M., Coulom, R., & Preux, P. (2006) Equi-gradient temporal difference learning. In 23th international conference on machine learning (ICML 2006), workshop on kernel machines and reinforcement learning.
18. A. W. Moore C. G. Atkeson 1995 The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces Machine Learning 21 199 233
19. R. Munos A. W. Moore 2002 Variable resolution discretization in optimal control Machine Learning 49 291 323 1005.68086 10.1023/A:1017992615625 (Pubitemid 34325691)
20. R. Munos C. Szepesvári 2008 Finite-time bounds for fitted value iteration Journal of Machine Learning Research 9 815 857
21. Nouri, A., & Littman, M. L. (2008). Multi-resolution exploration in continuous spaces. In Advances in neural information processing systems (NIPS) 21 (pp. 1209-1216).
22. D. Ormoneit S. Sen 2002 Kernel-based reinforcement learning Machine Learning 49 161 178 1014.68069 10.1023/A:1017928328829 (Pubitemid 34325684)
23. Powell, W. B. (2007). Approximate dynamic programming for operations research: solving the curses of dimensionality. New York: Wiley.
24. Puterman, M. L. (1994). Markov decision processes: discrete stochastic dynamic programming. New York: Wiley.
25. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1995). Reinforcement learning with soft state aggregation. In Advances in neural information processing systems (NIPS) 7 (pp. 361-368).
26. Strehl, A. L., & Littman, M. L. (2005). A theoretical analysis of model-based interval estimation. In Proceedings of the 22nd international conference on machine learning (pp. 857-864).
27. Strehl, A. L., Li, L., & Littman, M. L. (2006a). Incremental model-based learners with formal learning-time guarantees. In Proceedings of the 22nd international conference on uncertainty in artificial intelligence (pp. 485-493).
28. Strehl, A. L., Wiewiora, E., Langford, J., & Littman, M. L. (2006b). PAC model-free reinforcement learning. In Proceedings of the 23nd international conference on machine learning (pp. 881-888).
29. Sutton, R. S. (1996). Generalization in reinforcement learning: successful examples using sparse coarse coding. In Advances in neural information processing systems 8 (NIPS) (pp. 1038-1044).
30. Tewari, A., & Bartlett, P. L. (2007). Optimistic linear programming gives logarithmic regret for irreducible MDPs. In Proceedings of neural information processing systems conference (NIPS).
31. M. Unser 1999 Splines: A perfect fit for signal and image processing IEEE Signal Processing Magazine 16 22 38 10.1109/79.799930
32. W. Whitt 1978 Approximations of dynamic programs, I Mathematics of Operations Research 3 3 231 243 0393.90094 10.1287/moor.3.3.231 506661