Adaptive importance sampling for value function approximation in off-policy reinforcement learning

Neural Networks

Volumn 22, Issue 10, 2009, Pages 1399-1410

  Hachiya, Hirotaka ^a   Akiyama, Takayuki ^a   Sugiayma, Masashi ^a   Peters, Jan ^b  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

Author keywords

Adaptive importance sampling; Efficient sample reuse; Importance weighted cross validation; Off policy reinforcement learning; Policy iteration; Value function approximation

Indexed keywords

ADAPTIVE IMPORTANCE SAMPLING; BIAS AND VARIANCE; CROSS VALIDATION; DATA SAMPLE; EFFICIENT SAMPLE REUSE; IMPORTANCE SAMPLING; VALUE FUNCTION APPROXIMATION; VALUE FUNCTIONS;

EDUCATION; REINFORCEMENT LEARNING;

REINFORCEMENT;

ALGORITHM; ARTICLE; LEARNING; MATHEMATICAL ANALYSIS; POLICY; PRIORITY JOURNAL; PROBABILITY; REINFORCEMENT; SAMPLING; SIMULATION; VALIDATION PROCESS;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DATA INTERPRETATION, STATISTICAL; LEARNING; MARKOV CHAINS; MODELS, NEUROLOGICAL; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PUBLIC POLICY; REINFORCEMENT (PSYCHOLOGY); REPRODUCIBILITY OF RESULTS;

EID: 70549113878     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2009.01.002     Document Type: Article

References (19)

1. Bertsekas P.D., and Tsitsiklis J. Neuro-dynamic programming (1996), Athena Scientific, NH, USA
2. Bugeja, M. (2003). Non-linear swing-up and stabilizing control of an inverted pendulum system. In Proceedings of IEEE Region 8 EUROCON (pp. 437-441)
3. Fishman G.S. Monte carlo: Concepts, algorithms, and applications (1996), Springer-Verlag, Berlin
4. Hastie T., Tibshirani R., and Friedman J. The elements of statistical learning: Data mining, inference, and precition (2001), Springer-Verlag, New York
5. Kakade S. A natural policy gradient. Neural Information Processing Systems 14 (2002) 1531-1538
6. Lagoudakis M.G., and Parr R. Least-squares policy iteration. Journal of Machine Learning Research 4 Dec (2003) 1107-1149
7. Peshkin, L., & Shelton, C. R. (2002). Learning from scarce experience. In Proceedings of international conference on machine learning (pp. 498-505)
8. Peters, J., & Schaal, S. (2007). Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the international conference on machine learning
9. Peters J., and Schaal S. Reinforcement learning of motor skills with policy gradients. Neural Networks 21 (2008) 682-697
10. Precup, D., Sutton, R. S., & Dasgupta, S. (2001). Off-policy temporal-difference learning with function approximation. In Proceedings of international conference on machine learning (pp. 417-424)
11. Precup, D., Sutton, R. S., & Singh, S. (2000). Eligibility traces for off-policy policy evaluation. In Proceedings of international conference on machine learning (pp. 759-766)
12. Rao C.R. Linear statistical inference and its applications (1973), Wiley, New York
13. Schoknecht R. Optimality of reinforcement learning algorithms with linear function approximation. Neural Information Processing Systems 15 (2003) 1555-1562
14. Shelton, C. R. (2001). Policy improvement for pomdps using normalized importance sampling. In Proceedings of Uncertainty in Artificial Intelligence (pp. 496-503)
15. Shimodaira H. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90 (2000) 227-244
16. Sugiyama M., Kawanabe M., and Müller K.-R. Trading variance reduction with unbiasedness: The regularized subspace information criterion for robust model selection in kernel regression. Neural Computation 16 (2004) 1077-1104
17. Sugiyama M., Krauledat M., and Müller K.-R. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research 8 May (2007) 985-1005
18. Sutton R.S., and Barto A.G. Reinforcement learning: An introduction (1998), The MIT Press, MA, USA
19. Wang H.O., Tanaka K., and Griffin M.F. An approach to fuzzy control of nonlinear systems: Stability and design issues. IEEE Transactions on Fuzzy Systems (1996) 14-23