Weighted importance sampling for off-policy learning with linear function approximation

Advances in Neural Information Processing Systems

Volumn 4, Issue January, 2014, Pages 3014-3022

  Mahmood, A Rupam ^a   Van Hasselt, Hado ^a   Sutton, Richard S ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; INFORMATION SCIENCE; LEARNING ALGORITHMS; REINFORCEMENT LEARNING;

FUNCTION APPROXIMATION; LINEAR FUNCTIONS; POLICY LEARNING; POLICY MODEL; RELIABLE CONVERGENCE; TRAINING SAMPLE;

IMPORTANCE SAMPLING;

EID: 84937883130     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. Andradóttir, S., and Heyman, D. P., Ott, T. J. (1995). On the choice of alternative measures in importance sampling with markov chains. Operations Research, 43(3): 509-519.
2. Bertsekas, D. P., Yu, H. (2009). Projected equation methods for approximate solution of large linear systems. Journal of Computational and Applied Mathematics, 227(1): 27-50.
3. Boyan, J. A. (1999). Least-squares temporal difference learning. In Proceedings of the 17th International Conference, pp. 49-56.
4. Casella, G., Robert, C. P. (1998). Post-processing accept-reject samples: recycling and rescaling. Journal of Computational and Graphical Statistics, 7(2): 139-157.
5. Dann, C., Neumann, G., Peters, J. (2014). Policy evaluation with temporal differences: a survey and comparison. Journal of Machine Learning Research, 15: 809-883.
6. Geist, M., Scherrer, B. (2014). Off-policy learning with eligibility traces: A survey. Journal of Machine Learning Research, 15: 289-333.
7. Hachiya, H., Akiyama, T., Sugiayma, M., Peters, J. (2009). Adaptive importance sampling for value function approximation in off-policy reinforcement learning. Neural Networks, 22(10): 1399-1410.
8. Hachiya, H., Sugiyama, M., Ueda, N. (2012). Importance-weighted least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition. Neurocomputing, 80: 93-101.
9. Hesterberg, T. C. (1988), Advances in importance sampling, Ph.D. Dissertation, Statistics Department, Stanford University.
10. Kahn, H., Marshall, A. W. (1953). Methods of reducing sample size in Monte Carlo computations. In Journal of the Operations Research Society of America, 1(5): 263-278.
11. Koller, D., Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques. MIT Press, 2009.
12. Liu, J. S. (2001). Monte Carlo strategies in scientific computing. Berlin, Springer-Verlag.
13. Maei, H. R., Sutton, R. S. (2010). GQ(λ): A general gradient algorithm for temporal-difference prediction learning with eligibility traces. In Proceedings of the Third Conference on Artificial General Intelligence, pp. 91-96. Atlantis Press.
14. Maei, H. R. (2011). Gradient temporal-difference learning algorithms. PhD thesis, University of Alberta.
15. Precup, D., and Sutton, R. S., Singh, S. (2000). Eligibility traces for off-policy policy evaluation. In Proceedings of the 17th International Conference on Machine Learning, pp. 759-766. Morgan Kaufmann.
16. Precup, D., and Sutton, R. S., Dasgupta, S. (2001). Off-policy temporal-difference learning with function approximation. In Proceedings of the 18th International Conference on Machine Learning.
17. Robert, C. P., and Casella, G., (2004). Monte Carlo Statistical Methods, New York, Springer-Verlag.
18. Rubinstein, R. Y. (1981). Simulation and the Monte Carlo Method, New York, Wiley.
19. Shelton, C. R. (2001). Importance Sampling for Reinforcement Learning with Multiple Objectives. PhD thesis, Massachusetts Institute of Technology.
20. Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2): 227-244.
21. Sutton, R. S., Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press.
22. Sutton, R. S., and Mahmood, A. R., Precup, D., van Hasselt, H. (2014). A new Q(λ) with interim forward view and Monte Carlo equivalence. In Proceedings of the 31st International Conference on Machine Learning, Beijing, China.
23. Yu, H. (2010). Convergence of least squares temporal difference methods under general conditions. In Proceedings of the 27th International Conference on Machine Learning, pp. 1207-1214.