Off-policy learning with eligibility traces: A survey

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 289-333

  Geist, Matthieu ^a   Scherrer, Bruno ^b  

^a SUPELEC   (France)

^b INRIA   (France)

Author keywords

Eligibility Traces; Off policy learning; Reinforcement learning; Value function estimation

Indexed keywords

MARKOV PROCESSES; REINFORCEMENT LEARNING;

CONVERGENCE PROPERTIES; ELIGIBILITY TRACES; GRADIENT BASED; LINEAR APPROXIMATIONS; MARKOV DECISION PROCESSES; OFF-POLICY LEARNING; STANDARD ALGORITHMS; VALUE FUNCTIONS;

LEARNING ALGORITHMS;

EID: 84897081792     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (33)

1. A. Antos, Cs. Szepesvári, and R. Munos. Learning near-optimal policies with Bellmanresidual minimization based fitted policy iteration and a single sample path. In Conference on Learning Theory (COLT), 2006.
2. T. Archibald, K. McKinnon, and L. Thomas. On the generation of Markov decision processes. Journal of the Operational Research Society, 46:354-361, 1995.
3. L.C. Baird. Residual algorithms: Reinforcement learning with function approximation. In International Conference on Machine Learning (ICML), 1995.
4. D.P. Bertsekas and S. Ioffe. Temporal differences-based policy iteration and applications in neuro-dynamic programming. Technical report, MIT, 1996.
5. D.P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.
6. D.P. Bertsekas and H. Yu. Projected equation methods for approximate solution of large linear systems. Journal of Computational and Applied Mathematics, 227:27-50, 2009.
7. L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In S. Sra, S. Nowozin, and S.J. Wright, editors, Optimization for Machine Learning, pages 351-368. MIT Press, 2011.
8. J.A. Boyan. Technical update: Least-squares temporal difference learning. Machine Learning, 49 (2-3):233-246, 1999. (Pubitemid 34325688)
9. S.J. Bradtke and A.G. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22 (1-3):33-57, 1996. (Pubitemid 126724362)
10. D. Choi and B. Van Roy. A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning. Discrete Event Dynamic Systems, 16:207-239, 2006.
11. Y. Engel. Algorithms and Representations for Reinforcement Learning. PhD thesis, Hebrew University, 2005.
12. M. Geist and O. Pietquin. Eligibility traces through colored noises. In IEEE International Conference on Ultra Modern Control Systems (ICUMT), 2010a.
13. M. Geist and O. Pietquin. Kalman temporal differences. Journal of Artifical Intelligence Research (JAIR), 39:483-532, 2010b.
14. M. Geist and O. Pietquin. Algorithmic survey of parametric value function approximation. IEEE Transactions on Neural Networks and Learning Systems, 24 (6):845-867, 2013.
15. M. Kearns and S. Singh. Bias-variance error bounds for temporal difference updates. In Conference on Learning Theory (COLT), 2000.
16. J.Z. Kolter. The fixed points of off-policy TD. In Advances in Neural Information Processing Systems (NIPS), 2011.
17. M.G. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of Machine Learning Research, 4:1107-1149, 2003. ISSN 1533-7928.
18. H.R. Maei and R.S. Sutton. GQ (-): A general gradient algorithm for temporal-difference prediction learning with eligibility traces. In Conference on Artificial General Intelligence (AGI), 2010.
19. R. Munos. Error bounds for approximate policy iteration. In International Conference on Machine Learning (ICML), 2003.
20. A. Nedic and D.P. Bertsekas. Least squares policy evaluation algorithms with linear function approximation. Discrete Event Dynamic Systems, 13:79-110, 2003.
21. D. Precup, R.S. Sutton, and S.P. Singh. Eligibility traces for off-policy policy evaluation. In International Conference on Machine Learning (ICML), 2000.
22. D. Precup, R.S. Sutton, and S. Dasgupta. Off-policy temporal-difference learning with function approximation. In International Conference on Machine Learning (ICML), 2001.
23. R.S. Randhawa and S. Juneja. Combining importance sampling and temporal difference control variates to simulate Markov chains. ACM Transactions on Modeling and Computer Simulation, 14 (1):1-30, 2004.
24. B.D. Ripley. Stochastic Simulation. Wiley & Sons, 1987.
25. B. Scherrer. Should one compute the temporal difference fix point or minimize the Bellman residual? The unified oblique projection view. In International Conference on Machine Learning (ICML), 2010.
26. B. Scherrer and M. Geist. Recursive least-squares learning with eligibility traces. In European Workshop on Reinforcement Learning (EWRL), 2011.
27. R. Schoknecht. Optimality of reinforcement learning algorithms with linear function approximation. In Advances in Neural Information Processing Systems (NIPS), 2002.
28. R.S. Sutton and A.G. Barto. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). MIT Press, 3rd edition, 1998.
29. R.S. Sutton, H.R. Maei, D. Precup, S. Bhatnagar, D. Silver, Cs. Szepesvári, and E. Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In International Conference on Machine Learning (ICML), 2009.
30. J.N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42 (5):674-690, 1997. (Pubitemid 127760263)
31. H. Yu. Convergence of least-squares temporal difference methods under general conditions. In International Conference on Machine Learning (ICML), 2010a.
32. H. Yu. Least squares temporal difference methods: An analysis under general condtions. Technical Report C-2010-39, University of Helsinki, September 2010b.
33. H. Yu and D.P. Bertsekas. New error bounds for approximations from projected linear equations. Technical Report C-2008-43, Dept. Computer Science, Univ. of Helsinki, July 2008.