Policy evaluation with temporal differences: A survey and comparison

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 809-883

  Dann, Christoph ^a   Neumann, Gerhard ^a   Peters, Jan ^a,b  

^a DARMSTADT UNIVERSITY OF TECHNOLOGY   (Germany)

^b MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

Author keywords

Policy evaluation; Reinforcement learning; Temporal differences; Value function estimation

Indexed keywords

REINFORCEMENT LEARNING; UNCERTAINTY ANALYSIS;

COMPARATIVE EVALUATIONS; HIGH-DIMENSIONAL FEATURE SPACE; POLICY EVALUATION; PROBABILISTIC TREATMENTS; REINFORCEMENT LEARNING APPROACH; RESIDUAL MINIMIZATION; TEMPORAL DIFFERENCES; VALUE FUNCTIONS;

FUNCTION EVALUATION;

EID: 84899800132     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Review

References (89)

1. J. S. Albus. A new approach to manipulator control: The cerebellar model articulation controller (CMAC). Journal of Dynamic Systems Measurement and Control, 97(September): 220-227, 1975.
2. S.-i. Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2): 251-276, Feb. 1998. (Pubitemid 128463152)
3. A. Antos, C. Szepesvári, and R. Munos. Learning near-optimal policies with Bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning, 71(1):89-129, 2008.
4. F. Bach and E. Moulines. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems 24, 2011.
5. L. Baird. Residual algorithms : Reinforcement learning with function approximation. In Proceedings of the Twelfth International Conference on Machine Learning, 1995.
6. P. Balakrishna, R. Ganesan, and L. Sherry. Accuracy of reinforcement learning algorithms for predicting aircraft taxi-out times: A case-study of Tampa Bay departures. Transportation Research Part C: Emerging Technologies, 18(6):950-962, 2010.
7. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, Massachusetts, 1996. ISBN 1-886529-10-8.
8. D. P. Bertsekas and H. Yu. Projected equation methods for approximate solution of large linear systems. Journal of Computational and Applied Mathematics, 227(1):27-50, 2009.
9. J. A. Boyan. Technical update: Least-squares temporal difference learning. Machine Learning, 49(2):233-246, 2002. (Pubitemid 34325688)
10. S. J. Bradtke and A. G. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22(1-3):33-57, 1996. (Pubitemid 126724362)
11. E. Candes and T. Tao. The Dantzig selector: statistical estimation when p is much larger than n. The Annals of Statistics, 35(6):2313-2351, 2005.
12. D. Choi and B. Roy. A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning. Discrete Event Dynamic Systems, 16(2):207-239, 2006.
13. R. W. Cottle, J.-S. Pang, and R. E. Stone. The Linear Complementarity Problem. Computer Science and Scientific Computing. Academic Press, 1992. ISBN 0121923509.
14. R. H. Crites and A. G. Barto. Elevator group control using multiple reinforcement learning agents. Machine Learning, 33(2-3):235-262, 1998. (Pubitemid 128522644)
15. W. Dabney and A. G. Barto. Adaptive step-size for online temporal difference learning. In Proceedings of the 26th AAAI Conference on Artificial Intelligence, 2012.
16. P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. A tutorial on the crossentropy method. Annals of Operations Research, (1):19-67, 2010.
17. M. P. Deisenroth. Efficient Reinforcement Learning using Gaussian Processes. PhD thesis, Karlsruhe Institute of Technology, 2010.
18. M. P. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on Machine Learning, 2011.
19. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. The Annals of Statistics, 32(2):407-499, 2004. (Pubitemid 41250302)
20. Y. Engel. Algorithms and Representations for Reinforcement Learning. PhD thesis, Hebrew University, 2005.
21. Y. Engel, S. Mannor, and R. Meir. Bayes meets Bellman: The Gaussian process approach to temporal difference learning. In Proceedings of the Twentieth International Conference on Machine Learning, 2003.
22. Y. Engel, S. Mannor, and R. Meir. Reinforcement learning with Gaussian processes. In Proceedings of the 22nd International Conference on Machine Learning, 2005.
23. A.-m. Farahmand and C. Szepesvári. Model selection in reinforcement learning. Machine Learning, 85(3):299-332, 2011.
24. A.-m. Farahmand, M. Ghavamzadeh, C. Szepesvári, and S. Mannor. Regularized policy iteration. In Advances in Neural Information Processing Systems 21, 2008.
25. J. Frank, S. Mannor, and D. Precup. Reinforcement learning in the presence of rare events. In Proceedings of the 25th International Conference on Machine Learning, 2008.
26. M. Geist and O. Pietquin. Kalman temporal differences. Journal of Artificial Intelligence Research, 39(1):483-532, 2010.
27. M. Geist and B. Scherrer. l1-penalized projected Bellman residual. In Proceedings of the Nineth European Workshop on Reinforcement Learning, 2011.
28. M. Geist and B. Scherrer. Off-policy learning with eligibility traces : A survey. Technical report, INRIA Lorraine - LORIA, 2013.
29. M. Geist, B. Scherrer, A. Lazaric, and M. Ghavamzadeh. A Dantzig selector approach to temporal difference learning. In Proceedings of the 29th International Conference on Machine Learning, 2012.
30. S. Gelly and D. Silver. Achieving master level play in 9 x 9 computer go. In Proceedings of the 23th AAAI Conference on Artificial Intelligence, 2008.
31. A. Geramifard, M. Bowling, and R. S. Sutton. Incremental least-squares temporal difference learning. Proceedings of the 21th AAAI Conference on Artificial Intelligence, 2006a.
32. A. Geramifard, M. Bowling, M. Zinkevich, and R. S. Sutton. iLSTD: Eligibility traces and convergence analysis. In Advances in Neural Information Processing Systems 19, 2006b.
33. A. Geramifard, F. Doshi, J. Redding, N. Roy, and J. P. How. Online discovery of feature dependencies. In Proceedings of the 28th International Conference on Machine Learning, 2011.
34. A. Geramifard, T. J. Walsh, and J. P. How. Batch-iFDD for representation expansion in large MDPs. In Conference on Uncertainty in Artificial Intelligence, 2013.
35. M. Ghavamzadeh, A. Lazaric, O.-A. Maillard, and R. Munos. LSTD with random projections. In Advances in Neural Information Processing Systems 23, 2010.
36. M. Ghavamzadeh, A. Lazaric, R. Munos, and M. Hoffman. Finite-sample analysis of Lasso- TD. In Proceedings of the 28th International Conference on Machine Learning, 2011.
37. P. W. Glynn and D. L. Iglehart. Importance sampling for stochastic simulations. Management Science, 35(11):1367-1392, 1989.
38. H. Hachiya and M. Sugiyama. Feature selection for reinforcement learning: Evaluating implicit state-reward dependency via conditional mutual information. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2010.
39. M. Hoffman, A. Lazaric, M. Ghavamzadeh, and R. Munos. Regularized least squares temporal difference learning with nested l2 and l1 penalization. In Proceedings of the Nineth European Workshop on Reinforcement Learning, 2011.
40. M. Hutter and S. Legg. Temporal difference updating without a learning rate. In Advances in Neural Information Processing Systems 20, 2007.
41. R. Jenatton, J. Mairal, G. Obozinski, and F. Bach. Proximal methods for sparse hierarchical dictionary learning. In Proceedings of the 27th International Conference on Machine Learning, 2010.
42. J. Johns and S. Mahadevan. Sparse approximate policy evaluation using graph-based basis functions. Technical report, University of Massachusetts Amherst, 2009.
43. J. Johns, C. Painter-Wakefield, and R. Parr. Linear complementarity for regularized policy evaluation and improvement. In Advances in Neural Information Processing Systems 23, 2010.
44. T. Jung and D. Polani. Least squares SVM for least squares TD learning. In European Conference on Artificial Intelligence, 2006.
45. P. W. Keller, S. Mannor, and D. Precup. Automatic basis function construction for approximate dynamic programming and reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning, 2006.
46. J. Z. Kolter and A. Y. Ng. Regularization and feature selection in least-squares temporal difference learning. In Proceedings of the 26th Annual International Conference on Machine Learning, 2009.
47. R. M. Kretchmar and C. W. Anderson. Comparison of CMACs and radial basis functions for kocal function approximators in reinforcement learning. In International Conference on Neural Networks, 1997.
48. M. G. Lagoudakis and R. Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4(Dec):1107-1149, 2003.
49. A. Lazaric, M. Ghavamzadeh, and R. Munos. Finite-sample analysis of LSTD. In Proceedings of the 27th International Conference on Machine Learning, 2010.
50. L. Li. A worst-case comparison between temporal difference and residual gradient with linear function approximation. In Proceedings of the 25th International Conference on Machine Learning, 2008.
51. B. Liu, S. Mahadevan, and J. Liu. Regularized off-policy TD-learning. In Advances in Neural Information Processing Systems 25, 2012.
52. M. Loth, M. Davy, and P. Preux. Sparse temporal difference learning using LASSO. In IEEE Symposium on Adaptive Dynamic Programming And Reinforcement Learning, 2007.
53. H. R. Maei. Gradient Temporal-Difference Learning Algorithms. PhD thesis, University of Alberta, 2011.
54. S. Mahadevan and M. Maggioni. Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research, 8(Oct):2169-2231, 2007. (Pubitemid 350046199)
55. A. R. Mahmood, R. S. Sutton, T. Degris, and P. M. Pilarski. Tuning-free step-size adaptation. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2012.
56. I. Menache, S. Mannor, and N. Shimkin. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research, 134(1):215-238, 2005. (Pubitemid 40550047)
57. D. Meyer, H. Shen, and K. Diepold. l1-regularized gradient temporal-difference learning. In Proceedings of the Tenth European Workshop on Reinforcement Learning, 2012.
58. A. Nedic and D. P. Bertsekas. Least squares policy evaluation algorithms with linear function approximation. Discrete Event Dynamic Systems, 13(1-2):79-110, 2003.
59. C. Painter-Wakefield and R. Parr. Greedy algorithms for sparse reinforcement learning. In Proceedings of the 29th International Conference on Machine Learning, 2012a.
60. C. Painter-Wakefield and R. Parr. L1 regularized linear temporal difference learning. Technical report, Duke University, Durham, NC, 2012b.
61. R. Parr, C. Painter-Wakefield, L. Li, and M. Littman. Analyzing feature generation for value-function approximation. In Proceedings of the 24th International Conference on Machine Learning, 2007.
62. R. Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. L. Littman. An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In Proceedings of the 25th International Conference on Machine Learning, 2008.
63. Y. Pati, R. Rezaiifar, and P. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers, 1993.
64. M. Petrik, G. Taylor, R. Parr, and S. Zilberstein. Feature selection using regularization in approximate linear programs for Markov decision processes. In Proceedings of the 27th International Conference on Machine Learning, 2010.
65. B. A. Pires. Statistical Analysis of L1-penalized Linear Estimation with Applications. Master thesis, University of Alberta, 2011.
66. C. E. Rasmussen and M. Kuss. Gaussian processes in reinforcement learning. In Advances in Neural Information Processing Systems 16, 2003.
67. M. Riedmiller and T. Gabel. On experiences in a complex and competitive gaming domain: Reinforcement learning meets robocup. In IEEE Symposium on Computational Intelligence and Games, 2007.
68. H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400-407, 1951.
69. M. Rosenblatt. Markov Processes. Structure and Asymptotic Behavior. Springer, 1971. ISBN 978-3642652400.
70. N. L. Roux and A. Fitzgibbon. A fast natural Newton method. In Proceedings of the 27th International Conference on Machine Learning, 2010.
71. A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):210-229, 1959.
72. B. Scherrer. Should one compute the temporal difference fix point or minimize the Bellman residual? the unified oblique projection view. In Proceedings of the 27th International Conference on Machine Learning, 2010.
73. B. Scherrer and M. Geist. Recursive least-squares learning with eligibility traces. In Proceedings of the Nineth European Workshop on Reinforcement Learning, 2011.
74. R. Schoknecht. Optimality of reinforcement learning algorithms with linear function approximation. In Advances in Neural Information Processing Systems 15, 2002.
75. P. J. Schweitzer and A. Seidmann. Generalized polynomial approximations in Markovian decision processes. Journal of Mathematical Analysis and Applications, 110(2):568-582, 1985.
76. D. Silver, R. Sutton, and M. Müller. Reinforcement learning of local shape in the game of go. In International Joint Conference on Artificial Intelligence, 2007.
77. S. Sra, S. Nowozin, and S. J. Wright. Optimization for Machine Learning. MIT Press, 2012.ISBN 9780262016469.
78. R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, 1988.
79. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 9780262193986.
80. R. S. Sutton, D. Precup, and S. Singh. Intra-option learning about temporally abstract actions. In Proceedings of the 15th International Conference on Machine Learning, 1998.
81. R. S. Sutton, C. Szepesvári, and H. R. Maei. A convergent O(n) algorithm for off-policy temporal-difference learning with linear function approximation. In Advances in Neural Information Processing Systems 21, 2008.
82. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th Annual International Conference on Machine Learning, 2009.
83. G. Taylor and R. Parr. Kernelized value function approximation for reinforcement learning. In Proceedings of the 26th Annual International Conference on Machine Learning, 2009.
84. G. Tesauro. TD-gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 6(2):215-219, 1994.
85. J. N. Tsitsiklis and B. van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions On Automatic Control, 42(5):674-690, 1997. (Pubitemid 127760263)
86. R. J. Williams and L. C. Baird. Tight performance bounds on greedy policies based on imperfect value functions. In Yale Workshop on Adaptive and Learning Systems, 1993.
87. X. Xu, T. Xie, D. Hu, and X. Lu. Kernel least-squares temporal difference learning. International Journal of Information Technology, 11(9):54-63, 2005.
88. H. Yu. Convergence of least squares temporal difference methods under general conditions In Proceedings of the 27th International Conference on Machine Learning, 2010.
89. P. Zhao, G. Rocha, and B. Yu. The composite absolute penalties family for grouped and hierarchical variable selection. The Annals of Statistics, 37(6A):3468-3497, 2009.