A tutorial on linear function approximators for dynamic programming and reinforcement learning

Foundations and Trends in Machine Learning

Volumn 6, Issue 4, 2013, Pages 375-454

  Geramifard, Alborz ^a   Walsh, Thomas J ^a   Tellex, Stefanie ^b   Chowdhary, Girish ^a   Roy, Nicholas ^b   How, Jonathan P ^a  

^a MIT LIDS   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATE DYNAMIC PROGRAMMING; COMPLEXITY ANALYSIS; DECISION-MAKING PROBLEM; DYNAMIC PROGRAMMING METHODS; EMPIRICAL EVALUATIONS; MARKOV DECISION PROCESSES; REINFORCEMENT LEARNING METHOD; UNIFIED FRAMEWORK;

DYNAMIC PROGRAMMING; LEARNING ALGORITHMS; LEAST SQUARES APPROXIMATIONS; MARKOV PROCESSES; REINFORCEMENT LEARNING;

ITERATIVE METHODS;

EID: 84890920160     PISSN: 19358237     EISSN: 19358245     Source Type: Journal    
DOI: 10.1561/2200000042     Document Type: Review

References (107)

1. RL competition. http://www.rl-competition.org/, 2012. Accessed: 20/08/2012.
2. M. Ahmadi, M. E. Taylor, and P. Stone. IFSA: incremental feature-set augmentation for reorcement learning tasks. In International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1-8, New York, NY, USA, 2007.
3. ACM. . A. Antos, R. Munos, and C. Szepesvári. Fitted Q-iteration in continuous action-space MDPs. In Proceedings of Neural Information Processing Systems Conference (NIPS), 2007.
4. A. Antos, C. Szepesvári, and R. Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71(1):89-129, 2008.
5. J. Asmuth, L. Li, M. Littman, A. Nouri, and D. Wingate. A Bayesian sampling approach to exploration in reinforcement learning. In International Conference on Uncertainty in Artificial Intelligence (UAI), pages 19-26, Arlington, Virginia, United States, 2009.
6. AUAI Press. L. C. Baird. Residual algorithms: Reinforcement learning with function approximation. In ICML, pages 30-37, 1995.
7. A. d. M. S. Barreto and C. W. Anderson. Restricted gradient-descent algorithm for value-function approximation in reinforcement learning. Artificial Intelligence, 172:454 - 482, 2008.
8. A. Barto and M. Duff. Monte carlo matrix inversion and reinforcement learning. In Neural Information Processing Systems (NIPS), pages 687-694. Morgan Kaufmann, 1994.
9. A. Barto, S. Bradtke, and S. Singh. Learning to act using real-time dynamic programming. Artificial Intelligence, 72:81-138, 1995.
10. J. Baxter and P. Bartlett. Direct gradient-based reinforcement learning. In Circuits and Systems, 2000.
11. Proceedings. ISCAS 2000 Geneva. The 2000 IEEE International Symposium on, volume 3, pages 271-274. IEEE, 2000.
12. R. E. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957.
13. D. Bertsekas and J. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.
14. D. P. Bertsekas. Dynamic Programming and Optimal Control. AP, 1976.
15. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming (Optimization and Neural Computation Series, 3). Athena Scientific, May 1996.
16. B. Bethke and J. How. Approximate Dynamic Programming Using Bellman Residual Elimination and Gaussian Process Regression. In American Control Conference (ACC), St. Louis, MO, 2009.
17. S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee. Incremental natural actorcritic algorithms. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems (NIPS), pages 105-112. MIT Press, 2007.
18. M. Bowling, A. Geramifard, and D. Wingate. Sigma Point Policy Iteration. In International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), volume 1, pages 379-386, Richland, SC, 2008.
19. J. Boyan and A. Moore. Generalization in reinforcement learning: Safely approximating the value function. In G. Tesauro, D. Touretzky, and T. Lee, editors, Neural Information Processing Systems (NIPS), pages 369-376, Cambridge, MA, 1995.
20. The MIT Press. J. A. Boyan. Least-squares temporal difference learning. In International Conference on Machine Learning (ICML), pages 49-56. Morgan Kaufmann, San Francisco, CA, 1999.
21. S. J. Bradtke and A. G. Barto. Linear least-squares algorithms for temporal difference learning. Journal of Machine Learning Research (JMLR), 22:33-57, 1996. (Pubitemid 126724362)
22. R. Brafman and M. Tennenholtz. R-Max - A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research (JMLR), 3:213-231, 2002.
23. L. Bu̧soniu, R. Babuška, B. De Schutter, and D. Ernst. Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC Press, Boca Raton, Florida, 2010.
24. T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence and Research (JAIR), 13(1):227- 303, Nov. 2000. (Pubitemid 33682087)
25. A. Dutech, T. Edmunds, J. Kok, M. Lagoudakis, M. Littman, M. Riedmiller, B. Russell, B. Scherrer, R. Sutton, S. Timmer, et al. Reinforcement learning benchmarks and bake-offs II. In Advances in Neural Information Processing Systems (NIPS) 17 Workshop, 2005.
26. Y. Engel, S. Mannor, and R. Meir. Bayes meets bellman: The gaussian process approach to temporal difference learning. In International Conference on Machine Learning (ICML), pages 154-161, 2003.
27. A. Farahmand. Regularities in Sequential Decision-Making Problems. PhD thesis, Department of Computing Science, University of Alberta, 2009.
28. A. Farahmand, M. Ghavamzadeh, C. Szepesvári, and S. Mannor. Regularized policy iteration. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems (NIPS), pages 441-448. MIT Press, 2008.
29. P. Geibel and F. Wysotzki. Risk-sensitive reinforcement learning applied to chance constrained control. Journal of Artificial Intelligence and Research (JAIR), 24, 2005.
30. A. Geramifard, F. Doshi, J. Redding, N. Roy, and J. How. Online discovery of feature dependencies. In L. Getoor and T. Scheffer, editors, International Conference on Machine Learning (ICML), pages 881-888. ACM, June 2011.
31. A. Geramifard, J. Redding, J. Joseph, N. Roy, and J. P. How. Model estimation within planning and learning. In American Control Conference (ACC), June 2012.
32. A. Geramifard, R. H. Klein, and J. P. How. RLPy: The Reinforcement Learning Library for Education and Research. http://acl.mit.edu/RLPy, April 2013a.
33. Geramifard, T. J. Walsh, N. Roy, and J. How. Batch iFDD: A Scalable Matching Pursuit Algorithm for Solving MDPs. In Proceedings of the 29th Annual Conference on Uncertainty in Artificial Intelligence (UAI), Bellevue, Washington, USA, 2013b. AUAI Press.
34. S. Girgin and P. Preux. Feature Discovery in Reinforcement Learning using Genetic Programming. Research Report RR-6358, INRIA, 2007.
35. G. H. Golub and C. F. V. Loan. Matrix Computations. The John Hopkins University Press, 1996.
36. G. Gordon. Stable function approximation in dynamic programming. In International Conference on Machine Learning (ICML), page 261, Tahoe City, California, July 9-12 1995.
37. Morgan Kaufmann. A. Gosavi. Reinforcement learning: A tutorial survey and recent advances. INFORMS J. on Computing, 21(2):178-192, April 2009.
38. H. Hachiya, T. Akiyama, M. Sugiyama, and J. Peters. Adaptive importance sampling with automatic model selection in value function approximation. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1351-1356, 2008.
39. S. Haykin. Neural Networks: A Comprehensive Foundation. Macmillan, New York, 1994.
40. R. A. Howard. Dynamic Programming and Markov Processes. MIT Press, Cambridge, Massachusetts, 1960.
41. T. Jaakkola, M. Jordan, and S. Singh. on the convergence of stochastic iterative dynamic programming algorithms. Technical report, Massachusetts Institute of Technology, Cambridge, MA, August 1993.
42. T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research (JMLR), 11:1563-1600, 2010.
43. W. Josemans. Generalization in Reinforcement Learning. PhD thesis, University of Amsterdam, 2009.
44. T. Jung and P. Stone. Gaussian processes for sample efficient reinforcement learning with RMAX-like exploration. In European Conference on Machine Learning (ECML), September 2010.
45. L. P. Kaelbling, M. L. Littman, and A.W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence and Research (JAIR), 4:237-285, 1996. (Pubitemid 126646155)
46. S. Kalyanakrishnan and P. Stone. Characterizing reinforcement learning methods through parameterized learning problems. Machine Learning, 2011.
47. J. Z. Kolter and A. Y. Ng. Regularization and feature selection in least-squares temporal difference learning. In International Conference on Machine Learning (ICML), pages 521-528, New York, NY, USA, 2009.
48. ACM. R. Kretchmar and C. Anderson. Comparison of cmacs and radial basis functions for local function approximators in reinforcement learning. In International Conference on Neural Networks, volume 2, pages 834-837 vol.2, 1997.
49. O. Kroemer and J. Peters. A non-parametric approach to dynamic programming. In Advances in Neural Information Processing Systems (NIPS), pages 1719-1727, 2011.
50. L. Kuvayev and R. Sutton. Model-based reinforcement learning with an approximate, learned model. In Proceeding of the ninth Yale workshop on adaptive and learning systems, pages 101-105, 1996.
51. M. G. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of Machine Learning Research (JMLR), 4:1107-1149, 2003.
52. L. Li. Sample complexity bounds of exploration. In M. Wiering and M. van Otterlo, editors, Reinforcement Learning: State of the Art. Springer Verlag, 2012.
53. L. Li, M. L. Littman, and C. R. Mansley. Online exploration in least-squares policy iteration. In International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 733-739, Richland, SC, 2009a.
54. International Foundation for Autonomous Agents and Multiagent Systems. L. Li, J. D. Williams, and S. Balakrishnan. Reinforcement learning for dialog management using least-squares policy iteration and fast feature selection. In New York Academy of Sciences Symposium on Machine Learning, 2009b.
55. W. Liu, P. Pokharel, and J. Principe. The kernel least-mean-square algorithm. IEEE Transactions on Signal Processing, 56(2):543-554, 2008. (Pubitemid 351285052)
56. W. Liu, J. C. Principe, and S. Haykin. Kernel Adaptive Filtering: A Comprehensive Introduction. Wiley, Hoboken, New Jersey, 2010.
57. H. R. Maei and R. S. Sutton. GQ(-): A general gradient algorithm for temporaldifference prediction learning with eligibility traces. In Proceedings of the Third Conference on Artificial General Intelligence (AGI), Lugano, Switzerland, 2010.
58. H. R. Maei, C. Szepesvári, S. Bhatnagar, and R. S. Sutton. Toward off-policy learning control with function approximation. In J. Fürnkranz and T. Joachims, editors, International Conference on Machine Learning (ICML), pages 719-726. Omnipress, 2010.
59. S. Mahadevan. Representation policy iteration. International Conference on Uncertainty in Artificial Intelligence (UAI), 2005.
60. Mausam and A. Kolobov. Planning with Markov Decision Processes: An AI Perspective. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2012.
61. F. S. Melo, S. P. Meyn, and M. I. Ribeiro. An analysis of reinforcement learning with function approximation. In International Conference on Machine Learning (ICML), pages 664-671, 2008.
62. O. Mihatsch and R. Neuneier. Risk-sensitive reinforcement learning. Journal of Machine Learning Research (JMLR), 49(2-3):267-290, 2002. (Pubitemid 34325690)
63. J. Moody and C. J. Darken. Fast learning in networks of locally-tuned processing units. Neural Computation, 1(2):281-294, June 1989.
64. A.W. Moore and C. G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. In Machine Learning, pages 103-130, 1993.
65. A. Nouri and M. L. Littman. Multi-resolution exploration in continuous spaces. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems (NIPS), pages 1209-1216. MIT Press, 2009.
66. R. Parr, C. Painter-Wakefield, L. Li, and M. Littman. Analyzing feature generation for value-function approximation. In International Conference on Machine Learning (ICML), pages 737-744, New York, NY, USA, 2007.
67. ACM. R. Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. L. Littman. An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In International Conference on Machine Learning (ICML), pages 752-759, New York, NY, USA, 2008.
68. ACM. J. Peters and S. Schaal. Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2219-2225. IEEE, October 2006. (Pubitemid 46928224)
69. J. Peters and S. Schaal. Natural actor-critic. Neurocomputing, 71:1180-1190, March 2008.
70. M. Petrik, G. Taylor, R. Parr, and S. Zilberstein. Feature selection using regularization in approximate linear programs for Markov decision processes. In International Conference on Machine Learning (ICML), 2010.
71. M. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994.
72. C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006.
73. B. Ratitch and D. Precup. Sparse distributed memories for on-line value-based reinforcement learning. In European Conference on Machine Learning (ECML), pages 347-358, 2004. (Pubitemid 41050104)
74. M. Riedmiller, J. Peters, and S. Schaal. Evaluation of policy gradient methods and variants on the Cart-Pole benchmark. In IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL), pages 254-261, April 2007. (Pubitemid 47431393)
75. G. A. Rummery and M. Niranjan. Online Q-learning using connectionist systems (tech. rep. no. cued/f-infeng/tr 166). Cambridge University Engineering Department, 1994.
76. S. Sanner. International Probabilistic Planning Competition (IPPC) at International Joint Conference on Artificial Intelligence (IJCAI). http://users.cecs. anu.edu.au/~ssanner/IPPC-2011/, 2011. Accessed: 26/09/2012.
77. B. Scherrer. Should one compute the temporal difference fix point or minimize the bellman residual the unified oblique projection view. In International Conference on Machine Learning (ICML), 2010.
78. B. Schölkopf and A. Smola. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computations, 10(5):1299-1319, 1998. (Pubitemid 128463674)
79. B. Schölkopf and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, 2002.
80. P. Schweitzer and A. Seidman. Generalized polynomial approximation in Markovian decision processes. Journal of mathematical analysis and applications, 110:568- 582, 1985.
81. D. Silver, R. S. Sutton, and M. Müller. Sample-based learning and search with permanent and transient memories. In International Conference on Machine Learning (ICML), pages 968-975, New York, NY, USA, 2008.
82. ACM. D. Silver, R. S. Sutton, and M. Müller. Temporal-difference search in computer go. Machine Learning, 87(2):183-219, 2012.
83. S. P. Singh. Reinforcement learning with a hierarchy of abstract models. In Proceeding of the Tenth National Conference on Artificial Intelligence, pages 202-207. MIT/AAAI Press, 1992.
84. S. P. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvári. Convergence results for single-step on-policy reinforcement-learning algorithms. Journal of Machine Learning Research (JMLR), 38:287-308, 2000. (Pubitemid 30572449)
85. P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning for RoboCupsoccer keepaway. International Society for Adaptive Behavior, 13(3):165-188, 2005a.
86. P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning for RoboCup soccer keepaway. Adaptive Behavior, 13(3):165-188, September 2005b. (Pubitemid 41546119)
87. A. L. Strehl, L. Li, and M. L. Littman. Reinforcement learning in finite mdps: Pac analysis. Journal of Machine Learning Research (JMLR), 10:2413-2444, Dec. 2009.
88. R. S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Neural Information Processing Systems (NIPS), pages 1038-1044.The MIT Press, 1996.
89. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
90. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems (NIPS), 12(22):1057-1063, 2000.
91. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E.Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In International Conference on Machine Learning (ICML), pages 993-1000, New York, NY, USA, 2009.
92. ACM. C. Szepesvári. Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2010.
93. I. Szita and C. Szepesvári. Model-based reinforcement learning with nearly tight exploration complexity bounds. In International Conference on Machine Learning (ICML), pages 1031-1038, 2010.
94. G. Taylor and R. Parr. Kernelized value function approximation for reinforcement learning. In International Conference on Machine Learning (ICML), pages 1017- 1024, New York, NY, USA, 2009.
95. ACM. J. N. Tsitsiklis and B. V. Roy. An analysis of temporal difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674- 690, May 1997. (Pubitemid 127760263)
96. J. N. Tsitsiklis and B. V. Roy. Average cost temporal-difference learning. Automatica, 35(11):1799 - 1808, 1999. (Pubitemid 32078092)
97. N. K. Ure, A. Geramifard, G. Chowdhary, and J. P. How. Adaptive Planning for Markov Decision Processes with Uncertain Transition Models via Incremental Feature Dependency Discovery. In European Conference on Machine Learning (ECML), 2012.
98. C. J. Watkins. Models of Delayed Reinforcement Learning. PhD thesis, Cambridge Univ., 1989.
99. C. J. Watkins. Q-learning. Machine Learning, 8(3):279-292, 1992.
100. C. J. C. H. Watkins and P. Dayan. Q-learning. Machine Learning, 8(3):279-292, May 1992.
101. S. D. Whitehead. A complexity analysis of cooperative mechanisms in reinforcement learning. In Association for the Advancement of Artificial Intelligence (AAAI), pages 607-613, 1991.
102. S. Whiteson and M. Littman. Introduction to the special issue on empirical evaluations in reinforcement learning. Machine Learning, pages 1-6, 2011.
103. B. Widrow and F. Smith. Pattern-recognizing control systems. In Computer and Information Sciences: Collected Papers on Learning, Adaptation and Control in Information Systems, COINS symposium proceedings, volume 12, pages 288-317, Washington DC, 1964.
104. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, pages 229-256, 1992.
105. T. Winograd. Procedures as a representation for data in a computer program for understanding natural language. Technical Report 235, Massachusetts Institute of Technology, 1971.
106. Y. Ye. The simplex and policy-iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate. Math. Oper. Res., 36(4): 593-603, 2011.
107. H. Yu and D. P. Bertsekas. Error bounds for approximations from projected linear equations. Math. Oper. Res., 35(2):306-329, 2010.