Algorithmic survey of parametric value function approximation

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 6, 2013, Pages 845-867

  Geist, Matthieu ^a   Pietquin, Olivier ^a,b  

^a UMI Georgia Tech CNRS 2958   (France)

^b CNRS   (France)

Author keywords

Reinforcement learning (RL); survey; value function approximation

Indexed keywords

EXACT REPRESENTATIONS; MINIMIZATION METHODS; OPTIMAL CONTROL POLICY; OPTIMAL CONTROL PROBLEM; RECURSIVE LEAST SQUARES; STATE-OF-THE-ART METHODS; STOCHASTIC GRADIENT DESCENT; VALUE FUNCTION APPROXIMATION;

ALGORITHMS; OPTIMAL CONTROL SYSTEMS; REINFORCEMENT LEARNING;

SURVEYS;

EID: 84876108223     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2247418     Document Type: Article

References (98)

1. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming (Optimization and Neural Computation). Belmont, MA, USA: Athena Scientific, 1996.
2. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 3rd ed. Cambridge, MA, USA: MIT Press, Mar. 1998.
3. O. Sigaud and O. Buffet, Markov Decision Processes and Artificial Intelligence. New York, USA: Wiley, 2010.
4. C. Szepesvári, Algorithms for Reinforcement Learning. San Mateo, CA, USA: Morgan & Claypool, 2010.
5. M. Wiering and M. van Otterlo, Reinforcement Learning: State-ofthe-Art (Adaptation, Learning, and Optimization). New York, USA: Springer-Verlag, 2012.
6. L. Busoniu, R. Babuska, B. D. Schutter, and D. Ernst, Reinforcement Learning and Dynamic Programming Using Function Approximators (Automation and Control Engineering). Boca Raton, FL, USA: CRC Press, 2010.
7. W. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. New York, USA: Wiley, 2007.
8. D. Choi and B. Van Roy, "A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning," Discrete Event Dyn. Syst., vol. 16, no. 2, pp. 207-239, Apr. 2006.
9. L. C. Baird, "Residual algorithms: Reinforcement learning with function approximation," in Proc. Int. Conf. Mach. Learn., 1995, pp. 30-37.
10. Y. Engel, "Algorithms and representations for reinforcement learning," Ph.D. dissertation, Interdisciplinary Center Neural Comput., Hebrew Univ., Jerusalem, Israel, Apr. 2005.
11. M. Geist and O. Pietquin, "Kalman temporal differences," J. Artif. Intell. Res., vol. 39, no. 1, pp. 483-532, 2010.
12. S. J. Bradtke and A. G. Barto, "Linear least-squares algorithms for temporal difference learning," Mach. Learn., vol. 22, nos. 1-3, pp. 33-57, 1996. (Pubitemid 126724362)
13. M. Geist and O. Pietquin, "Statistically linearized least-squares temporal differences," in Proc. IEEE Int. Conf. Ultra Modern Control Syst., Oct. 2010, pp. 450-457.
14. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora, "Fast gradient-descent methods for temporal-difference learning with linear function approximation," in Proc. Int. Conf. Mach. Learn., 2009, pp. 993-1000.
15. H. Maei, C. Szepesvari, S. Bhatnagar, D. Precup, D. Silver, and R. S. Sutton, "Convergent temporal-difference learning with arbitrary smooth function approximation," in Advances in Neural Information Processing Systems, Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, Eds. Cambridge, MA, USA: MIT Press, 2009, pp. 1204-1212.
16. H. R. Maei and R. S. Sutton, "GQ(a;): A general gradient algorithm for temporal-differences prediction learning with eligibility traces," in Proc. Conf. Artif. General Intell., 2010, pp. 1-6.
17. H. R. Maei, C. Szepesvari, S. Bhatnagar, and R. S. Sutton, "Toward off-policy learning control with function approximation," in Proc. Int. Conf. Mach. Learn., 2010, pp. 1-8.
18. A. Nedic and D. P. Bertsekas, "Least squares policy evaluation algorithms with linear function approximation," Discrete Event Dyn. Syst., Theory Appl., vol. 13, nos. 1-2, pp. 79-110, Jan. 2003.
19. H. Yu and D. P. Bertsekas, "Q-learning algorithms for optimal stopping based on least squares," in Proc. Eur. Control Conf., Jul. 2007, pp. 1-15.
20. G. Gordon, "Stable function approximation in dynamic programming," in Proc. Int. Conf. Mach. Learn., 1995, pp. 1-8.
21. D. Ernst, P. Geurts, and L. Wehenkel, "Tree-based batch mode reinforcement learning," J. Mach. Learn. Res., vol. 6, pp. 503-556, Apr. 2005.
22. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, no. 1, pp. 9-44, 1988.
23. G. A. Rummery and M. Niranjan, "On-line Q-learning using connectionist systems," Dept. Eng., Cambridge Univ., Cambridge, U.K., Tech. Rep. CUED/F-INFENG/TR 166, 1994.
24. C. J. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, no. 3, pp. 279-292, 1992.
25. Y. Engel, S. Mannor, and R. Meir, "Bayes meets Bellman: The Gaussian process approach to temporal difference learning," in Proc. Int. Conf. Mach. Learn., 2003, pp. 154-161.
26. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Autom. Control, vol. 42, no. 5, pp. 674-690, May 1997. (Pubitemid 127760263)
27. G. Tesauro, "Temporal difference learning and TD-Gammon," Commun. ACM, vol. 38, no. 3, pp. 58-68, Mar. 1995.
28. H. van Seijen, H. van Hasselt, S. Whiteson, and M. Wiering, "A theoretical and empirical analysis of expected Sarsa," in Proc. IEEE Int. Symp. Adapt. Dynamic Program. Reinforce. Learn., Mar.-Apr. 2009, pp. 177-184.
29. H. Yu, "Least squares temporal difference methods: An analysis under general conditions," Dept. Math. Stat., Univ. Helsinki, Helsinki, Finland, Tech. Rep. C-2010-39, 2010.
30. F. S. Melo, S. P. Meyn, and M. I. Ribeiro, "An analysis of reinforcement learning with function approximation," in Proc. Int. Conf. Mach. Learn., 2009, pp. 664-671.
31. A. Antos, C. Szepesvári, and R. Munos, "Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path," Mach. Learn., vol. 71, no. 1, pp. 89-129, Apr. 2008.
32. A. Y. Kruger, "On Fréchet subdifferentials," J. Math. Sci., vol. 116, no. 3, pp. 3325-3358, 2003.
33. T. W. Anderson, An Introduction to Multivariate Statistical Analysis (Probability and Statistics). New York, USA: Wiley, 1984.
34. Y. Engel, S. Mannor, and R. Meir, "Reinforcement learning with gaussian processes," in Proc. Int. Conf. Mach. Learn., 2005, pp. 201-208. (Pubitemid 43183334)
35. D. Precup, R. S. Sutton, and S. P. Singh, "Eligibility traces for offpolicy policy evaluation," in Proc. Int. Conf. Mach. Learn., 2000, pp. 759-766.
36. S. J. Julier and J. K. Uhlmann, "A new extension of the Kalman filter to nonlinear systems," in Proc. 3rd Int. Symp. Aerosp. Defense Sens. Simul. Controls, 1997, pp. 182-193.
37. S. J. Julier, "The scaled unscented transformation," in Proc. Amer. Control Conf., vol. 6. 2002, pp. 4555-4559.
38. P. M. Nørgard, N. K. Poulsen, and O. Ravn, "New developments in state estimation for nonlinear systems," Automatica, vol. 36, no. 11, pp. 1627-1638, Nov. 2000.
39. R. van der Merwe, "Sigma-point kalman filters for probabilistic inference in dynamic state-space models," Ph.D. dissertation, OGI School Science Eng., Oregon Health & Science Univ., Portland, OR, USA, Apr. 2004.
40. M. Geist and O. Pietquin, "Eligibility traces through colored noises," in Proc. IEEE Int. Conf. Ultra Modern Control Syst., Oct. 2010, pp. 458-465.
41. M. Geist and O. Pietquin, "Managing uncertainty within the KTD framework," J. Mach. Learn. Res. (W&C Proc.), vol. 15, pp. 157-168, Mar. 2011.
42. L. Daubigney, M. Gasic, S. Chandramohan, M. Geist, O. Pietquin, and S. Young, "Uncertainty management for on-line optimisation of a POMDP-based large-scale spoken dialogue system," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Aug. 2011, pp. 1301-1304.
43. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res., vol. 4, pp. 1107-1149, Dec. 2003.
44. T. Söderström and P. Stoica, "Instrumental variable methods for system identification," Circuits Syst. Signal Process., vol. 21, no. 1, pp. 1-9, Jan.-Feb. 2002. (Pubitemid 34414642)
45. A. Geramifard, M. Bowling, and R. S. Sutton, "Incremental least-squares temporal difference learning," in Proc. 21st Nat. Conf. Artif. Intell., 2006, pp. 356-361. (Pubitemid 44705310)
46. J. Johns, M. Petrik, and S. Mahadevan, "Hybrid least-squares algorithms for approximate policy evaluation," Mach. Learn., vol. 76, nos. 2-3, pp. 243-256, Sep. 2009.
47. M. Geist and O. Pietquin, "Statistically linearized recursive least squares," in Proc. IEEE Int. Workshop Mach. Learn. Signal Process., Sep. 2010, pp. 272-276.
48. R. S. Sutton, C. Szepesvari, and H. R. Maei, "A convergent O(n) Algorithm for off-policy temporal-difference learning with linear function approximation," in Advances in Neural Information Processing Systems, Vancouver, BC, Canada: MIT Press, 2008.
49. B. D. Ripley, Stochastic Simulation, New York, USA: Wiley, 1987.
50. A. L. Samuel, "Some studies in machine learning using the game of checkers," IBM J. Res. Develop., vol. 3, no. 3, pp. 210-229, Jul. 1959.
51. R. Munos, "Performance bounds in L p-norm for approximate value iteration," SIAM J. Control Optim., vol. 46, no. 2, pp. 541-561. Jan. 2007. (Pubitemid 351456426)
52. D. Ormoneit and ́ S. Sen, "Kernel-based reinforcement learning," Mach. Learn., vol. 49, nos. 2-3, pp. 161-178, Nov. 2002. (Pubitemid 34325684)
53. M. Riedmiller, "Neural fitted q iteration-First experiences with a data efficient neural reinforcement learning method," in Proc. Eur. Conf. Mach. Learn., 2005, pp. 317-328.
54. D. P. Bertsekas and S. Ioffe, "Temporal differences-based policy iteration and applications in neuro-dynamic programming," Labs Information & Decision Systems, MIT, Cambridge, MA, USA, Tech. Rep. LIDS-P-2349, 1996.
55. D. P. Bertsekas, V. Borkar, and A. Nedic, "Improved temporal difference methods with linear function approximation," in Learning and Approximate Dynamic Programming. Piscataway, NJ, USA: IEEE Press, 2004, pp. 231-235.
56. D. P. Bertsekas and H. Yu, "Projected equation methods for approximate solution of large linear systems," J. Comput. Appl. Math., vol. 227, no. 1, pp. 27-50, May 2009.
57. D. P. Bertsekas, "Projected equations, variational inequalities, and temporal difference methods," in Proc. IEEE Int. Symp. Adapt. Dynamic Program. Reinforce. Learn., 2009, pp. 1-28.
58. D. P. de Farias and B. V. Roy, "The linear programming approach to approximate dynamic programming," Oper. Res., vol. 51, no. 6, pp. 850-865, 2003.
59. V. V. Desai, V. F. Farias, and C. C. Moallemi, "The smoothed approximate linear program," in Advances in Neural Information Processing Systems, New York, USA: Columbia Univ. Press, Aug. 2009.
60. B. Scherrer, V. Gabillon, M. Ghavamzadeh, and M. Geist, "Approximate modified policy iteration," in Proc. Int. Conf. Mach. Learn., 2012, pp. 1-21.
61. S. Kakade and J. Langford, "Approximately optimal approximate reinforcement learning," in Proc. 19th Int. Conf. Mach. Learn., 2002, pp. 267-274.
62. A. Barto, R. Sutton, and C. Anderson, "Neuronlike adaptive elements that can solve difficult learning control problems," IEEE Trans. Syst., Man, Cybern., vol. 13, no. 5, pp. 834-846, Sep.-Oct. 1983.
63. V. R. Konda and J. N. Tsitsiklis, "On actor-critic algorithms," SIAM J. Control Optim., vol. 42, no. 4, pp. 1143-1166, 2003.
64. R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 1999, pp. 1057-1063.
65. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, nos. 7-9, pp. 1180-1190, 2008.
66. S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee, "Incremental natural actor-critic algorithms," in Advances in Neural Information Processing Systems, Vancouver, Canada: MIT Press, 2007.
67. M. Geist and O. Pietquin, "Revisiting natural actor-critics with value function approximation," in Proc. Int. Conf. Model. Decisions Artif. Intell., 2010, pp. 207-218.
68. B. Scherrer and M. Geist, "Recursive least-squares learning with eligibility traces," in Proc. Eur. Workshop Reinforce. Learn., 2011, pp. 11-12.
69. R. Munos, "Error bounds for approximate policy iteration," in Proc. Int. Conf. Mach. Learn., 2003, pp. 560-567.
70. B. Scherrer, "Should one compute the temporal difference fix point or minimize the Bellman residual? The unified oblique projection view," in Proc. Int. Conf. Mach. Learn., 2010, pp. 1-8.
71. A. Lazaric, M. Ghavamzadeh, and R. Munos, "Finite-sample analysis of LSTD," in Proc. Int. Conf. Mach. Learn., 2010, pp. 1-14.
72. O. Pietquin, M. Geist, and S. Chandramohan, "Sample efficient on-line learning of optimal dialogue policies with Kalman temporal differences," in Proc. Int. Joint Conf. Artif. Intell., Jul. 2011, pp. 1878-1883.
73. J. A. Boyan, "Technical update: Least-squares temporal difference learning," Mach. Learn., vol. 49, nos. 2-3, pp. 233-246, 1999. (Pubitemid 34325688)
74. L. Daubigney, M. Geist, and O. Pietquin, "Off-policy learning in large-scale POMDP-based dialogue systems," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Mar. 2012, pp. 4989-4992.
75. C. W. Phua and R. Fitch, "Tracking value function dynamics to improve reinforcement learning with piecewise linear function approximation," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 751-758.
76. M. Keramati, A. Dezfouli, and P. Piray, "Speed/accuracy trade-off between the habitual and the goal-directed processes," PLoS Comput. Biol., vol. 7, no. 5, p. e1002055, 2011.
77. O. Pietquin, M. Geist, S. Chandramohan, and H. Frezza-Buet, "Sample-efficient batch reinforcement learning for dialogue management optimization," ACM Trans. Speech Lang. Process., vol. 7, no. 3, pp. 7:1-7:21, 2011.
78. M. Kearns and S. Singh, "Bias-variance error bounds for temporal difference updates," in Proc. Conf. Learn. Theory, 2000, pp. 142-147.
79. B. Scherrer and M. Geist, "Recursive least-squares off-policy learning with eligibility traces," INRIA, France, Tech. Rep. hal-00644516, 2012.
80. X. Xu, D. Hu, and X. Lu, "Kernel-based least squares policy iteration for reinforcement learning," IEEE Trans. Neural Netw., vol. 18, no. 4, pp. 973-992, Jul. 2007. (Pubitemid 47098876)
81. T. Jung and D. Polani, "Kernelizing LSPE(a;)," in Proc. IEEE Symp. Approx. Dynamic Program. Reinforce. Learn., Apr. 2007, pp. 338-345.
82. G. Taylor, and R. Parr, "Kernelized value function approximation for reinforcement learning," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 1017-1024.
83. M. Loth, M. Davy, and P. Preux, "Sparse temporal difference learning using LASSO," in Proc. IEEE Int. Symp. Approx. Dynamic Program. Reinforce. Learn., Apr. 2007, pp. 352-359. (Pubitemid 47431407)
84. J. Z. Kolter and A. Y. Ng, "Regularization and feature selection in leastsquares temporal difference learning," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 521-528.
85. J. Johns, C. Painter-Wakefield, and R. Parr, "Linear complementarity for regularized policy evaluation and improvement," in Advances in Neural Information Processing Systems, J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, Eds. Durham, NC, USA: Duke Univ. Press, 2010, pp. 1009-1017.
86. M. Geist and B. Scherrer, "1-penalized projected Bellman residual," in Proc. Eur. Workshop Reinforce. Learn., 2011, pp. 1-12.
87. M. W. Hoffman, A. Lazaric, M. Ghavamzadeh, and R. Munos, "Regularized least squares temporal difference learning with nested 2 and 1 penalization," in Proc. Eur. Workshop Reinforce. Learn., 2011, pp. 1-12.
88. M. Geist, B. Scherrer, A. Lazaric, and M. Ghavamzadeh, "A Dantzig selector approach to temporal difference learning," in Proc. Int. Conf. Mach. Learn., 2012, pp. 1-8.
89. I. Menache, S. Mannor, and N. Shimkin, "Basis function adaptation in temporal difference reinforcement learning," Ann. Oper. Res., vol. 134, no. 1, pp. 215-238, 2005. (Pubitemid 40550047)
90. P. W. Keller, S. Mannor, and D. Precup, "Automatic basis function construction for approximate dynamic programming and reinforcement learning," in Proc. Int. Conf. Mach. Learn., 2006, pp. 449-456.
91. R. Parr, C. Painter-Wakefield, L. Li, and M. Littman, "Analyzing feature generation for value-function approximation," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 1-8.
92. R. Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. L. Littman, "An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning," in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 752-759.
93. J. Wu and R. Givan, "Automatic induction of bellman-error features for probabilistic planning," J. Artif. Intell. Res., vol. 38, no. 1, pp. 687-755, 2010.
94. S. Mahadevan and M. Maggioni, "Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes," J. Mach. Learn. Res., vol. 8, no. 10, pp. 2169-2231, 2007. (Pubitemid 350046199)
95. D. Bertsekas and D. Castanon, "Adaptive aggregation methods for infinite horizon dynamic programming," IEEE Trans. Autom. Control, vol. 34, no. 6, pp. 589-598, Jun. 1989.
96. S. Singh, T. Jaakkola, and M. Jordan, "Reinforcement learning with soft state aggregation," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 1995, pp. 361-368.
97. J. Ma and W. B. Powell, "Convergence analysis of kernel-based onpolicy approximate policy iteration algorithms for Markov decision processes with continuous, multidimensional states and actions," Dept. Oper. Res. Financial Eng., Princeton Univ., Princeton, NJ, USA, Tech. Rep. 08544, 2010.
98. A. Barreto, D. Precup, and J. Pineau, "Reinforcement learning using Kernel-based stochastic factorization," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2011.