Parametric value function approximation: A unified view

IEEE SSCI 2011: Symposium Series on Computational Intelligence - ADPRL 2011: 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning

Volumn , Issue , 2011, Pages 9-16

  Geist, Matthieu ^a   Pietquin, Olivier ^b  

^a UMI Georgia Tech CNRS 2958   (France)

^b CNRS   (France)

Author keywords

Reinforcement learning; survey; value function approximation

Indexed keywords

ARTIFICIAL INTELLIGENCE; COST FUNCTIONS; DYNAMIC PROGRAMMING; OPTIMAL CONTROL SYSTEMS; QUALITY CONTROL; STOCHASTIC SYSTEMS; SURVEYING; SURVEYS;

EXACT REPRESENTATIONS; OPTIMAL CONTROL POLICY; OPTIMAL CONTROL PROBLEM; RECURSIVE LEAST SQUARE (RLS); RELATED ALGORITHMS; STATE-OF-THE-ART METHODS; STOCHASTIC GRADIENT DESCENT; VALUE FUNCTION APPROXIMATION;

REINFORCEMENT LEARNING;

EID: 80052232599     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ADPRL.2011.5967355     Document Type: Conference Paper

References (54)

1. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming (Optimization and Neural Computation Series, 3). Athena Scientific, 1996.
2. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 3rd ed. The MIT Press, March 1998.
3. O. Sigaud and O. Buffet, Eds., Markov Decision Processes and Artificial Intelligence. Wiley - ISTE, 2010.
4. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, 1994.
5. R. Bellman, Dynamic Programming, 6th ed. Dover Publications, 1957.
6. M. Geist and O. Pietquin, "A Brief Survey of Parametric Value Function Approximation," Supélec, Tech. Rep., September 2010.
7. D. Choi and B. Van Roy, "A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning," Discrete Event Dynamic Systems, vol. 16, pp. 207-239, 2006.
8. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Transactions on Automatic Control, vol. 42, 1997
9. G. Tesauro, "Temporal Difference Learning and TD-Gammon," Communications of the ACM, vol. 38, no. 3, March 1995.
10. F. S. Melo, S. P. Meyn, and M. I. Ribeiro, "An analysis of reinforcement learning with function approximation," in Proceedings of the 25th International Conference on Machine Learning, 2009, pp. 664-671.
11. A. Antos, C. Szepesvári, and R. Munos, "Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path," Machine Learning, vol. 71, no. 1, pp. 89-129, 2008.
12. L. C. Baird, "Residual Algorithms: Reinforcement Learning with Function Approximation," in Proc. of the International Conference on Machine Learning (ICML 95), 1995, pp. 30-37.
13. Y. Engel, S. Mannor, and R. Meir, "Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning," in Proceedings of the International Conference on Machine Learning (ICML 03), 2003, pp. 154-161.
14. C. E. Rassmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. The MIT Press, 2006.
15. Y. Engel, S. Mannor, and R. Meir, "The Kernel Recursive Least Squares Algorithm," IEEE Transactions on Signal Processing, vol. 52, pp. 2275-2285, 2004. [Online]. Available: http://www.cs.ualberta.ca/yaki/
16. Y. Engel, "Algorithms and Representations for Reinforcement Learning," Ph.D. dissertation, Hebrew University, April 2005.
17. Z. Chen, "Bayesian Filtering : From Kalman Filters to Particle Filters, and Beyond," Adaptive Systems Lab, McMaster University, Tech. Rep., 2003.
18. Y. Engel, S. Mannor, and R. Meir, "Reinforcement Learning with Gaussian Processes," in Proceedings of the International Conference on Machine Learning (ICML 05), 2005.
19. D. Precup, R. S. Sutton, and S. P. Singh, "Eligibility Traces for Off- Policy Policy Evaluation," in Proceedings of the Seventeenth International Conference on Machine Learning (ICML 00). San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2000, pp. 759-766.
20. M. Geist, O. Pietquin, and G. Fricout, "Kalman Temporal Differences: the deterministic case," in Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2009), Nashville, TN, USA, April 2009.
21. M. Geist and O. Pietquin, "Kalman Temporal Differences," Journal of Artificial Intelligence Research (JAIR), 2010.
22. -, "Eligibility Traces through Colored Noises," in Proceedings of the IEEE International Conference on Ultra Modern Control Systems (ICUMT 2010). Moscow (Russia): IEEE, October 2010.
23. C. W. Phua and R. Fitch, "Tracking value function dynamics to improve reinforcement learning with piecewise linear function approximation," in Proceedings of the International Conference on Machine Learning (ICML 07), 2007.
24. M. Geist, O. Pietquin, and G. Fricout, "Tracking in Reinforcement Learning," in Proceedings of the 16th International Conference on Neural Information Processing (ICONIP 2009). Bangkok (Thailande): Springer, 2009.
25. R. E. Kalman, "A new approach to linear filtering and prediction problems," Transactions of the ASME-Journal of Basic Engineering, vol. 82, no. Series D, pp. 35-45, 1960.
26. M. Geist and O. Pietquin, "Statistically Linearized Recursive Least Squares," in Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2010). Kittilá (Finland): IEEE, 2010.
27. -, "Managing Uncertainty within Value Function Approximation in Reinforcement Learning," in Active Learning and Experimental Design Workshop (collocated with AISTATS 2010), ser. Journal of Machine Learning Research - Workshop and Conference Proceedings, Sardinia, Italy, 2010.
28. S. J. Bradtke and A. G. Barto, "Linear Least-Squares algorithms for temporal difference learning," Machine Learning, vol. 22, no. 1-3, pp. 33-57, 1996. (Pubitemid 126724362)
29. T. Söderström and P. Stoica, "Instrumental variable methods for system identification," Circuits, Systems, and Signal Processing, vol. 21, pp. 1-9, 2002. (Pubitemid 34414642)
30. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," Journal of Machine Learning Research, vol. 4, pp. 1107-1149, 2003.
31. J. A. Boyan, "Technical Update: Least-Squares Temporal Difference Learning," Machine Learning, vol. 49, no. 2-3, pp. 233-246, 1999. (Pubitemid 34325688)
32. H. Yu, "Convergence of Least Squares Temporal Difference Methods Under General Conditions," in International Conference on Machine Learning (ICML 2010), 2010, pp. 1207-1214.
33. M. Geist and O. Pietquin, "Statistically Linearized Least-Squares Temporal Differences," in Proceedings of the IEEE International Conference on Ultra Modern Control Systems (ICUMT 2010). Moscow (Russia): IEEE, October 2010.
34. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora, "Fast gradient-descent methods for temporal-difference learning with linear function approximation," in ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning. New York, NY, USA: ACM, 2009, pp. 993- 1000.
35. S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee, "Incremental natural actor-critic algorithms," in Conference on Neural Information Processing Systems (NIPS), Vancouver, Canada, 2007.
36. H. Maei, C. Szepesvari, S. Bhatnagar, D. Precup, D. Silver, and R. S. Sutton, "Convergent temporal-difference learning with arbitrary smooth function approximation," in Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, Eds., 2009, pp. 1204-1212.
37. H. R. Maei and R. S. Sutton, "GQ(λ): a general gradient algorithm for temporal-differences prediction learning with eligibility traces," in Third Conference on Artificial General Intelligence, 2010.
38. H. R. Maei, C. Szepesvari, S. Bhatnagar, and R. S. Sutton, "Toward Off-Policy Learning Control with Function Approximation," in 27th conference on Machine Learning (ICML 2010), 2010.
39. A. Samuel, "Some studies in machine learning using the game of checkers," IBM Journal on Research and Development, pp. 210-229, 1959.
40. G. Gordon, "Stable Function Approximation in Dynamic Programming," in Proceedings of the International Conference on Machine Learning (IMCL 95), 1995.
41. R. Munos, "Performance Bounds in Lp norm for Approximate Value Iteration," SIAM Journal on Control and Optimization, 2007.
42. D. Ormoneit and S. Sen, "Kernel-Based Reinforcement Learning," Machine Learning, vol. 49, pp. 161-178, 2002. (Pubitemid 34325684)
43. M. Riedmiller, "Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method ," in Europeac Conference on Machine Learning (ECML), 2005.
44. D. Ernst, P. Geurts, and L. Wehenkel, "Tree-Based Batch Mode Reinforcement Learning," Journal of Machine Learning Research, vol. 6, pp. 503-556, 2005.
45. A. Nedić and D. P. Bertsekas, "Least Squares Policy Evaluation Algorithms with Linear Function Approximation," Discrete Event Dynamic Systems: Theory and Applications, vol. 13, pp. 79-110, 2003.
46. D. P. Bertsekas, V. Borkar, and A. Nedic, Learning and Approximate Dynamic Programming. IEEE Press, 2004, ch. Improved Temporal Difference Methods with Linear Function Approximation, pp. 231-235.
47. D. P. Bertsekas and H. Yu, "Projected Equation Methods for Approximate Solution of Large Linear Systems," Journal of Computational and Applied Mathematics, vol. 227, pp. 27-50, 2007.
48. D. P. Bertsekas, "Projected Equations, Variational Inequalities, and Temporal Difference Methods," in IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning, 2009.
49. H. Yu and D. P. Bertsekas, "Q-Learning Algorithms for Optimal Stopping Based on Least Squares," in Proceedings of European Control Conference, Kos, Greece, 2007.
50. X. Xu, D. Hu, and X. Lu, "Kernel-Based Least Squares Policy Iteration for Reinforcement Learning," IEEE Transactions on Neural Networks, vol. 18, no. 4, pp. 973-992, July 2007. (Pubitemid 47098876)
51. T. Jung and D. Polani, "Kernelizing LSPE(λ)," in IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning, 2007, pp. 338-345.
52. A. Geramifard, M. Bowling, and R. S. Sutton, "Incremental Least- Squares Temporal Difference Learning," in 21st Conference of American Association for Artificial Intelligence (AAAI 06), 2006, pp. 356-361. (Pubitemid 44705310)
53. A. Farahmand, M. Ghavamzadeh, C. Szepesvári, and S. Mannor, "Regularized policy iteration," in 22nd Annual Conference on Neural Information Processing Systems (NIPS 21), Vancouver, Canada, 2008.
54. J. Z. Kolter and A. Y. Ng, "Regularization and Feature Selection in Least-Squares Temporal Difference Learning," in proceedings of the 26th International Conference on Machine Learning (ICML 2009), Montreal Canada, 2009.