Kalman temporal differences

Journal of Artificial Intelligence Research

Volumn 39, Issue , 2010, Pages 483-532

  Geist, Matthieu ^a   Pietquin, Olivier ^a  

^a UMI Georgia Tech CNRS 2958   (France)

Author keywords

[No Author keywords available]

Indexed keywords

MARKOV PROCESSES; REINFORCEMENT LEARNING;

APPROXIMATION SCHEME; FUNCTION APPROXIMATION; MARKOV DECISION PROCESSES; NON-STATIONARITIES; NONLINEAR APPROXIMATION; STOCHASTIC TRANSITIONS; TEMPORAL DIFFERENCES; UNCERTAINTY MANAGEMENT;

STOCHASTIC SYSTEMS;

EID: 78651465938     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3077     Document Type: Article

References (45)

1. Antos, A., Szepesv́ari, C., & Munos, R. (2008). Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71(1), 89-129.
2. Baird, L. C. (1995). Residual Algorithms: Reinforcement Learning with Function Approximation. In Proceedings of the International Conference on Machine Learning (ICML 95), pp. 30-37.
3. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific.
4. Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2008). Incremental Natural Actor-Critic Algorithms. In Proceedings of the Twenty-First Annual Conference on Advances in Neural Information Processing Systems (NIPS), Vancouver, Canada.
5. Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford University Press, New York, USA.
6. Boyan, J. A. (1999). Technical Update: Least-Squares Temporal Difference Learning. Machine Learning, 49(2-3), 233-246.
7. Bradtke, S. J., & Barto, A. G. (1996). Linear Least-Squares Algorithms for Temporal Difference Learning. Machine Learning, 22(1-3), 33-57.
8. Chen, Z. (2003). Bayesian Filtering : From Kalman Filters to Particle Filters, and Beyond. Tech. rep., Adaptive Systems Lab, McMaster University.
9. Choi, D., & Van Roy, B. (2006). A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning. Discrete Event Dynamic Systems, 16, 207-239.
10. Dearden, R., Friedman, N., & Russell, S. J. (1998). Bayesian q-learning. In AAAI/IAAI, pp. 761-768.
11. Engel, Y. (2005). Algorithms and Representations for Reinforcement Learning. Ph.D. thesis, Hebrew University.
12. Engel, Y., Mannor, S., & Meir, R. (2003). Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning. In Proceedings of the International Conference on Machine Learning (ICML 2003), pp. 154-161.
13. Engel, Y., Mannor, S., & Meir, R. (2005). Reinforcement Learning with Gaussian Processes. In Proceedings of International Conference on Machine Learning (ICML-05).
14. Geist, M., & Pietquin, O. (2010a). Eligibility Traces through Colored Noises. In Proceedings of the IEEE International Conference on Ultra Modern Control systems (ICUMT 2010), Moscow (Russia). IEEE.
15. Geist, M., & Pietquin, O. (2010b). Managing Uncertainty within Value Function Approximation in Reinforcement Learning. In Active Learning and Experimental Design workshop (collocated with AISTATS 2010), Sardinia, Italy.
16. Geist, M., & Pietquin, O. (2010c). Revisiting natural actor-critics with value function approximation. In Torra, V., Narukawa, Y., & Daumas, M. (Eds.), Proceedings of 7th International Conference on Modeling Decisions for Artificial Intelligence (MDAI 2010), Vol. 6408 of Lecture Notes in Artificial Intelligence (LNAI), pp. 207-218, Per-pinya (France). Springer Verlag-Heidelberg Berlin.
17. Geist, M., & Pietquin, O. (2010d). Statistically Linearized Least-Squares Temporal Differences. In Proceedings of the IEEE International Conference on Ultra Modern Control systems (ICUMT 2010), Moscow (Russia). IEEE.
18. Geist, M., & Pietquin, O. (2010e). Statistically Linearized Recursive Least Squares. In Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2010), Kittil̈a (Finland).
19. Geist, M., Pietquin, O., & Fricout, G. (2008). Bayesian Reward Filtering. In et al., S. G. (Ed.), Proceedings of the European Workshop on Reinforcement Learning (EWRL 2008), Vol. 5323 of Lecture Notes in Artificial Intelligence, pp. 96-109. Springer Verlag, Lille (France).
20. Geist, M., Pietquin, O., & Fricout, G. (2009a). Kalman Temporal Differences: the deterministic case. In Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2009), Nashville, TN, USA.
21. Geist, M., Pietquin, O., & Fricout, G. (2009b). Tracking in Reinforcement Learning. In Proceedings of the 16th International Conference on Neural Information Processing (ICONIP 2009), Bangkok (Thailande). Springer.
22. Geramifard, A., Bowling, M., & Sutton, R. S. (2006). Incremental Least-Squares Temporal Difference Learning. In Proceedings of the 21st Conference, American Association for Artificial Intelligence, pp. 356-361.
23. Gill, P. E., Golub, G. H., Murray, W., & Saunders, M. A. (1974). Methods for Modifying Matrix Factorization. Mathematics of Computation, 28(126), 505-535.
24. Goodwin, G. C., & Sin, K. S. (2009). Adaptive Filtering Prediction and Control. Dover Publications, Inc., New York, NY, USA.
25. Jo, S., & Kim, S. W. (2005). Consistent Normalized Least Mean Square Filtering with Noisy Data Matrix. IEEE Transactions on Signal Processing, 53(6), 2112-2123.
26. Julier, S. J., & Uhlmann, J. K. (2004). Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 92(3), 401-422.
27. Kakade, S. (2001). A natural policy gradient. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems (NIPS 2001), pp. 1531-1538, Vancouver, British Columbia, Canada.
28. Kalman, R. E. (1960). A New Approach to Linear Filtering and Prediction Problems. Transactions of the ASME-Journal of Basic Engineering, 82(Series D), 35-45.
29. Kolter, J. Z., & Ng, A. Y. (2009). Regularization and Feature Selection in Least-Squares Temporal Difference Learning. In proceedings of the 26th International Conference on Machine Learning (ICML 2009), Montreal Canada.
30. Konda, V. R., & Tsitsiklis, J. N. (2003). On actor-critic algorithms. SIAM J. Control Optim., 42(4), 1143-1166.
31. Lagoudakis, M. G., & Parr, R. (2003). Least-Squares Policy Iteration. Journal of Machine Learning Research, 4, 1107-1149.
32. Peters, J., Vijayakumar, S., & Schaal, S. (2005). Natural Actor-Critic. In et al., J. G. (Ed.), Proceedings of the European Conference on Machine Learning (ECML 2005), Lecture Notes in Artificial Intelligence. Springer Verlag.
33. Phua, C. W., & Fitch, R. (2007). Tracking Value Function Dynamics to Improve Reinforcement Learning with Piecewise Linear Function Approximation. In Proceedings of the International Conference on Machine Learning (ICML 07).
34. Precup, D., Sutton, R. S., & Singh, S. P. (2000). Eligibility Traces for Off-Policy Policy Evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML00), pp. 759-766, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
35. Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience.
36. Schneegaß, D., Udluft, S., & Martinetz, T. (2007). Improving optimality of neural rewards regression for data-efficient batch near-optimal policy identification.. In de Śa, J. M., Alexandre, L. A., Duch, W., & Mandic, D. P. (Eds.), ICANN, Vol. 4668 of Lecture Notes in Computer Science, pp. 109-118. Springer.
37. Schoknecht, R. (2002). Optimality of Reinforcement Learning Algorithms with Linear Function Approximation. In Proceedings of the Conference on Neural Information Processing Systems (NIPS 15). Sigaud, O., & Buffet, O. (Eds.). (2010). Markov Decision Processes and Artificial Intelligence. Wiley-ISTE.
38. Simon, D. (2006). Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches (1. Auflage edition). Wiley & Sons.
39. Strehl, A. L., Li, L., Wiewiora, E., Langford, J., & Littman, M. L. (2006). Pac model-free reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning (ICML 2006), pp. 881-888, Pittsburgh, PA, USA.
40. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction (3rd edition). The MIT Press.
41. Sutton, R. S., Koop, A., & Silver, D. (2007). On the role of tracking in stationary environments. In ICML '07: Proceedings of the 24th international conference on Machine learning, pp. 871-878, New York, NY, USA. ACM.
42. S̈oderstr̈om, T., & Stoica, P. (2002). Instrumental variable methods for system identification. Circuits, Systems, and Signal Processing, 21, 1-9.
43. Tsitsiklis, J. N., & Roy, B. V. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42, 674-690.
44. van der Merwe, R. (2004). Sigma-Point Kalman Filters for Probabilistic Inference in Dynamic State-Space Models. Ph.D. thesis, OGI School of Science & Engineering, Oregon Health & Science University, Portland, OR, USA.
45. Yu, H., & Bertsekas, D. P. (2007). Q-Learning Algorithms for Optimal Stopping Based on Least Squares. In Proceedings of European Control Conference, Kos, Greece.