The divergence of reinforcement learning algorithms with value- iteration and function approximation

Proceedings of the International Joint Conference on Neural Networks

Volumn , Issue , 2012, Pages

  Fairbank, Michael ^a   Alonso, Eduardo ^a  

^a CITY UNIVERSITY   (United Kingdom)

Author keywords

Adaptive Dynamic Programming; Divergence; Greedy Policy; Reinforcement Learning; Value Iteration

Indexed keywords

ADAPTIVE DYNAMIC PROGRAMMING; DIVERGENCE; FUNCTION APPROXIMATION; FUNCTION APPROXIMATORS; GREEDY POLICY; VALUE FUNCTIONS; VALUE ITERATION;

DYNAMIC PROGRAMMING; LEARNING ALGORITHMS; NEURAL NETWORKS; REINFORCEMENT LEARNING;

ADAPTIVE ALGORITHMS;

EID: 84865066281     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IJCNN.2012.6252792     Document Type: Conference Paper

References (21)

1. F.-Y. Wang, H. Zhang, and D. Liu, "Adaptive dynamic programming: An introduction," IEEE Computational Intelligence Magazine, pp. 39-47, 2009.
2. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, Massachussetts, USA: The MIT Press, 1998.
3. R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: Princeton University Press, 1957.
4. G. Rummery and M. Niranjan, "On-line q-learning using connectionist systems," Tech. Rep. Technical Report CUED/F-INFENG/TR 166, Cambridge University Engineering Department, 1994.
5. R. S. Sutton, "Learning to predict by the methods of temporal differences," Machine Learning, vol. 3, pp. 9-44, 1988.
6. P. J. Werbos, "Approximating dynamic programming for real-time control and neural modeling." Handbook of Intelligent Control, editors White and Sofge, Chapter 13, pp. 493-525, 1992.
7. D. Prokhorov and D. Wunsch, "Adaptive critic designs," IEEE Transactions on Neural Networks, vol. September, pp. 997-1007, 1997.
8. S. Ferrari and R. F. Stengel, "Model-based adaptive critic designs," Handbook of learning and approximate dynamic programming, editors Jennie Si et al., pp. 65-96, 2004.
9. M. Fairbank and E. Alonso, "The local optimality of reinforcement learning by value gradients and its relationship to policy gradient learning," eprint arXiv:1101.0428, 2011.
10. - "Value-gradient learning," in Proceedings of the IEEE International Joint Conference on Neural Networks 2012 (IJCNN'12). IEEE Press, 2012.
11. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Transactions on Automatic Control, Tech. Rep., 1996.
12. R. S. Sutton, H. R. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora, "Fast gradient-descent methods for temporal-difference learning with linear function approximation," in Proceedings of the 26th Annual International Conference on Machine Learning, ser. ICML '09. New York, NY, USA: ACM, 2009, pp. 993-1000.
13. H. Maei, C. Szepesvari, S. Bhatnager, D. Precup, D. Silver, and R. Sutton, "Convergent temporal-difference learning with arbitrary smooth function approximation," in Advances in Neural Information Processing Systems (NIPS'09). MIT Press, 2009.
14. P. J. Werbos, "Stable adaptive control using new critic designs," eprint arXiv:adap-org/9810001, 1998.
15. M. Fairbank, "Reinforcement learning by value gradients," eprint arXiv:0803.3539, 2008.
16. C. J. C. H. Watkins, "Learning from delayed rewards," Ph.D. dissertation, Cambridge University, 1989.
17. M. Fairbank and E. Alonso, "A comparison of learning speed and ability to cope without exploration between DHP and TD(0)," in Proceedings of the IEEE International Joint Conference on Neural Networks 2012 (IJCNN'12). IEEE Press, 2012.
18. - "The divergence of reinforcement learning algorithms with valueiteration and function approximation," eprint arXiv:1107.4606, 2011.
19. L. C. Baird, "Residual algorithms: Reinforcement learning with function approximation," in International Conference on Machine Learning, 1995, pp. 30-37.
20. J. N. Tsitsiklis and B. Van Roy, "Feature-based methods for large scale dynamic programming," Machine Learning, vol. 22, no. 1-3, pp. 59-94, 1996.
21. R. S. Sutton, D. Mcallester, S. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," in Advances in Neural Information Processing Systems 12, vol. 12, 2000, pp. 1057-1063.