A unified analysis of value-function-based reinforcement-learning algorithms

Neural Computation

Volumn 11, Issue 8, 1999, Pages 2017-2060

  Szepesvári, Csaba ^a   Littman, Michael L ^b  

^a Mindmaker Ltd   (Hungary)

^b Duke University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; LEARNING; REINFORCEMENT;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; LEARNING; REINFORCEMENT (PSYCHOLOGY);

EID: 0033570798     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976699300016070     Document Type: Article

References (46)

1. Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence, 72(1), 81-138.
2. Barto, A. G., Sutton, R. S., & Watkins, C. J. C. H. (1989). Learning and sequential decision making (Tech. Rep. No. 89-95). Amherst, MA: Department of Computer and Information Science, University of Massachusetts.
3. Benveniste, A., Métivier, M., & Priouret, P. (1990). Adaptive algorithms and stochastic approximations. New York: Springer-Verlag.
4. Bertsekas, D. P., & Castañon, D. A. (1989). Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Transactions on Automatic Control, 34(6), 589-598.
5. Bertsekas, D. P., & Tsitsiklis, J. N. (1989). Parallel and distributed computation: Numerical methods. Englewood Cliffs, NJ: Prentice-Hall.
6. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Belmont, MA: Athena Scientific.
7. Gordon, G. J. (1995). Stable function approximation in dynamic programming. In A. Prieditis & S. Russell (Eds.), Proceedings of the Twelfth International Conference on Machine Learning (pp. 261-268). San Mateo: Morgan Kaufmann.
8. Grossberg, S. (1969). Embedding fields: A theory of learning with physiological implications. Journal of Mathematical Psychology, 6, 209-239.
9. Gullapalli, V., & Barto, A. G. (1994). Convergence of indirect adaptive asynchronous value iteration algorithms. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems, 6 (pp. 695-702). San Mateo, CA: Morgan Kaufmann.
10. Hecht-Nielsen, R. (1991). Neurocomputing. Reading, MA: Addison-Wesley.
11. Heger, M. (1994). Consideration of risk in reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning (pp. 105-111). San Mateo, CA: Morgan Kaufmann.
12. Hu, J., & Wellman, M. P. (1998). Multiagent reinforcement learning: Theoretical framework and an algorithm. In J. Shavlik (Ed.), Proceedings of the Fifteenth International Conference on Machine Learning. San Mateo, CA: Morgan Kaufmann.
13. Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6), 1185-1201.
14. John, G. H. (1994). When the best move isn't optimal: Q-learning with exploration. In Proceedings of the Twelfth National Conference on Artificial Intelligence (p. 1464).
15. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237-285.
16. Konda, V., & Borkar, V. (1997). Actor-critic type learning algorithms for Markov decision processes. Unpublished manuscript. Available at: http://donald-duck.mit.edu/-konda/siam.ps.gz.
17. Korf, R. E. (1990). Real-time heuristic search. Artificial Intelligence, 42, 189-211.
18. Kushner, H., & Clark, D. (1978). Stochastic approximation methods for constrained and unconstrained systems. Berlin: Springer-Verlag.
19. Kushner, H., & Yin, G. (1997). Stochastic approximation algorithms and applications. New York: Springer-Verlag.
20. Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning (pp. 157-163). San Mateo, CA: Morgan Kaufmann.
21. Littman, M. L. (1996). Algorithms for sequential decision making. Unpublished Ph.D. dissertation, Brown University.
22. Littman, M. L., & Szepesvári, C. (1996). A generalized reinforcement-learning model: Convergence and applications. In L. Saitta (Ed.), Proceedings of the Thirteenth International Conference on Machine Learning (pp. 310-318).
23. Ljung, L. (1977). Analysis of recursive stochastic algorithms. IEEE Trans. Automat. Control, 22, 551-575.
24. Mahadevan, S. (1996). Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine Learning, 22(1/2/3), 159-196.
25. Moore, A. W., & Atkeson, C. G. (1993). Prioritized sweeping: Reinforcement learning with less data and less real time. Machine Learning, 13, 103-130.
26. Owen, G. (1982). Game theory (2nd ed.). Orlando, FL: Academic Press.
27. Puterman, M. L. (1994). Markov decision processes - Discrete stochastic dynamic programming. New York: Wiley.
28. Ribeiro, C. (1995). Attentional mechanisms as a strategy for generalisation in the Q-learning algorithm. In Proceedings of ICANN'95 (Vol. 1, pp. 455-460).
29. Ribeiro, C., & Szepesvári, C. (1996). Q-learning combined with spreading: Convergence and results. In Proceedings of ISRF-IEE International Conference: Intelligent and Cognitive Systems, Neural Networks Symposium (pp. 32-36).
30. Robbins, H., & Monro, S. (1951). A stochastic approximation method. Annals of Mathematical Statistics, 22, 400-407.
31. Robbins, H., & Siegmund, D. (1971). A convergence theorem for non-negative almost supermartingales and some applications. In J. Rustagi (Ed.), Optimizing methods in statistics (pp. 235-257). New York: Academic Press.
32. Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems (Tech. Rep. No. CUED/F-INFENG/TR 166). Cambridge: Cambridge University, Engineering Department.
33. Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards. In Proceedings of the Tenth International Conference on Machine Learning (pp. 298-305). San Mateo, CA: Morgan Kaufmann.
34. Schweitzer, P. J. (1984). Aggregation methods for large Markov chains. In G. Iazola, P. J. Coutois, & A. Hordijk (Eds.), Mathematical computer performance and reliability (pp. 275-302). Amsterdam: Elsevier.
35. Singh, S., Jaakkola, T., & Jordan, M. (1995). Reinforcement learning with soft state aggregation. In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems, 7 (pp. 361-368). Cambridge, MA: MIT Press.
36. Singh, S., Jaakkola, T., Littman, M. L., & Szepesvári, C. (in press). Convergence results for single-step on-policy reinforcement-learning algorithms.
37. Singh, S. P., & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22(1/2/3), 123-158.
38. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
39. Szepesvári, M. (1998a). The asymptotic convergence rate of Q-learning. In M. I. Jordan, M. J. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems, 10. Cambridge, MA: MIT Press.
40. Szepesvári, C. (1998b). Static and dynamic aspects of optimal sequential decision making. Unpublished Ph.D. dissertation, Bolyai Institute of Mathematics, "József Attila" University, Szeged, Hungary.
41. Szepesvári, C., & Littman, M. L. (1996). Generalized Markov decision processes: Dynamic-programming and reinforcement-learning algorithms (Tech. Rep. No. CS-96-11). Providence, RI: Brown University.
42. Tsitsiklis, J. N. (1994). Asynchronous stochastic approximation and Q-learning. Machine Learning, 16(3), 185-202.
43. Vrieze, O. J., & Tijs, S. H. (1982). Fictitious play applied to sequences of games and discounted stochastic games. International Journal of Game Theory, 11(2), 71-85.
44. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Unpublished Ph.D. dissertation, King's College, Cambridge.
45. Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3), 279-292.
46. Williams, R. J., & Baird, III, L. C. (1993). Analysis of some incremental variants of policy iteration: First steps toward understanding actor-critic learning systems (Tech. Rep. No. NU-CCS-93-11). Boston: Northeastern University, College of Computer Science.