A learning algorithm for discrete-time stochastic control

Probability in the Engineering and Informational Sciences

Volumn 14, Issue 2, 2000, Pages 243-258

  Borkar, V S ^a  

^a TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

Author keywords

[No Author keywords available]

Indexed keywords

REINFORCEMENT LEARNING; STOCHASTIC CONTROL SYSTEMS; STOCHASTIC SYSTEMS;

ACTION SPACES; ALMOST SURE CONVERGENCE; COMPACT SUBSETS; DISCRETE STATE; EUCLIDEAN SPACES; SIMULATION-BASED ALGORITHMS; STOCHASTIC CONTROL; SUITABLE CONDITIONS;

LEARNING ALGORITHMS;

EID: 0034550848     PISSN: 02699648     EISSN: None     Source Type: Journal    
DOI: 10.1017/s0269964800142081     Document Type: Article

References (23)

1. Abounady, J., Bertsekas, D., & Borkar, V.S. (1998). Stochastic approximation for nonexpansive maps: Application to Q-learning algorithms. Technical Report LIDS-P-2433, Laboratory for Information and Decision Systems, MIT, Cambridge, MA.
2. Abounady, J., Bertsekas, D., & Borkar, V.S. (1998). Learning algorithms for Markov decision processes with average cost. Technical Report LIDS-P-2434, Laboratory for Information and Decision Systems, MIT, Cambridge, MA.
3. Adams, R.A. (1975). Sobolev spaces. New York: Academic Press.
4. Baker, W.L. (1997). Learning via stochastic approximation in function space. [Unpublished] Ph.D. thesis, Harvard University, Cambridge, MA.
5. Benveniste, A., Metivier, M., & Priouret, P. (1990). Adaptive algorithms and stochastic approximation. New York: Springer-Verlag.
6. Bertsekas, D. & Tsitsiklis, J. (1996). Neurodynamic programming. Belmont, MA: Athena Scientific.
7. Bhatnagar, S. & Borkar, V.S. (1997). Multiscale stochastic approximation for parametric optimization of hidden Markov models. Probability in the Engineering and Informational Sciences 11: 509-522.
8. Borkar, V.S. (1996). Distributed computation of fixed points of ∞-nonexpansive maps. Proceedings of the Indian Academy of Sciences (Mathematical Sciences) 106(3): 289-300.
9. Borkar, V.S. (1997). Stochastic approximation with two time scales. Systems & Control Letters 29: 291-294.
10. Borkar, V.S. & Meyn, S.P. (1998). The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM Journal of Control and Optimization (to appear).
11. Borkar, V.S. & Soumyanath, K. (1997). An analog scheme for fixed point computation, Part 1: Theory. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications CS-44(4): 351-355.
12. Boussios, C.I. (1998). An approach for nonlinear control via approximate dynamic programming. [Unpublished] Ph.D. thesis, MIT, Cambridge, MA.
13. Hirsch, M.W. (1987). Convergent activation dynamics in continuous time networks. Neural Networks 2: 331-349.
14. Jaakola, T., Jordan, M., & Singh, S. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation 6: 1185-1201.
15. Konda, V.R. & Borkar, V.S. (1998). Actor-critic type learning algorithms for Markov decision processes. SIAM Journal of Control and Optimization (to appear).
16. Ljung, L. (1977). Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control AC-22: 551-575.
17. Neveu, J. (1975). Discrete parameter martingales. Amsterdam: North-Holland.
18. Puterman, M.I. (1994). Markov decision processes. New York: Wiley.
19. Sutton, R.S. & Barto, A. (1998). Reinforcement learning. Cambridge, MA: MIT Press.
20. Sutton, R.S., Precup, D., & Singh, S.P. (1998). Between MDPs and semi-MDPs: Learning, planning and representing knowledge as multiple temporal scales. Journal of A.I. Research 1: 1-39.
21. Tsitsiklis, J.N. (1994). Asynchronous stochastic approximation and Q-learning. Machine Learning 16: 185-202.
22. Watkins, C. (1989). Learning from delayed rewards. [Unpublished] Ph.D. thesis, Cambridge University, Cambridge, UK.
23. Watkins, C. & Dayan, P. (1992). Q-learning. Machine Learning 8: 279-292.