Actor-critic-type learning algorithms for Markov decision processes

SIAM Journal on Control and Optimization

Volumn 38, Issue 1, 1999, Pages 94-123

  Konda, Vijaymohan R ^a   Borkar, Vivek S ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; ARTIFICIAL INTELLIGENCE; DECISION THEORY; LEARNING ALGORITHMS;

ACTOR-CRITIC ALGORITHMS; MARKOV DECISION PROCESSES (MDP); REINFORCEMENT LEARNING;

MARKOV PROCESSES;

EID: 0343893613     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S036301299731669X     Document Type: Article

References (28)

1. J. ABOUNADI, D. P. BERTSEKAS, AND V. S. BORKAR, Stochastic Approximation for Non-expansive Maps: Applications to Q-Learning Algorithms, preprint LIDS-P-2438, Lab for Info. and Decision Sciences, MIT, Cambridge, MA, 1988.
2. J. ABOUNADI, D. P. BERTSEKAS, AND V. S. BORKAR, Learning Algorithms for Markov Decision Processes with Average Cost, preprint LIDS-P-2434, Lab for Info. and Decision Sciences, MIT, Cambridge, MA, 1988.
3. A. BARTO, R. SUTTON, AND C. ANDERSON, Neuron-like elements that can solve difficult learning control problems, IEEE Trans. Systems, Man and Cybernetics, 13 (1983), pp. 835-846.
4. D. P. BERTSEKAS, A new value iteration method for average cost dynamic programming problem, SIAM J. Control Optim., 36 (1998), pp. 742-759.
5. D. P. BERTSEKAS AND J. N. TSITSIKLIS, An analysis of stochastic shortest path problem, Math. Oper. Res., 16 (1991), pp. 580-595.
6. D. P. BERTSEKAS AND J. N. TSITSIKLIS, Neurodynamic Programming, Athena Scientific, Belmont, MA, 1996.
7. V. S. BORKAR, Recursive self-tuning control of finite Markov chains, Applicationes Mathematicae, 24 (1996), pp. 169-188.
8. V. S. BORKAR, Stochastic approximation with two time scales, Systems Control Lett., 29 (1996), pp. 291-294.
9. V. S. BORKAR, Asynchronous stochastic approximations, SIAM J. Control Optim., 36 (1998), pp. 840-851.
10. V. S. BORKAR AND V. R. KONDA, Actor-critic algorithm as multi-time scale stochastic approximation, Sādhanā, 22 (1997), pp. 525-543.
11. V. S. BORKAR AND S. P. MEYN, Stability and convergence of stochastic approximation using the ODE method, SIAM J. Control Optim., to appear.
12. V. S. BORKAR AND K. SOUMYANATH, A new analog parallel scheme for fixed point computation part I: Theory. IEEE Trans. Circuits Systems I Fund. Theory Appl., 44 (1997), pp. 351-355.
13. M. W. HIRSCH, Convergent activation dynamics in continuous time networks, Neural Networks, 2 (1989), pp. 331-349.
14. S. S. KEERTHI AND B. RAVINDRAN, A tutorial survey of reinforcement learning, Sādhanā, 19 (1994), pp. 851-889.
15. P. V. KOKOTOVIC, Applications of singular perturbation techniques to control problems, SIAM Rev., 26 (1984), pp. 501-550.
16. V. R. KONDA, Learning Algorithms for Markov Decision Processes, M.S. thesis, Department of Computer Science and Automation, Indian Institute of Science, Bangalore, India, 1997.
17. H. J. KUSHNER AND D. S. CLARK, Stochastic Approximation for Constrained and Unconstrained Systems, Springer-Verlag, New York, 1978.
18. J. NEVEU, Discrete Parameter Martingales, North-Holland, Amsterdam, 1975.
19. R. PEMANTLE, Non-convergence to unstable points in urn models and stochastic approximations, Ann. Probab., 18 (1990), pp. 698-712.
20. B. T. POLYAK, New method of stochastic approximation type, Automat. Remote Control, 51 (1990), pp. 937-946.
21. M. PUTERMAN, Markov Decision Processes, John Wiley, New York, 1994.
22. G. SANTHARAM AND P. S. SASTRY, A reinforcement learning neural network for adaptive control of Markov chains, IEEE Trans. Systems, Man and Cybernetics, 27 (1997), pp. 588-600.
23. M. SCHÄL, Estimation and control in discounted dynamic programming, Stochastics, 20 (1987), pp. 51-71.
24. J. N. TSITSIKLIS, Asynchronous stochastic approximation and Q-learning, Mach. Learning, 16 (1994), pp. 185-202.
25. J. N. TSITSIKLIS AND B. VAN ROY, Feature-based methods for large scale dynamic programming, Mach. Learning, 22 (1996), pp. 59-94.
26. C. WATKINS AND P. DAYAN, Q-learning, Mach. Learning, 8 (1992), pp. 279-292.
27. R. WILLIAMS AND L. BAIRD, A mathematical analysis of actor-critic architectures for learning optimal controls through incremental dynamic programming, in Sixth Yale Workshop on Adaptive and Learning Systems, New Haven. CT, 1990, pp. 96-101.
28. F. W. WILSON, Smoothing derivatives of functions and applications, Trans. Amer. Math. Soc., 139 (1969), pp. 413-428.