On actor-critic algorithms

SIAM Journal on Control and Optimization

Volumn 42, Issue 4, 2003, Pages 1143-1166

  Konda, Vijay R ^a   Tsitsiklis, John N ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Actor critic algorithms; Markov decision processes; Reinforcement learning; Stochastic approximation

Indexed keywords

ACTOR-CRITIC ALGORITHMS; MARKOV DECISION PROCESS (MDP); REINFORCEMENT LEARNING; STOCHASTIC APPROXIMATION;

APPROXIMATION THEORY; COMPUTER SIMULATION; CONVERGENCE OF NUMERICAL METHODS; FUNCTIONS; GRADIENT METHODS; ITERATIVE METHODS; LEARNING SYSTEMS; MARKOV PROCESSES; OPTIMIZATION; PARAMETER ESTIMATION; RANDOM PROCESSES; STOCHASTIC CONTROL SYSTEMS;

LEARNING ALGORITHMS;

EID: 4043069840     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0363012901385691     Document Type: Article

References (24)

1. K. B. ATHREYA AND P. NEY, A new approach to the limit theory of recurrent Markov chains, Trans. Amer. Math. Soc., 245 (1978), pp. 493-501.
2. A. BARTO, R. SUTTON, AND C. ANDERSON, Neuron-like elements that can solve difficult learning control problems, IEEE Transactions on Systems, Man and Cybernetics, 13 (1983), pp. 835-846.
3. A. BENVENISTE, M. METIVIER, AND P. PRIOURET, Adaptive Algorithms and Stochastic Approximations, Springer-Verlag, Berlin, Heidelberg, 1990.
4. D. P. BERTSEKAS, Dynamic Programming and Optimal Control, Athena Scientific, Belmont, MA, 1995.
5. D. P. BERTSEKAS AND J. N. TSITSIKLIS, Neuro-Dynamic Programming, Athena Scientific, Belmont, MA, 1996.
6. V. S. BORKAR, Stochastic approximation with two time scales, Systems Control Lett., 29 (1997), pp. 291-294.
7. X. R. CAO AND H. F. CHEN, Perturbation realization, potentials, and sensitivity analysis of Markov processes, IEEE Trans. Automat. Control, 42 (1997), pp. 1382-1393.
8. P. W. GLYNN, Stochastic approximation for Monte Carlo optimization, in Proceedings of the 1986 Winter Simulation Conference, Washington, DC, 1986, pp. 285-289.
9. P. W. GLYNN AND P. L'EcuYER, Likelihood ratio gradient estimation for stochastic recursions, Adv. Appl. Probab., 27 (1995), pp. 1019-1053.
10. T. JAAKKOLA, S. P. SINGH, AND M. I. JORDAN, Reinforcement learning algorithms for partially observable Markov decision problems, in Advances in Neural Information Processing Systems 7, G. Tesauro and D. Touretzky, eds., Morgan Kaufman, San Francisco, CA, 1995, pp. 345-352.
11. V. R. KONDA, Actor-Critic Algorithms, Ph.D. thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 2002.
12. V. R. KONDA AND V. S. BORKAR, Actor-critic-type learning algorithms for Markov decision processes, SIAM J. Control Optim., 38 (1999), pp. 94-123.
13. V. R. KONDA AND J. N. TSITSIKLIS, Actor-critic algorithms, in Advances in Neural Information Processing Systems 12, S. A. Solla, T. K. Leen, and K.-R. Muller, eds., MIT Press, Cambridge, MA, 2000, pp. 1008-1014.
14. V. R. KONDA AND J. N. TSITSIKLIS, Linear stochastic approximation driven by slowly varying Markov chains, 2002, submitted.
15. V. R. KONDA AND J. N. TSITSIKLIS, Appendix to "On Actor-critic algorithms,"http://web.mit.edu/jnt/www/Papers.html/actor-app.pdf, July 2002.
16. P. MARBACH AND J. N. TSITSIKLIS, Simulation-based optimization of Markov reward processes, IEEE Trans. Automat. Control, 46 (2001), pp. 191-209.
17. S. P. MEYN AND R. L. TWEEDIE, Markov Chains and Stochastic Stability, Springer-Verlag, London, 1993.
18. E. NUMMELIN, A splitting technique for Harris recurrent chains, Z. Wahrscheinlichkeitstheorie and Verw. Geb., 43 (1978), pp. 119-143.
19. R. BUTTON AND A. BARTO, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
20. R. S. SUTTON, D. MCALLESTER, S. SINGH, AND Y. MANSOUR, Policy gradient methods for reinforcement learning with function approximation, in Advances in Neural Information Processing Systems 12, S. A. Solla, T. K. Leen, and K.-R. Muller, eds., MIT Press, Cambridge, MA, 2000, pp. 1057-1063.
21. J. N. TSITSIKLIS AND B. VAN ROY, An analysis of temporal-difference learning with function approximation, IEEE Trans. Automat. Control, 42 (1997), pp. 674-690.
22. J. N. TSITSIKLIS AND B. VAN ROY, Average cost temporal-difference learning, Automatica J. IFAC, 35 (1999), pp. 1799-1808.
23. R. WILLIAMS, Simple statistical gradient following algorithms for connectionist reinforcement learning, Machine Learning, 8 (1992), pp. 229-256.
24. B. T. POLYAK, Pseudogradient adaptation and training algorithms, Autom. Remote Control, 34 (1973), pp. 377-397.