O.D.E. method for convergence of stochastic approximation and reinforcement learning

SIAM Journal on Control and Optimization

Volumn 38, Issue 2, 2000, Pages 447-469

  Borkar, V S ^a   Meyn, S P ^b  

^a TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; CONVERGENCE OF NUMERICAL METHODS; LEARNING SYSTEMS; RANDOM PROCESSES;

REINFORCEMENT LEARNING;

ORDINARY DIFFERENTIAL EQUATIONS;

EID: 0033876515     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0363012997331639     Document Type: Article

References (21)

1. J. ABOUNADI, D. BERTSEKAS, AND V. S. BORKAR, Learning algorithms for Markov decision processes with average cost, SIAM J. Control Optim., submitted.
2. A. G. BARTO, R. S. SUTTON, AND C. W. ANDERSON, Neuron-like elements that can solve difficult learning control problems, IEEE Trans. Systems, Man and Cybernetics, 13 (1983), pp. 835-846.
3. A. BENVENISTE, M. MÉTIVIER, AND P. PRIOURET, Adaptive Algorithms and Stochastic Approximations, Springer-Verlag, Berlin, 1990.
4. D. BERTSEKAS AND J. TSITSIKLIS, Neuro-Dynamic Programming, Athena Scientific, Belmont, MA, 1996.
5. V. S. BORKAR, Stochastic approximation with two time scales. Systems Control Lett., 29 (1997), pp. 291-294.
6. V. S. BORKAR, Asynchronous stochastic approximation, SIAM J. Control Optim., 36 (1998), pp. 840-851.
7. V. S. BORKAR, Recursive self-tuning control of finite Markov chains, Appl. Math., 24 (1996), pp. 169-188.
8. V. S. BORKAR AND K. SOUMYANATH, An analog scheme for fixed-point computation, part I: Theory, IEEE Trans. Circuits Systems I Fund. Theory Appl., 44 (1997), pp. 351-355.
9. J. G. DAI, On positive Harris recurrence for multiclass queueing networks: A unified approach via fluid limit models, Ann. Appl. Probab., 5 (1995), pp. 49-77.
10. J. G. DAI AND S. P. MEYN, Stability and convergence of moments for multiclass queueing networks via fluid limit models, IEEE Trans. Automat. Control, 40 (1995), pp. 1889-1904.
11. M. W. HIRSCH, Convergent activation dynamics in continuous time networks, Neural Networks, 2 (1989), pp. 331-349.
12. T. JAAKOLA, M. I. JORDAN, AND S. P. SINGH, On the convergence of stochastic iterative dynamic programming algorithms, Neural Computation, 6 (1994), pp. 1185-1201.
13. V. R. KONDA AND V. S. BORKAR, Actor-critic-type learning algorithms for Markov decision processes, SIAM J. Control Optim., 38 (1999), pp. 94-123.
14. H. J. KUSHNER AND G. G. YIN, Stochastic Approximation Algorithms and Applications, Springer-Verlag, New York, 1997.
15. V. A. MALYSHEV AND M. V. MEN'SIKOV, Ergodicity, continuity and analyticity of countable Markov chains, Trans. Moscow Math. Soc., 1 (1982), pp. 1-48.
16. S. P. MEYN AND R. L. TWEEDIE, Markov Chains and Stochastic Stability, Springer-Verlag, London, 1993.
17. S. P. MEYN AND R. L. TWEEDIE, Computable bounds for geometric convergence rates of Markov chains, Ann. Appl. Probab., 4 (1994), pp. 981-1011.
18. J. NEVEU, Discrete Parameter Martingales, North Holland, Amsterdam, 1975.
19. T. SARGENT, Bounded Rationality in Macroeconomics, Clarendon Press, Oxford, 1993.
20. J. TSITSIKLIS, Asynchronous stochastic approximation and q-learning, Mach. Learning, 16 (1994), pp. 195-202.
21. C. J. C. H. WATKINS AND P. DAYAN, Q-learning, Mach. Learning, 8 (1992), pp. 279-292.