Stochastic approximation for nonexpansive maps: Application to Q-learning algorithms

SIAM Journal on Control and Optimization

Volumn 41, Issue 1, 2003, Pages 1-22

  Abounadi, Jinane ^a   Bertsekas, Dimitri P ^a   Borkar, Vivek ^b  

^a USA   (United States)

^b TATA INSTITUTE OF FUNDAMENTAL RESEARCH   (India)

Author keywords

Neuro dynamic programming; Q learning; Stochastic approximation

Indexed keywords

APPROXIMATION THEORY; COMPUTER SIMULATION; CONVERGENCE OF NUMERICAL METHODS; ITERATIVE METHODS; LEARNING ALGORITHMS; LYAPUNOV METHODS; MATHEMATICAL MODELS; PROBLEM SOLVING; RANDOM PROCESSES; THEOREM PROVING;

CONVERGENCE ANALYSIS; STOCHASTIC APPROXIMATION;

DYNAMIC PROGRAMMING;

EID: 0037225359     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0363012998346621     Document Type: Article

References (26)

1. J. Abounadi, D. P. Bertsekas, and V. S. Borkar (2001), Learning algorithms for Markov decision processes with average cost, SIAM J. Control Optim., 40, pp. 681-698.
2. D. P. Bertsekas and J. N. Tsitsiklis (1989), Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Englewood Cliffs, NJ.
3. D. P. Bertsekas and J. N. Tsitsiklis (1991), An analysis of stochastic shortest path problems, Math. Oper. Res., 16, pp. 580-595.
4. D. P. Bertsekas and J. N. Tsitsiklis (1996), Neuro-Dynamic Programming, Athena Scientific, Belmont, MA.
5. D. P. Bertsekas (2001), Dynamic Programming and Optimal Control, 2nd ed., Athena Scientific, Belmont, MA.
6. A. Benveniste, M. Metivier, and P. Priouret (1990), Adaptive Algorithms and Stochastic Approximations, Springer-Verlag, New York.
7. P. Billingsley (1968), Convergence of Probability Measures, John Wiley, New York.
8. V. S. Borkar (1993), White noise representations in stochastic realization theory, SIAM J. Control Optim., 31, pp. 1093-1102.
9. V. S. Borkar (1995), Probability Theory: An Advanced Course, Springer-Verlag, New York.
10. V. S. Borkar (1998), Asynchronous stochastic approximations, SIAM J. Control Optim., 36, pp. 840-851. Correction note in ibid, 38 (2000), pp. 662-663.
11. V. S. Borkar and S. P. Meyn (2000), The O.D.E. method for convergence of stochastic approximation and reinforcement learning, SIAM J. Control Optim., 38, pp. 447-469.
12. V. S. Borkar and K. Soumyanath (1997), An analog parallel scheme for fixed point computation-Part I: Theory, IEEE Trans. Circuits Systems I Fund. Theory Appl., 44, pp. 351-355.
13. S. Csibi (1975), Learning under computational constraints from weakly dependent samples, Prob. Control Inform. Theory, 4, pp. 3-21.
14. L. Gerencsér (1992), Rate of convergence of recursive estimators, SIAM J. Control Optim., 30, pp. 1200-1227.
15. T. Jaakola, M. I. Jordan, and S. P. Singh (1994), On the convergence of stochastic iterative dynamic programming algorithms, Neural Computation, 6, pp. 1185-1201.
16. Y. Kifer (1986), Ergodic Theory of Random Transformations, Birkhäuser Boston, Cambridge, MA.
17. H. J. Kushner and D. S. Clark (1978), Stochastic Approximation Methods for Constrained and Unconstrained Systems, Springer-Verlag, New York.
18. H. J. Kushner and G. Yin (1997), Stochastic Approximation Algorithms and Applications, Springer-Verlag, New York.
19. L. Ljung (1977), Analysis of recursive stochastic algorithms, IEEE Trans. Automat. Control, 22, pp. 551-575.
20. K. Soumyanath and V. S. Borkar (1999), An analog scheme for fixed point computation-Part II: Applications, IEEE Trans. Circuits Systems I Fund. Theory Appl., 46, pp. 442-451.
21. P. Tseng, D. P. Bertsekas, and J. N. Tsitsiklis (1990), Partially asynchronous parallel algorithms for network flow and other problems, SIAM J. Control Optim., 28, pp. 678-710.
22. J. N. Tsitsiklis (1994), Asynchronous stochastic approximation and Q-learning, Machine Learning, 16, pp. 185-202.
23. C. J. C. H. Watkins (1989), Learning from delayed rewards, Ph.D. thesis, Cambridge University, Cambridge, England.
24. C. J. C. H. Watkins and P. Dayan (1992), Q-learning, Machine Learning, 8, pp. 279-292.
25. F. W. Wilson (1969), Smoothing derivatives of functions and applications, Trans. Amer. Math. Soc., 139, pp. 413-428.
26. T. Yoshizawa (1966). Stability Theory by Lyapunov's Second Method, Mathematical Society of Japan, Tokyo.