Analytical mean squared error curves for temporal difference learning

Machine Learning

Volumn 32, Issue 1, 1998, Pages 5-40

  Singh, Satinder ^a   Dayan, Peter ^b  

^a UCB 450   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Bias; Eligibility trace; Markov reward process; Monte Carlo; MSE; Reinforcement learning; Temporal difference; Variance

Indexed keywords

BIAS; ELIGIBILITY TRACE; MARKOV REWARD PROCESS; MEAN SQUARED ERROR CURVES; REINFORCEMENT LEARNING; TEMPORAL DIFFERENCE LEARNING; VARIANCE;

ALGORITHMS; CURVE FITTING; ERROR ANALYSIS; MARKOV PROCESSES; MAXIMUM LIKELIHOOD ESTIMATION; MONTE CARLO METHODS; PARAMETER ESTIMATION; PROBLEM SOLVING; COMPUTER SOFTWARE; TABLE LOOKUP;

LEARNING SYSTEMS;

MEAN SQUARED ERROR CURVES; TEMPORAL DIFFERENCE;

EID: 0032114627     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1007495401240     Document Type: Article

References (15)

1. Barnard, E. (1993). Temporal-difference methods and Markov models. IEEE Transactions on Systems, Man, and Cybernetics, 23(2), 357-365.
2. Barto, A. G. & Duff, M. (1994). Monte Carlo matrix inversion and reinforcement learning. In Advances in Neural Information Processing Systems 6, pages 687-694, San Mateo, CA. Morgan Kaufmann.
3. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 835-846.
4. Bucklew, J. A. (1990). Large Deviation Techniques in Decision, Simulation and Estimation. New York: Wiley-Interscience.
5. Dayan, P. (1992). The convergence of TD(λ) for general λ. Machine Learning, 8(3/4), 341-362.
6. Dayan, P. & Sejnowski, T. (1994). TD(λ) converges with probability 1. Machine Learning, 14, 295-301.
7. Haussler, D., Kearns, M., Seung, H. S., & Tishby, N. (1994). Rigorous learning curve bounds from statistical mechanics. In Proceedings of the 7th Annual ACM Workshop on Computational Learning Theory, pages 76-87, San Mateo, CA. Morgan Kauffman.
8. Jaakkola, T., Jordan, M. I., & Singh, S. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6), 1185-1201.
9. Saul, L. K. & Singh, S. (1996). Learning curves bounds for Markov decision processes with undiscounted rewards. In Proceedings of COLT.
10. Singh, S. & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, Vol. 22, 123-158.
11. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
12. Tsitsiklis, J. (1994). Asynchronous stochastic approximation and Q-learning. Machine Learning, 16(3), 185-202.
13. Wasow, W. R. (1952). A note on the inversion of matrices by random walks. Math. Tables Other Aids Comput., 6, 78-81.
14. Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. Ph.D Thesis, Cambridge Univ., Cambridge, England.
15. Widrow, B. & Stearns, S. D. (1985). Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall.