A worst-case comparison between temporal difference and residual gradient with linear function approximation

Proceedings of the 25th International Conference on Machine Learning

Volumn , Issue , 2008, Pages 560-567

  Li, Lihong ^a  

^a RUTGERS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING ALGORITHMS; MACHINE LEARNING; MARKOV PROCESSES; FUNCTIONS; INTERNET; LEARNING SYSTEMS; PROBABILITY DENSITY FUNCTION; ROBOT LEARNING;

FORMAL ANALYSIS; FUNCTION APPROXIMATION; LINEAR FUNCTIONS; NON-PROBABILISTIC; POLICY EVALUATION; PREDICTION ERRORS; RESIDUAL GRADIENT; TEMPORAL DIFFERENCES;

APPROXIMATION ALGORITHMS;

FORMAL ANALYSIS; FUNCTION APPROXIMATIONS; LINEAR FUNCTIONS; MARKOV CHAINS; MARKOVIAN; NON-PROBABILISTIC; ON-LINE LEARNING; POLICY EVALUATIONS; PREDICTION ERRORS; RESIDUAL GRADIENTS; TEMPORAL DIFFERENCES;

EID: 56449125197     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390156.1390227     Document Type: Conference Paper

References (15)

1. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. Proceedings of the Twelfth International Conference on Machine Learning (ICML-95) (pp. 30-37).
2. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Athena Scientific.
3. Boyan, J. A., & Moore, A. W. (1995). Generalization in reinforcement learning: Safely approximating the value function. Advances in Neural Information Processing Systems 7 (NIPS-94) (pp. 369-376).
4. Cesa-Bianchi, N., Long, P. M., & Warmuth, M. (1996). Worst-case quadratic loss bounds for prediction using linear functions and gradient descent. IEEE Transactions on Neural Networks, 7, 604-619.
5. Horn, R. A., & Johnson, C. R. (1986). Matrix analysis. Cambridge University Press.
6. Kivinen, J., & Warmuth, M. K. (1997). Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132, 1-63.
7. Lagoudakis, M. G., & Parr, R. (2003). Least-squares policy iteration. Journal of Machine Learning Research, 4, 1107-1149.
8. Munos, R. (2003). Error bounds for approximate policy iteration. Proceedings of the Twentieth International Conference on Machine Learning (ICML-03) (pp. 560-567).
9. Precup, D., & Sutton, R. S. (1997). Exponentiated gradient methods for reinforcement learning. Proceedings of the Fourteenth International Conference on Machine Learning (ICML-97) (pp. 272-277).
10. Puterman, M. L. (1994). Markov decision processes: Discrete stochastic dynamic programming. New York: Wiley-Interscience.
11. Schapire, R. E., & Warmuth, M. K. (1996). On the worst-case analysis of temporal-difference learning algorithms. Machine Learning, 22, 95-122.
12. Schoknecht, R., & Merke, A. (2003). TD(0) converges provably faster than the residual gradient algorithm. Proceedings of the Twentieth International Conference on Machine Learning (ICML-03) (pp. 680-687).
13. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
14. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
15. Tsitsiklis, J. N., & Van Roy, B. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42, 674-690.