Variance reduction techniques for gradient estimates in reinforcement learning

Advances in Neural Information Processing Systems

Volumn , Issue , 2002, Pages

  Greensmith, Evan ^a   Bartlett, Peter L ^b   Baxter, Jonathan ^c  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b BIOwulf Technologies   (Australia)

^c WhizBang Labs East   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ESTIMATION; OPTIMIZATION;

ACTOR-CRITIC METHODS; APPROXIMATE VALUE FUNCTION; EXPECTED SQUARED ERRORS; GRADIENT ESTIMATES; OPTIMAL VALUE FUNCTIONS; POSITIVE VARIANCES; VARIANCE OF ESTIMATES; VARIANCE REDUCTION TECHNIQUES;

REINFORCEMENT LEARNING;

EID: 84898983933     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (15)

1. L. Baird and A. Moore. Gradient descent for general reinforcement learning. In Advances in Neural Information Processing Systems 11, pages 968-974. MIT Press, 1999.
2. P. L. Bartlett and J. Baxter. Estimation and approximation bounds for gradient-based reinforcement learning. Journal of Computer and Systems Sciences, 2002. To appear.
3. A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13:834-846, 1983.
4. J. Baxter and P. L. Bartlett. Infinite-horizon gradient-based policy search. Journal of Artificial Intelligence Research, 15:319-350, 2001.
5. J. Baxter, P. L. Bartlett, and L. Weaver. Infinite-horizon gradient-based policy search: II. Gradient ascent algorithms and experiments. Journal of Artificial Intelligence Research, 15:351-381, 2001.
6. M. Evans and T. Swartz. Approximating integrals via Monte Carlo and deterministic methods. Oxford University Press, 2000.
7. P. W. Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33:75-84, 1990.
8. E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Technical report, ANU, 2002.
9. H. Kimura, K. Miyazaki, and S. Kobayashi. Reinforcement learning in POMDPs with function approximation. In D. H. Fisher, editor, Proceedings of the Fourteenth International Conference on Machine Learning (ICML'97), pages 152-160, 1997.
10. V. R. Konda and J. N. Tsitsiklis. Actor-Critic Algorithms. In Advances in Neural Information Processing Systems 12, pages 1008-1014. MIT Press, 2000.
11. P. Marbach and J. N. Tsitsiklis. Simulation-Based Optimization of Markov Reward Processes. Technical report, MIT, 1998.
12. R. Y. Rubinstein. How to optimize complex stochastic systems from a single sample path by the score function method. Ann. Oper. Res., 27:175-211, 1991.
13. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge MA, 1998. ISBN 0-262-19398-1.
14. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems 12, pages 1057-1063. MIT Press, 2000.
15. R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8:229-256, 1992.