Geometric variance reduction in Markov chains: Application to value function and gradient estimation

Journal of Machine Learning Research

Volumn 7, Issue , 2006, Pages 413-427

  Munos, Rémi ^a  

^a CENTRE DE MATHÉMATIQUES APPLIQUÉES   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; ERROR ANALYSIS; ITERATIVE METHODS; MONTE CARLO METHODS; OPTIMIZATION; PARAMETER ESTIMATION; SEQUENTIAL CIRCUITS; TRAJECTORIES; VALUE ENGINEERING;

APPROXIMATION ERRORS; GEOMETRIC VARIANCE REDUCTION; SEQUENTIAL CONTROL; VALUE FUNCTION ESTIMATION;

MARKOV PROCESSES;

EID: 33646384929     PISSN: 15337928     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (18)

1. C. G. Atkeson, S. A. Schaal, and Andrew W. Moore. Locally weighted learning. AI Review, 11, 1997.
2. J. Baxter and P. L. Bartlett. Infinite-horizon gradient-based policy search. Journal of Artificial Intelligence Research, 15:319-350, 2001.
3. P. W. Glynn. Likelihood ratio gradient estimation: an overview. In A. Thesen, H. Grant, and W. D. Kelton, editors, Proceedings of the 1987 Winter Simulation Conference, pages 366-375, 1987.
4. E. Gobet and S. Maire. Sequential control variates for functionals of Markov processes. SIAM Journal on Numerical Analysis, 43(3): 1256-1275, 2005.
5. E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5:1471-1530, 2005.
6. J. H. Halton. A retrospective and prospective survey of the Monte-Carlo method. SIAM Review, 12 (1):1-63, 1970.
7. J. H. Halton. Sequential Monte-Carlo techniques for the solution of linear systems. Journal of Scientific Computing, 9:213-257, 1994.
8. J. M. Hammersley and D. C. Handscomb. Monte-Carlo Methods. Chapman and Hall, 1964.
9. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer Series in Statistics, 2001.
10. C. Kollman, K. Baggerly, D. Cox, and R. Picard. Adaptive importance sampling on discrete Markov chains. The Annals of Applied Probability, 9(2):391-412, 1999.
11. V. R. Konda and V. S. Borkar. Actor-critic-type learning algorithms for Markov decision processes. SIAM Journal of Control and Optimization, 38:1:94-123, 1999.
12. S. Maire. An iterative computation of approximations on Korobov-like spaces. J. Comput. Appl. Math., 54(6):261-281, 2003.
13. P. Marbach and J. N. Tsitsiklis. Approximate gradient methods in policy-space optimization of Markov reward processes. Journal of Discrete Event Dynamical Systems, 13:111-148, 2003.
14. M. I. Reiman and A. Weiss. Sensitivity analysis via likelihood ratios. In J. Wilson, J. Henriksen, and S. Roberts, editors, Proceedings of the 1986 Winter Simulation Conference, pages 285-289, 1986.
15. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Neural Information Processing Systems. MIT Press, pages 1057-1063, 2000.
16. V. Vapnik, Statistical Learning Theory. John Wiley & Sons, New York, 1998.
17. V. Vapnik, S. E. Golowich, and A. Smola. Support vector method for function approximation, regression estimation and signal processing. In Advances in Neural Information Processing Systems, pages 281-281, 1997.
18. R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.