Uncertainty handling CMA-ES for reinforcement learning

Proceedings of the 11th Annual Genetic and Evolutionary Computation Conference, GECCO-2009

Volumn , Issue , 2009, Pages 1211-1218

  Heidrich Meisner, Verena ^a   Igel, Christian ^a  

^a RUHR UNIVERSITY BOCHUM   (Germany)

Author keywords

Covariance matrix adaptation evolution strategy; Direct policy search; Reinforcement learning; Uncertainty handling

Indexed keywords

COVARIANCE MATRIX ADAPTATION EVOLUTION STRATEGIES; DIRECT POLICY SEARCH; EVOLUTIONARY LEARNING; LEARNING SPEED; NOISY OBSERVATIONS; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; POLICY GRADIENT METHODS; RANDOM SEARCHES; UNCERTAINTY HANDLING;

EDUCATION; EVOLUTIONARY ALGORITHMS; GRADIENT METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; REINFORCEMENT; REINFORCEMENT LEARNING; SIGNAL TO NOISE RATIO;

COVARIANCE MATRIX;

EID: 72749104931     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1569901.1570064     Document Type: Conference Paper

References (28)

1. S. Amari and H. Nagaoka. Methods of Information Geometry. Number 191 in Translations of Mathematical Monographs. American Mathematical Society and Oxford University Press, 2000.
2. D. V. Arnold. Noisy Optimization With Evolution Strategies. Kluwer Academic Publishers, 2002.
3. H.-G. Beyer. Evolution strategies. Scholarpedia, 2(8):1965, 2007.
4. C. Chen and E. Yucesan. An alternative simulation budget allocation scheme for efficient simulation. International Journal of Simulation and Process Modeling, 1(1):49-57, 2005.
5. R. Coulom. Apprentissage par renforcement utilisant des reseaux de neurones, avec des applications au controle moteur. These de doctorat, Institut National Polytechnique de Grenoble, 2002.
6. T. Glasmachers and C. Igel. Uncertainty handling in model selection for support vector machines. In G. Rudolph, editor, Parallel Problem Solving from Nature (PPSN X), volume 5199 of LNCS, pages 185-194. Springer-Verlag, 2008.
7. F. Gomez, J. Schmidhuber, and R. Miikkulainen. Accelerated neural evolution through cooperatively coevolved synapses. Journal of Machine Learning Research, 9:937-965, 2008.
8. N. Hansen, S. D. Müller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1):1-18, 2003.
9. N. Hansen, A. S. P. Niederberger, L. Guzzella, and P. Koumoutsakos. Evolutionary optimization of feedback controllers for thermoacoustic instabilities. In J. F. Morrison, D. M. Birch, and P. Lavoie, editors, IUTAM Symposium on Flow Control and MEMS. Springer-Verlag, 2008.
10. N. Hansen, A. S. P. Niederberger, L. Guzzella, and P. Koumoutsakos. A method for handling uncertainty in evolutionary optimization with an application to feedback control of combustion. IEEE Transactions on Evolutionary Computation, 13(1):180-197, 2009.
11. N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2):159-195, 2001.
12. V. Heidrich-Meisner and C. Igel. Evolution strategies for direct policy search. In G. Rudolph, editor, Parallel Problem Solving from Nature (PPSN X), number 5199 in LNCS, pages 428-437. Springer-Verlag, 2008.
13. V. Heidrich-Meisner and C. Igel. Uncertainty handling in evolutionary direct policy search. In Y. Engel, M. Ghavamzadeh, P. Poupart, and S. Mannor, editors, NIPS-08 Workshop on Model Uncertainty and Risk in Reinforcement Learning. 2008.
14. V. Heidrich-Meisner and C. Igel. Variable metric reinforcement learning methods applied to the noisy mountain car problem. In S. Girgin et al., editors, European Workshop on Reinforcement Learning (EWRL 2008), number 5323 in LNAI, pages 136-150. Springer-Verlag, 2008.
15. C. Igel. Neuroevolution for reinforcement learning using evolution strategies. In Congress on Evolutionary Computation (CEC 2003), volume 4, pages 2588-2595. IEEE Press, 2003.
16. C. Igel, T. Glasmachers, and V. Heidrich-Meisner. Shark. Journal of Machine Learning Research, 9:993-996, 2008.
17. L. Kaelbling, M. Littman, and A. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-134, 1998.
18. S. Kakade. A natural policy gradient. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems (NIPS14). MIT Press, 2002.
19. A. Pellecchia, C. Igel, J. Edelbrunner, and G. Schöner. Making driver modeling attractive. IEEE Intelligent Systems, 20(2):8-12, 2005.
20. J. Peters and S. Schaal. Natural actor-critic. Neurocomputing, 71(7-9):1180-1190, 2008.
21. J. Peters, S. Vijayakumar, and S. Schaal. Reinforcement learning for humanoid robotics. In Proc. 3rd IEEE-RAS Int'l Conf. on Humanoid Robots, pages 29-30, 2003.
22. M. Riedmiller, J. Peters, and S. Schaal. Evaluation of policy gradient methods and variants on the cart-pole benchmark. In Proc. IEEE Int'l Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2007), pages 254-261, 2007.
23. C. Schmidt, J. Branke, and S. Chick. Integrating techniques from statistical ranking into evolutionary algorithms. In Applications of Evolutionary Computing, volume 3907 of LNCS, pages 752-763. Springer, 2006.
24. N. T. Siebel and G. Sommer. Evolutionary reinforcement learning of artificial neural networks. International Journal of Hybrid Intelligent Systems, 4(3):171-183, 2007.
25. R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
26. R. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, pages 1057-1063, 2000.
27. T. Suttorp, N. Hansen, and C. Igel. Efficient covariance matrix update for variable metric evolution strategies. Machine Learning, 75(2):167-197, 2009.
28. S. Whiteson and P. Stone. Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7:877-917, 2006.