Neuroevolution strategies for episodic reinforcement learning

Journal of Algorithms

Volumn 64, Issue 4, 2009, Pages 152-168

  Heidrich Meisner, Verena ^a   Igel, Christian ^a  

^a RUHR UNIVERSITY BOCHUM   (Germany)

Author keywords

Covariance matrix adaptation; Direct policy search; Evolution strategy; Partially observable Markov decision process; Reinforcement learning

Indexed keywords

COMMON POLES; CONTINUOUS OPTIMIZATION; COVARIANCE MATRIX ADAPTATION; COVARIANCE MATRIX ADAPTATION EVOLUTION STRATEGIES; DIRECT POLICY SEARCH; EVOLUTION STRATEGY; MARKOVIAN; NEUROEVOLUTION; NON-MARKOVIAN; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; REINFORCEMENT SIGNAL; SEARCH ALGORITHMS; SEARCH DIRECTION;

COVARIANCE MATRIX; EDUCATION; EVOLUTIONARY ALGORITHMS; MARKOV PROCESSES; REINFORCEMENT; REINFORCEMENT LEARNING;

LEARNING ALGORITHMS;

EID: 70350567816     PISSN: 01966774     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jalgor.2009.04.002     Document Type: Article

References (54)

1. Albus J.S. A new approach to manipulator control: The cerebellar model articulation controller (CMAC). Trans. ASME J. Dyn. Syst. Meas. Control 97 (1975) 220-227
2. Arnold D.V. Noisy Optimization with Evolution Strategies (2002), Kluwer Academic Publishers
3. Auger A. Convergence results for the (1, λ)-SA-ES using the theory of φ-irreducible Markov chains. Theoret. Comput. Sci. 334 1 (2005) 35-69
4. Auger A., and Hansen N. A restart CMA evolution strategy with increasing population size. Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005) (2005), IEEE Press 1769-1776
5. Barto A.G., Sutton R.S., and Anderson C.W. Neuronlike elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybernet. 13 5 (1983) 835-846
6. Beyer H.-G. The Theory of Evolution Strategies. Nat. Comput. Ser. (2001), Springer-Verlag
7. Beyer H.-G. Evolution strategies. Scholarpedia 2 8 (2007) 1965
8. Beyer H.-G., and Schwefel H.-P. Evolution strategies: A comprehensive introduction. Nat. Comput. 1 1 (2002) 3-52
9. Chellapilla K., and Fogel D.B. Evolution neural networks, games, and intelligence. Proc. IEEE 87 9 (1999) 1471-1496
10. Fiete I.R., Fee M.S., and Seung H.S. Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. J. Neurophysiology 98 4 (2007) 2038
11. Floreano D., Dürr P., and Mattiussi C. Neuroevolution: From architectures to learning. Evol. Intelligence 1 1 (2008) 47-62
12. Gomez F., and Miikkulainen R. Solving non-markovian tasks with neuroevolution. In: Dean T. (Ed). Proceeding of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI) (1999), Morgan Kaufmann 1356-1361
13. Gomez F., Schmidhuber J., and Miikkulainen R. Efficient non-linear control through neuroevolution. Proc. European Conference on Machine Learning (ECML 2006). Lecture Notes in Comput. Sci. vol. 4212 (2006), Springer-Verlag 654-662
14. Gomez F., Schmidhuber J., and Miikkulainen R. Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9 (2008) 937-965
15. Gruau F., Whitley D., and Pyeatt L. A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza J.R., Goldberg D.E., Fogel D.B., and Riol R.L. (Eds). Genetic Programming 1996: Proceedings of the First Annual Conference (1996), MIT Press 81-89
16. Hansen N. (2006). http://www.bionik.tu-berlin.de/user/niko/cec2005.html
17. Hansen N. The CMA evolution strategy: A comparing review. In: Lozano J.A., Larranaga P., Inza I., and Bengoetxea E. (Eds). Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms (2006), Springer-Verlag 75-102
18. Hansen N., Müller S.D., and Koumoutsakos P. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11 1 (2003) 1-18
19. Hansen N., and Ostermeier A. Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9 2 (2001) 159-195
20. V. Heidrich-Meisner, C. Igel, Learning behavioral policies using extrinsic perturbations on the level of synapses, in: Frontiers in Computational Neuroscience, Conference Abstract: Bernstein Symposium 2008, doi:10.3389/conf.neuro.10.2008.01.060
21. Heidrich-Meisner V., and Igel C. Evolution strategies for direct policy search. In: Rudolph G. (Ed). Parallel Problem Solving from Nature (PPSN X). Lecture Notes in Comput. Sci. vol. 5199 (2008), Springer-Verlag 428-437
22. Heidrich-Meisner V., and Igel C. Similarities and differences between policy gradient methods and evolution strategies. In: Verleysen M. (Ed). 16th European Symposium on Artificial Neural Networks (ESANN) (2008), D-Side Publications, Evere, Belgium 149-154
23. Heidrich-Meisner V., and Igel C. Variable metric reinforcement learning methods applied to the noisy mountain car problem. In: Girgin S., et al. (Ed). European Workshop on Reinforcement Learning (EWRL 2008). Lecture Notes in Artificial Intelligence vol. 5323 (2008), Springer-Verlag 136-150
24. Igel C. Neuroevolution for reinforcement learning using evolution strategies. Congress on Evolutionary Computation (CEC 2003), vol. 4 (2003), IEEE Press 2588-2595
25. Igel C., Glasmachers T., and Heidrich-Meisner V. Shark. J. Mach. Learn. Res. 9 (2008) 993-996
26. Igel C., Erlhagen W., and Jancke D. Optimization of dynamic neural field. Neurocomputing 36 1-4 (2001) 225-233
27. Jägersküpper J. How the (1 + 1) ES using isotropic mutations minimizes positive definite quadratic forms. Theoret. Comput. Sci. 36 1 (2006) 38-56
28. Kakade S. A natural policy gradient. In: Dietterich T.G., Becker S., and Ghahramani Z. (Eds). Advances in Neural Information Processing Systems (NIPS14) (2002), MIT Press
29. Mandischer M. A comparison of evolution strategies and backpropagation for neural network training. Neurocomputing 42 1-4 (2002) 87-117
30. Michie D., and Chambers R.A. BOXES: An experiment in adaptive control. In: Dale E., and Michi D. (Eds). Machine Intelligence 2 (1968), Oliver & Boyd 137-152 (chap. 9)
31. Moriarty D.E., and Miikkulainen R. Efficient reinforcement learning through symbiotic evolution. Mach. Learn. 22 (1996) 11-32
32. Moriarty D.E., Schultz A.C., and Grefenstette J.J. Evolutionary algorithms for reinforcement learning. J. Artificial Intelligence Res. 11 (1999) 199-229
33. Pellecchia A., Igel C., Edelbrunner J., and Schöner G. Making driver modeling attractive. IEEE Intell. Syst. 20 2 (2005) 8-12
34. Peters J., and Schaal S. Natural actor-critic. Neurocomputing 71 7-9 (2008) 1180-1190
35. Rechenberg I. Evolutionsstrategie: Optimierung Technischer Systeme nach Prinzipien der Biologischen Evolution (1973), Frommann-Holzboog
36. M. Riedmiller, J. Peters, S. Schaal, Evaluation of policy gradient methods and variants on the cart-pole benchmark, in: Proc. IEEE Int'l Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2007), 2007, pp. 254-261
37. Santamaria J.C., Sutton R.S., and Ram A. Experiments with reinforcement learning in problems with continuous state and action spaces. Adapt. Behav. 6 2 (1997) 163
38. Saravanan N., and Fogel D.B. Evolving neural control systems. IEEE Expert 10 3 (1995) 23-27
39. J. Schmidhuber, S. Hochreiter, Guessing can outperform many long time lag algorithms, Technical Report IDSIA-19-96, Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale (IDSIA), 1996
40. Schwefel H.-P. Evolution and Optimum Seeking. Sixth-Generation Comput. Tech. Ser. (1995), John Wiley & Sons
41. Siebel N.T., and Sommer G. Evolutionary reinforcement learning of artificial neural networks. Internat. J. Hybrid Intell. Syst. 4 3 (2007) 171-183
42. Stagge P. Strukturoptimierung rückgekoppelter neuronaler Netze. Konzepte neuronaler Informationsverarbeitung (2001), ibidem-Verlag, Stuttgart
43. Stanley K.O., and Miikkulainen R. Evolving neural networks through augmenting topologies. Evol. Comput. 10 2 (2002) 99-127
44. Sutton R.S., and Barto A.G. Reinforcement Learning: An Introduction (1998), MIT Press
45. Sutton R.S., McAllester D., Singh S., and Mansour Y. Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst. vol. 12 (2000), MIT Press 1057-1063
46. Suttorp T., Hansen N., and Igel C. Efficient covariance matrix update for variable metric evolution strategies. Mach. Learn. 75 2 (2009) 167-197
47. Togelius J., Gomez F., and Schmidhuber J. Learning what to ignore: Memetic climbing in topology and weight space. Proceedings of the IEEE World Congress on Computational Intelligence (WCCI 2008) (2008), IEEE Press
48. Togelius J., Schaul T., Schmidhuber J., and Gomez F. Countering poisonous inputs with memetic neuroevolution. In: Rudolph G. (Ed). Parallel Problem Solving from Nature (PPSN X). Lecture Notes in Comput. Sci. vol. 5199 (2008), Springer-Verlag 610-619
49. Watkins C.J.C.H., and Dayan P. Q-learning. Mach. Learn. 8 (1992) 279-292
50. Whiteson S., and Stone P. Evolutionary function approximation for reinforcement learning. J. Mach. Learn. Res. 7 (2006) 877-917
51. Whitley D., Dominic S., Das R., and Anderson C.W. Genetic reinforcement learning for neurocontrol problems. Mach. Learn. 13 2-3 (1993) 259-284
52. Wieland A. Evolving controls for unstable systems. Proceedings of the International Joint Conference on Neural Networks, vol. 2 (1991), IEEE Press 667-673
53. Wierstra D., Foerster A., Peters J., and Schmidhuber J. Solving deep memory POMDPs with recurrent policy gradients. International Conference on Artificial Neural Networks (ICANN 2007). Lecture Notes in Comput. Sci. vol. 4668 (2007), Springer-Verlag 697
54. Yao X. Evolving artificial neural networks. Proc. IEEE 87 9 (1999) 1423-1447