Co-evolution of shaping rewards and meta-parameters in reinforcement learning

Adaptive Behavior

Volumn 16, Issue 6, 2008, Pages 400-412

  Elfwing, Stefan ^a,b   Uchibe, Eiji ^b   Doya, Kenji ^b   Christensen, Henrik I ^a  

^a ROYAL INSTITUTE OF TECHNOLOGY   (Sweden)

^b OKINAWA INSTITUTE OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY   (Japan)

Author keywords

Genetic algorithms; Meta parameters; Reinforcement learning; Shaping rewards

Indexed keywords

EID: 55949119833     PISSN: 10597123     EISSN: 17412633     Source Type: Journal    
DOI: 10.1177/1059712308092835     Document Type: Article

References (26)

1. Ackley, D.H., & Littman, M.L. (1991). Interactions between learning and evolution. In C. G. Langton, C. Taylor, C. D. Farmer, & S. Rasmussen (Eds.), Artificial Life II: Santa Fe Institute Studies in the Sciences of Complexity (Vol. 10, pp. 487-509). Redwood City, CA: Addison-Wesley.
2. Bower, G.H., & Hilgard, E.R. (1981). Theories of learning (5 th ed.). Englewood Cliffs, NJ: Prentice-Hall.
3. Doya, K., & Uchibe, E. (2005). The cyber rodent project: Exploration of adaptive mechanisms for self-preservation and self-reproduction. Adaptive Behavior, 13 (2). 149-160.
4. Elfwing, S., Uchibe, U., Doya, K., & Christensen, H.I. (2005). Biologically inspired embodied evolution of survival. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC2005 (Vol. 3, pp. 2210-2216). Piscataway, NJ: IEEE.
5. Elfwing, S., Uchibe, U., Doya, K., & Christensen, H.I. (2007 a). Darwinian embodied evolution of the learning ability for survival. Manuscript submitted for publication.
6. Elfwing, S., Uchibe, U., Doya, K., & Christensen, H.I. (2007 b). Evolutionary development of hierarchical learning structures. IEEE Transactions on Evolutionary Computation, 11 (2). 249-264.
7. Eriksson, A., Capi, G., & Doya, K. (2003). Evolution of meta-parameters in reinforcement learning algorithm. In Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems, IROS2003 (pp. 412-417). Piscataway, NJ: IEEE.
8. Konidaris, G., & Barto, A. (2006). Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the International Conference on Machine Learning, ICML2006 (pp. 489-496). New York: ACM.
9. Laud, A., & DeJong, G. (2002). Reinforcement learning and shaping: Encouraging intended behaviors. In Proceedings of the International Conference on Machine Learning, ICML2002 (pp. 355-362). San Francisco, CA: Morgan Kaufmann.
10. Laud, A., & DeJong, G. (2003). The influence of reward on the speed of reinforcement learning: An analysis of shaping. In Proceedings of the International Conference on Machine Learning, ICML2003 (pp. 440-447). San Francisco, CA: Morgan Kaufmann.
11. Marthi, B. (2007). Automatic shaping and decomposition of reward functions. In Proceedings of the International Conference on Machine Learning, ICML2007 (pp. 601-608). New York: ACM.
12. Mataric, M. (1994). Reward functions for accelerated learning. In Proceedings of the International Conference on Machine Learning, ICML1994 (pp. 181-189). New York: ACM.
13. Mataric, M. (1997). Reinforcement learning in the multirobot domain. Autonomous Robots, 4 (1). 73-83.
14. Moriarty, D.E., Schultz, A.C., & Grefenstette, J.J. (1999). Evolutionary algorithms for reinforcement learning. Journal of Artificial Intelligence Research, 11, 241-276.
15. Ng, A.Y., Harada, D., & Russell, S.J. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning, ICML1999 (pp. 278-287). San Francisco, CA: Morgan Kaufmann.
16. Nolfi, S., & Floreano, D. (2000). Evolutionary robotics. The biology, intelligence, and technology of self-organizing machines. Cambridge, MA: MIT Press.
17. Randløv, J., & Alstrøm, P. (1998). Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the International Conference on Machine Learning, ICML1998. San Francisco, CA: Morgan Kaufmann.
18. Rummery, G.A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/ TR 166, Cambridge University Engineering Department.
19. Singh, S.P., & Sutton, R.S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22 (1-3). 123-158.
20. Skinner, B.F. (1938). The behavior of organisms: An experimental analysis. Englewood Cliffs, NJ: Prentice-Hall.
21. Stanley, K.O., & Miikkulainen, R. (2002). Effcient reinforcement learning through evolving neural network topologies. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO2002 (pp. 569-577). San Francisco, CA: Morgan Kaufmann.
22. Sutton, R.S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in Neural Information Processing Systems 8 (pp. 1038-1044). Cambridge, MA: MIT Press.
23. Sutton, R.S., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
24. Watson, R., Ficici, S., & Pollack, J. (2002). Embodied evolution: Distributing an evolutionary algorithm in a population of robots. Robotics and Autonomous Systems, 39 (1). 1-18.
25. Whiteson, S., & Stone, P. (2006). Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7, 877-917.
26. Wiewiora, E. (2003). Potential-based shaping and Q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19, 205-208.