Reinforcement learning for a biped robot based on a CPG-actor-critic method

Neural Networks

Volumn 20, Issue 6, 2007, Pages 723-735

  Nakamura, Yutaka ^a,b   Mori, Takeshi ^a   Sato, Masa aki ^c   Ishii, Shin ^a  

^a NARA INSTITUTE OF SCIENCE AND TECHNOLOGY   (Japan)

^b OSAKA UNIVERSITY   (Japan)

^c ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

Actor critic model; Biped walking; Central pattern generator; Policy gradient method; Reinforcement learning

Indexed keywords

AUTONOMOUS AGENTS; BIPED LOCOMOTION; GRADIENT METHODS; MOTION PLANNING; RANDOM PROCESSES; REINFORCEMENT LEARNING;

ACTOR-CRITIC MODEL; BIPED WALKING; CENTRAL PATTERN GENERATORS; POLICY GRADIENT METHODS;

MOBILE ROBOTS;

ALGORITHM; ANIMAL BEHAVIOR; ARTICLE; GAIT; LEARNING; LOCOMOTION; MATHEMATICAL MODEL; NERVE CELL; NERVE CELL NETWORK; PRIORITY JOURNAL; REINFORCEMENT; ROBOTICS;

ANIMALS; LOCOMOTION; NEURAL NETWORKS (COMPUTER); REINFORCEMENT (PSYCHOLOGY); ROBOTICS;

EID: 34547769694     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2007.01.002     Document Type: Article

References (53)

1. Amari S. Natural gradient works efficiently in learning. Neural Computation 10 2 (1998) 251-276
2. Asano F., Yamakita M., Kamamichi N., and Luo Z.-W. A novel gait generation for biped walking robots based on mechanical energy constraint. IEEE Transactions on Robotics and Automation 20 3 (2004) 565-573
3. Bentivegna, D. C., Ude, A., Atkeson, C. G., & Cheng, G. (2002). Humanoid robot learning and game playing using pc-based vision. In Proceedings of the 2002 IEEE/RSJ international conference on intelligent robots and systems (pp. 2449-2454)
4. Bradtke S.J., and Barto A.G. Linear least-squares algorithms for temporal difference learning. Machine Learning 22 (1996) 33-57
5. Cruse H., Kindermann T., Schumm M., Dean J., and Schmitz J. Walknet - A biologically inspired network to control six-legged walking. Neural Networks 11 7-8 (1998) 1435-1447
6. Doya K., and Yoshizawa S. Adaptive synchronization of neural and physical oscillators. Advances in Neural Information Processing Systems 4 (1992) 109-116
7. Ekeberg Ö. A combined neuronal and mechanical model of fish swimming. Biological Cybernetics 69 (1993) 363-374
8. Fukuoka Y., Kimura H., and Cohen A.H. Adaptive dynamic walking of a quadruped robot on irregular terrain based on biological concepts. International Journal of Robotics Research 22 3-4 (2003) 187-202
9. Grillner S., Wallen P., Brodin L., and Lansner A. Neuronal network generating locomotor behavior in lamprey: Circuitry, transmitters, membrane properties and simulations. Annual Review of Neuroscience 14 (1991) 169-199
10. Hirai, K., Hirose, M., Haikawa, Y., & Takenaka, T. (1998). The development of honda humanoid robot. In Proceedings of the 1998 IEEE international conference on robotics & automation (pp. 1321-1326)
11. Hitomi, K., Shibata, T., Nakamura, Y., & Ishii, S. (2005). On-line learning of a feedback controller for quasi-passive-dynamic walking by a stochastic policy gradient method. In IEEE/RSJ international conference on intelligent robots and systems (pp. 1923-1928)
12. Ijspeert A.J., and Cabeluguen J.-M. Gait transition from swimming to walking: Investigation of salamander locomotion control using nonlinear oscillators (2003)
13. Inada, H., & Ishii, K. (2003). Behavior generation of bipedal robot using central pattern generator(CPG) (1st report: Cpg parameters searching method by genetic algorithm). In Proceedings of international conference on intelligent robots and systems: Vol. 3 (pp. 2179-2184)
14. Inamura T., Nakamura Y., and Toshima I. Embodied symbol emergence based on mimesis theory. International Journal of Robotics Research 23 4 (2004) 363-377
15. Ishii S., Yoshida W., and Yoshimoto J. Control of exploitation-exploration meta-parameter in reinforcement learning. Neural Networks 15 4 (2002) 665-687
16. Itoh, Y., Taki, K., Kato, S., & Itoh, H. (2004). A stochastic optimization method of CPG-based motion control for humanoid locomotion. In IEEE conference on robotics, automation and mechatronics (pp. 347-351)
17. Kaelbling L.P., Littman M.L., and Moore A.P. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996) 237-285
18. Kakade S. A natural policy gradient. Advances in neural information processing systems Vol. 14 (2001) 1531-1538
19. Keller, P. W., Mannor, S., & Precup, D. (2006). Automatic basis function construction for approximate dynamic programming and reinforcement learning. In The 23rd international conference on machine learning
20. Kimura, H., & Kobayashi, S. (1998). An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value function. In 15th international conference on machine learning (pp. 278-286)
21. Konda V.R., and Tsitsiklis J.N. Actor-critic algorithms. SIAM Journal on Control and Optimization 42 4 (2003) 1143-1146
22. Kotosaka, S., & Schaal, S. (2000). Synchronized robot drumming by neural oscillator. In The international symposium on adaptive motion of animals and machines
23. Lagoudakis, M. G., Parr, R., & Littman, M. L. (2002). Least-squares methods in reinforcement learning for control. In Methods and applications of artificial intelligence, second hellenic conference on AI, SETN (pp. 249-260)
24. Lewis, M., Fagg, A., & Bekey, G. (1993). Genetic algorithms for gait synthesis in a hexapod robot. In Recent trends in mobile robots (pp. 317-331)
25. Lim H., Yamamoto Y., and Takanishi A. Stabilization control for biped follow walking. Advanced Robotics 16 4 (2002) 361-380
26. Matsuoka K. Sustained oscillations generated by mutually inhibiting neurons with adaption. Biological Cybernetics 52 (1985) 367-376
27. McGeer T. Passive dynamic walking. International Journal of Robotics Research 9 2 (1990) 62-82
28. Menache I., Mannor S., and Shimkin N. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research 134 1 (2005) 215-238
29. Miyashita K., Ok S., and Hase K. Evolutionary generation of human-like bipedal locomotion. Mechatronics 13 (2003) 791-807
30. Morimoto J., and Atkeson C.G. Minimax differential dynamic programming: An application to robust biped walking. Advances in Neural Information Processing Systems 15 (2003) 1539-1546
31. Morimoto J., and Doya K. Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. Robotics and Autonomous Systems 36 (2001) 37-51
32. Nakamura, Y., Sato, M., & Ishii, S. (2003). Reinforcement learning for biped robot. In 2nd international symposium on adaptive motion of animals and machines
33. Nakanishi J., Morimoto J., Endo G., Cheng G., Schaal S., and Kawato M. Learning from demonstration and adaptation of biped locomotion. Robotics and Autonomous Systems 47 (2004) 79-91
34. Nishii J. Legged insects select the optimal locomotor pattern based on the energetic cost. Journal of Biological Cybernetics 83 5 (2000)
35. Ogihara N., and Yamazaki N. Generation of human bipedal locomotion by a bio-mimetic neuro-musculo-skeletal model. Biological Cybernetics 84 (2001) 1-11
36. Park H., Amari S., and Fukumizu K. Natural gradient works efficiently in learning. Neural Networks 13 (2000) 755-764
37. Pearlmutter B.A. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks 6 5 (1995) 1212-1228
38. Peters, J., Vijayakumar, S., & Schaal, S. (2003). Reinforcement learning for humanoid robotics. In Third IEEE international conference on humanoid robotics 2003
39. Peters, J., Vijayakumar, S., & Schaal, S. (2005). Natural actor-critic. In The 16th European conference on machine learning
40. Ratitch, B., & Precup, D. (2004). Sparse distributed memories for on-line value-based reinforcement learning. In The 16th European conference on machine learning (pp. 347-358)
41. Sato M. A real time learning algorithm for recurrent analog neural networks. Biological Cybernetics 62 (1990) 237-241
42. Sato M., and Ishii S. Reinforcement learning based on on-line em algorithm. Advances in Neural Information Processing Systems 11 (1999) 1052-1058
43. Schaal, S., Peters, J., & Ijspeert, J. N. A. (2004). Learning movement primitives. In International symposium on robotics research
44. Sutton R.S., and Barto A.G. Reinforcement learning: An introduction (1998), MIT Press
45. Sutton R.S., McAllester D., Singh S., and Manour Y. Policy gradient method for reinforcement learning with function approximation. Advances in neural information processing systems Vol. 12 (2000) 1057-1063
46. Taga G., Yamaguchi Y., and Shimizu H. Self-organized control of bipedal locomotion by neural oscillators in unpredictable environment. Biological Cybernetics 65 (1991) 147-159
47. Tedrake, R., Zhang, T. W., & Seung, H. S. (2004). Stochastic policy gradient reinforcement learning on a simple 3d biped. In Proceedings of the IEEE international conference on intelligent robots and systems(pp. 2849-2854)
48. Thrun S.B. The role of exploration in learning control with neural networks. In: White D.A., and Sofge D.A. (Eds). Handbook of intelligent control: Neural, fuzzy and adaptive approaches (1992), Van Nostrand Reinhold, Florence, Kentucky
49. Wadden T., and Ekeberg O. A neuro-mechanical model of legged locomotion: Single leg control. Biological Cybernetics 79 2 (1998) 161-173
50. White D.A., and Sofge D.A. Applied learning: Optimal control for manufacturing (1992) (pp. 259-282)
51. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (1992) 229-256
52. Williamson M.M. Neural control of rhythmic arm movements. Neural Networks 11 7-8 (1998) 1379-1394
53. Yoshimoto, J., Ishii, S., & Sato, M. (2000). On-line em reinforcement learning. In IEEE-INNS-ENNS international joint conference on neural networks: Vol. 3 (pp. 163-168)