Reinforcement learning of motor skills with policy gradients

Neural Networks

Volumn 21, Issue 4, 2008, Pages 682-697

  Peters, Jan ^a,b   Schaal, Stefan ^b,c  

^a MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

Motor primitives; Motor skills; Natural Actor Critic; Natural gradients; Policy gradient methods; Reinforcement learning

Indexed keywords

ARTIFICIAL LIMBS; BEHAVIORAL RESEARCH; CONTINUOUS TIME SYSTEMS; ERROR ANALYSIS; GRADIENT METHODS; MOTOR TRANSPORTATION;

MOTOR PRIMITIVES; MOTOR SKILLS; NATURAL GRADIENTS; POLICY GRADIENT METHODS;

REINFORCEMENT LEARNING;

ALGORITHM; ARTICLE; LEARNING; MACHINE; MOTOR CONTROL; MOTOR PERFORMANCE; NEUROBIOLOGY; POLICY; PRIORITY JOURNAL; REINFORCEMENT; ROBOTICS; STOCHASTIC MODEL;

ALGORITHMS; ANIMALS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; EXTREMITIES; FEEDBACK; FORELIMB; HUMANS; LOCOMOTION; MOTOR SKILLS; MOVEMENT; REINFORCEMENT (PSYCHOLOGY); ROBOTICS; STOCHASTIC PROCESSES;

EID: 44949241322     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2008.02.003     Document Type: Article

References (76)

1. Aberdeen, D. (2006). POMDPs and policy gradients, presentation at the Machine Learning Summer School (MLSS)
2. Aleksandrov V., Sysoyev V., and Shemeneva V. Stochastic optimization. Engineering Cybernetics 5 (1968) 11-16
3. Amari S. Natural gradient works efficiently in learning. Neural Computation 10 (1998) 251
4. Atkeson C.G. Using local trajectory optimizers to speed up global optimization in dynamic programming. In: Hanson J.E., Moody S.J., and Lippmann R.P. (Eds). Advances in neural information processing systems 6 (1994), Morgan Kaufmann 503-521
5. Bagnell, J., & Schneider, J. (2003). Covariant policy search. In Proceedings of the international joint conference on artificial intelligence (pp. 1019-1024)
6. Baird, L. (1993). Advantage updating. Technical Report WL-TR-93-1146. Wright laboratory, Wright-Patterson air force base. OH
7. Balasubramanian V. Statistical inference, occam's razor, and statistical mechanics on the space of probability distributions. Neural Computation 9 2 (1997) 349-368
8. Barto A.G., Sutton R.S., and Anderson C.W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics SMC 13 5 (1983) 115-133
9. Baxter J., and Bartlett P. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research 15 (2001) 319-350
10. Baxter J., Bartlett P., and Weaver L. Experiments with infinite-horizon, policy- gradient estimation. Journal of Artificial Intelligence Research 15 (2001) 351-381
11. Ben-Itzhak S., and Karniel A. Minimum acceleration criterion with constraints implies bang-bang control as an underlying principle for optimal trajectories of arm reaching movements. Neural Computation 20 3 (2008) 779-812
12. Benbrahim, H., Doleac, J., Franklin, J., & Selfridge, O. (1992). Real-time learning: A ball on a beam. In Proceedings of the international joint conference on neural networks (pp. 92-103)
13. Benbrahim H., and Franklin J. Biped dynamic walking using reinforcement learning. Robotics and Autonomous Systems 22 (1997) 283-302
14. Berny A. Statistical machine learning and combinatorial optimization. Lecture notes in natural computing Vol. 33 (2000), Springer-Verlag, Heidelberg, Germany 287-306
15. Dyer P., and McReynolds S.R. The computation and theory of optimal control (1970), Academic Press, New York, NY
16. Endo, G., Morimoto, J., Matsubara, T., Nakanishi, J., & Cheng, G. (2005). Learning cpg sensory feedback with policy gradient for biped locomotion for a full-body humanoid. In Proceedings of the national conference on artificial intelligence (pp. 1267-1273)
17. Flash T., and Hochner B. Motor primitives in vertebrates and invertebrates. Current Opinions in Neurobiology 15 (2005) 660-666
18. Fletcher R., and Fletcher R. Practical methods of optimization (2000), John Wiley & Sons, New York, NY
19. Fu M.C. Feature article: Optimization for simulation: Theory vs. practice. INFORMS Journal on Computing 14 3 (2002) 192-215
20. Glynn, P. (1987). Likelihood ratio gradient estimation: An overview. In Proceedings of the winter simulation conference (pp. 366-375)
21. Glynn P. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM 33 10 (1990) 75-84
22. Greensmith E., Bartlett P., and Baxter J. Variance reduction techniques for gradient estimates in reinforcement learning. Advances in Neural Information Processing Systems 14 34 (2001)
23. Greensmith E., Bartlett P.L., and Baxter J. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research 5 (2004) 1471-1530
24. Guenter, F., Hersch, M., Calinon, S., & Billard, A. (2007). Reinforcement learning for imitating constrained reaching movements. In Imitative Robots [Special issue]. RSJ Advanced Robotics, 21, 1521-1544
25. Gullapalli V. A stochastic reinforcement learning algorithm for learning real-valued functions. Neural Networks 3 6 (1990) 671-692
26. Gullapalli, V. (1992). Learning control under extreme uncertainty. In Advances in neural information processing systems (pp. 327-334)
27. Gullapalli V., Franklin J., and Benbrahim H. Aquiring robot skills via reinforcement learning. IEEE Control Systems Journal, Special Issue on Robotics: Capturing Natural Motion 4 1 (1994) 13-24
28. Harville D.A. Matrix algebra from a statistician's perspective (2000), Springer Verlag, Heidelberg, Germany
29. Hasdorff L. Gradient optimization and nonlinear control (1976), John Wiley & Sons, New York, NY
30. Ijspeert, J.A., Nakanishi, J., & Schaal, S. (2002). Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings of IEEE international conference on robotics and automation (pp. 1398-1403)
31. Ijspeert A., Nakanishi J., and Schaal S. Learning attractor landscapes for learning motor primitives. In: Becker S., Thrun S., and Obermayer K. (Eds). Advances in neural information processing systems Vol. 15 (2003), MIT Press, Cambridge, MA 1547-1554
32. Jacobson D.H., and Mayne D.Q. Differential dynamic programming (1970), American Elsevier Publishing Company, Inc, New York, NY
33. Kakade, S. (2001). Optimizing average reward using discounted rewards. In Proceedings of the conference on computational learning theory (pp. 605-615)
34. Kakade S.A. Natural policy gradient. Advances in neural information processing systems Vol. 14 (2002), Vancouver, CA 1531-1538
35. Kakade, S.M. (2003). On the sample complexity of reinforcement learning. Ph.D. thesis, Gatsby computational Neuroscience Unit. University College London, London, UK
36. Kimura, H., & Kobayashi, S. (1997). Reinforcement learning for locomotion of a two-linked robot arm. In Proceedings of the Europian workshop on learning robots (pp. 144-153)
37. Kimura, H., & Kobayashi, S. (1998). Reinforcement learning for continuous action using stochastic gradient ascent. In Proceedings of the international conference on intelligent autonomous systems (IAS): Vol. 5 (pp. 288-295)
38. Kleinman N., Spall J., and Naiman D. Simulation-based optimization with stochastic approximation using common random numbers. Management Science 45 (1999) 1570-1578
39. Kohl, N., & Stone, P. (2004). Policy gradient reinforcement learning for fast quadrupedal locomotion. In Proceedings of the IEEE international conference on robotics and automation (pp. 2619-2624)
40. Konda V., and Tsitsiklis J. Actor-critic algorithms. Advances in Neural Information Processing Systems (2000) 12
41. Lawrence, G., Cowan, N., & Russell, S. (2003). Efficient gradient estimation for motor control learning. In Proceedings of the international conference on uncertainty in artificial intelligence (pp. 354-361)
42. Mitsunaga, N., Smith, C., Kanda, T., Ishiguro, H., & Hagita, N. (2005). Robot behavior adaptation for human-robot interaction based on policy gradient reinforcement learning. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1594-1601)
43. Miyamoto, H., Gandolfo, F., Gomi, H., Schaal, S., Koike, Y., & Osu, R. et al. (1995). A kendama learning robot based on a dynamic optimization theory. In Proceedings of the IEEE international workshop on robot and human communication (pp. 327-332)
44. Miyamoto, H., Gandolfo, F., Gomi, H., Schaal, S., Koike, Y., & Rieka, O. et al. (1996). A kendama learning robot based on a dynamic optimization principle. In Proceedings of the international conference on neural information processing (pp. 938-942)
45. Moon T., and Stirling W. Mathematical methods and algorithms for signal processing (2000), Prentice Hall, Upper Saddle River, NJ
46. Mori, T., Nakamura, Y., aki Sato, M., & Ishii, S. (2004). Reinforcement learning for cpg-driven biped robot. In Proceedings of the national conference on artificial intelligence (pp. 623-630)
47. Mori, T., Nakamura, Y., & Ishii, S. (2005). Efficient sample reuse by off-policy natural actor-critic learning. In Advances in neural information processing systems (NIPS '05 workshop presentation)
48. Morimoto J., and Atkeson C.A. Minimax differential dynamic programming: an application to robust biped walking. In: Becker S., Thrun S., and Obermayer K. (Eds). Advances in neural information processing systems 15 (2003), MIT Press, Cambridge, MA 1539-1546
49. Nakamura, Y., Mori, T., & Ishii, S. (2004). Natural policy gradient reinforcement learning for a CPG control of a biped robot. In Proceedings of the international conference on parallel problem solving from nature (pp. 972-981)
50. Nakanishi J., Morimoto J., Endo G., Cheng G., Schaal S., and Kawato M. Learning from demonstration and adaptation of biped locomotion. Robotics and Autonomous Systems 47 2-3 (2004) 79-91
51. Ng, A.Y., & Jordan, M. (2000). PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of the international conference on uncertainty in artificial intelligence (pp. 406-415)
52. Park J., Kim J., and Kang D. An RLS-based natural actor-critic algorithm for locomotion of a two-linked robot arm. In: Hao Y., Liu J., Wang Y., ming Cheung Y., Yin H., Jiao L., Ma J., and Jiao Y.-C. (Eds). Proceedings of the international conference on computational intelligence and security (CIS). Lecture notes in computer science Vol. 3801 (2005), Springer, Xi'an, China 65-72
53. Peters, J. (2005). Machine learning of motor skills for robotics. Technical Report CS-05-867. University of Southern California, Los Angeles, CA
54. Peters, J. (2007). Machine learning of motor skills for robotics. Ph.D. thesis University of Southern California, Los Angeles, CA, 90089, USA
55. Peters, J., & Schaal, S. (2006). Policy gradient methods for robotics. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2219-2225)
56. Peters, J., Vijayakumar, S., & Schaal, S. (2003). Reinforcement learning for humanoid robotics. In Proceedings of the IEEE-RAS international conference on humanoid robots (HUMANOIDS) (pp. 103-123)
57. Peters, J., Vijayakumar, S., & Schaal, S. (2005a). Natural actor-critic. In Proceedings of the European machine learning conference (pp. 280-291)
58. Peters J., Vijayakumar S., and Schaal S. Natural actor-critic. Proceedings of the European conference on machine learning (2005), Springer 280-291
59. Pongas D., Billard A., and Schaal S. Rapbid synchronization and accurate phase-locking of rhythmic motor primitives. Proceedings of the IEEE international conference on intelligent robots and systems (IROS 2005) Vol. 2005 (2005) 2911-2916
60. Richter S., Aberdeen D., and Yu J. Natural actor-critic for road traffic optimisation. In: Schoelkopf B., Platt J., and Hofmann T. (Eds). Advances in neural information processing systems Vol. 19 (2007), MIT Press, Cambridge, MA p. Online Preproceedings
61. Sadegh, P., & Spall, J. (1997). Optimal random perturbations for stochastic approximation using a simultaneous perturbation gradient approximation. In Proceedings of the american control conference (pp. 3582-3586)
62. Sato M., Nakamura Y., and Ishii S. Reinforcement learning for biped locomotion. Proceedings of the international conference on artificial neural networks (ICANN). Lecture notes in computer science (2002), Springer-Verlag 777-782
63. Schaal S. Learning from demonstration. In: Mozer M., Jordan M., and Petsche T. (Eds). Advances in neural information processing systems (NIPS) Vol. 9 (1997), MIT Press, Cambridge, MA 1040-1046
64. Schaal S., Peters J., Nakanishi J., and Ijspeert A. Learning movement primitives. International symposium on robotics research (ISRR2003). Springer tracts in advanced robotics (2004), Springer, Ciena, Italy 561-572
65. Sciavicco L., and Siciliano B. Modeling and control of robot manipulators (2007), MacGraw-Hill, Heidelberg, Germany
66. Spall J.C. Introduction to stochastic search and optimization: Estimation, simulation, and control (2003), Wiley, Hoboken, NJ
67. Su F., and Gibbs A. On choosing and bounding probability metrics. International Statistical Review 70 3 (2002) 419-435
68. Sutton R.S., McAllester D., Singh S., and Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Solla S.A., Leen T.K., and Mueller K.-R. (Eds). Advances in neural information processing systems (NIPS) (2000), MIT Press, Denver, CO 1057-1063
69. Tedrake, R., Zhang, T.W., & Seung, H.S. (2005). Learning to walk in 20 min. In Proceedings of the Yale workshop on adaptive and learning systems (pp. 10-22). Yale University, New Haven, New Haven, CT
70. Ueno, T., Nakamura, Y., Takuma, T., Shibata, T., Hosoda, K., & Ishii, S. (2006). Fast and stable learning of quasi-passive dynamic walking by an unstable biped robot based on off-policy natural actor-critic. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 5226-5231)
71. Vachenauer P., Rade L., and Westergren B. Springers Mathematische Formeln: Taschenbuch für Ingenieure, Naturwissenschaftler, Informatiker, Wirtschaftswissenschaftler (2000), Springer-Verlag, Heidelberg, Germany
72. Wada Y., and Kawato M. Trajectory formation of arm movement by a neural network with forward and inverse dynamics models. Systems and Computers in Japan 24 (1994) 37-50
73. Weaver, L., & Tao, N. (2001a). The optimal reward baseline for gradient-based reinforcement learning. In: Proceedings of the international conference on uncertainty in artificial intelligence (pp. 538-545). Vol. 17. Seattle, Washington
74. Weaver, L., & Tao, N. (2001b). The variance minimizing constant reward baseline for gradient-based reinforcement learning. Technical Report 30. Australian National University (ANU)
75. Werbos P. Changes in global policy analysis procedures suggested by new methods of optimization. Policy Analysis and Information Systems 3 1 (1979)
76. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 23 (1992)