Policy search for motor primitives in robotics

Machine Learning

Volumn 84, Issue 1-2, 2011, Pages 171-203

  Kober, Jens ^a   Peters, Jan ^a  

^a MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

Author keywords

Episodic reinforcement learning; Motor control; Motor primitives; Policy learning

Indexed keywords

ACTOR CRITIC; BENCH-MARK PROBLEMS; EPISODIC REINFORCEMENT LEARNING; EXPECTATION MAXIMIZATION; HIGH-DIMENSIONAL; HUMANOID ROBOTICS; IMITATION LEARNING; MOTOR CONTROL; MOTOR LEARNING; MOTOR PRIMITIVES; MOTOR SKILLS; POLICY GRADIENT; POLICY GRADIENT METHODS; POLICY SEARCH; REAL ROBOT; REINFORCEMENT LEARNING METHOD; ROBOT ARMS;

ANTHROPOMORPHIC ROBOTS; DYNAMICAL SYSTEMS; GRADIENT METHODS; REINFORCEMENT LEARNING; ROBOTICS;

ALGORITHMS;

EID: 78049390740     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-010-5223-6     Document Type: Article

References (61)

1. C. Andrieu N. de Freitas A. Doucet M. I. Jordan 2003 An introduction to MCMC for machine learning Machine Learning 50 1 5 43 1033.68081 10.1023/A:1020281327116
2. Atkeson, C. G. (1994). Using local trajectory optimizers to speed up global optimization in dynamic programming. In Advances in neural information processing systems (Vol. 6, pp. 503-521), Denver, CO, USA.
3. Attias, H. (2003). Planning by probabilistic inference. In Proceedings of the ninth international workshop on artificial intelligence and statistics (AISTATS), Key West, FL, USA.
4. Bagnell, J., & Schneider, J. (2003). Covariant policy search. In Proceedings of the international joint conference on artificial intelligence (IJCAI) (pp. 1019-1024), Acapulco, Mexico.
5. Bagnell, J., Kadade, S., Ng, A., & Schneider, J. (2004). Policy search by dynamic programming. In Advances in neural information processing systems (Vol. 16), Vancouver, BC, CA.
6. J. Binder D. Koller S. Russell K. Kanazawa 1997 Adaptive probabilistic networks with hidden variables Machine Learning 29 2-3 213 244 0892.68079 10.1023/A:1007421730016 (Pubitemid 127510039)
7. Chiappa, S., Kober, J., & Peters, J. (2009). Using Bayesian dynamical systems for motion template libraries. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems (Vol. 21, pp. 297-304).
8. DARPA (2010a). Learning locomotion (L2). http://www.darpa.mil/ipto/ programs/ll/ll.asp.
9. DARPA (2010b). Learning applied to ground robotics (LAGR). http://www.darpa.mil/ipto/programs/lagr/lagr.asp.
10. DARPA (2010c). Autonomous robot manipulation (ARM). http://www.darpa.mil/ ipto/programs/arm/arm.asp.
11. P. Dayan G. E. Hinton 1997 Using expectation-maximization for reinforcement learning Neural Computation 9 2 271 278 0876.68090 10.1162/neco.1997.9.2.271 (Pubitemid 127635391)
12. A. P. Dempster N. M. Laird D. B. Rubin 1977 Maximum likelihood from incomplete data via the EM algorithm Journal of the Royal Statistical Society, Series B (Methodological) 39 1 38 501537 0364.62022
13. El-Fakdi, A., Carreras, M., & Ridao, P. (2006). Towards direct policy search reinforcement learning for robot control. In Proceedings of the IEEE/RSJ 2006 international conference on intelligent robots and systems (IROS), Beijing, China.
14. Fantoni, I., & Lozano, R. (2001). Non-linear control for underactuated mechanical systems. New York: Springer.
15. F. Guenter M. Hersch S. Calinon A. Billard 2007 Reinforcement learning for imitating constrained reaching movements Advanced Robotics, Special Issue on Imitative Robots 21 13 1521 1544 (Pubitemid 47529845)
16. V. Gullapalli J. Franklin H. Benbrahim 1994 Acquiring robot skills via reinforcement learning IEEE Control Systems Journal, Special Issue on Robotics: Capturing Natural Motion 4 1 13 24
17. Hoffman, M., Doucet, A., de Freitas, N., & Jasra, A. (2007). Bayesian policy learning with trans-dimensional MCMC. In Advances in neural information processing systems (Vol. 20), Vancouver, BC, CA.
18. Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2002). Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings of IEEE international conference on robotics and automation (ICRA) (pp. 1398-1403), Washington, DC. (Pubitemid 34916443)
19. Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2003). Learning attractor landscapes for learning motor primitives. In Advances in neural information processing systems (Vol. 15, pp. 1547-1554), Vancouver, BC, CA.
20. Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994). Convergence of stochastic iterative dynamic programming algorithms. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems (Vol. 6, pp. 703-710). San Mateo: Morgan Kaufmann.
21. Kirk, D. E. (1970). Optimal control theory. Englewood Cliffs: Prentice-Hall.
22. Kober, J., & Peters, J. (2009a). Learning motor primitives for robotics. In Proceedings of IEEE international conference on robotics and automation (ICRA) (pp. 2112-2118).
23. Kober, J., & Peters, J. (2009b). Policy search for motor primitives in robotics. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems (Vol. 21, pp. 849-856).
24. Kober, J., Mohler, B., & Peters, J. (2008). Learning perceptual coupling for motor primitives. In Proceedings of the IEEE/RSJ 2008 international conference on intelligent robots and systems (IROS) (pp. 834-839), Nice, France.
25. Kormushev, P., Calinon, S., & Caldwell, D. G. (2010). Robot motor skill coordination with em-based reinforcement learning. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS).
26. Kwee, I., Hutter, M., & Schmidhuber, J. (2001). Gradient-based reinforcement planning in policy-search methods. In M. A. Wiering (Ed.), Cognitieve Kunstmatige Intelligentie: Vol. 27. Proceedings of the 5th European workshop on reinforcement learning (EWRL) (pp. 27-29), Lugano. Manno: Onderwijsinsituut CKI, Utrecht University.
27. Lawrence, G., Cowan, N., & Russell, S. (2003). Efficient gradient estimation for motor control learning. In Proceedings of the international conference on uncertainty in artificial intelligence (UAI) (pp. 354-361), Acapulco, Mexico.
28. Martín, H. J. A., de Lope, J., & Maravall, D. (2009). The knn-td reinforcement learning algorithm. In Proceedings of the 3rd international work-conference on the interplay between natural and artificial computation (IWINAC) (pp. 305-314). Berlin: Springer.
29. McLachan, G. J., & Krishnan, T. (1997). Wiley series in probability and statistics. The EM algorithm and extensions. New York: Wiley.
30. H. Miyamoto S. Schaal F. Gandolfo H. Gomi Y. Koike R. Osu E. Nakano Y. Wada M. Kawato 1996 A Kendama learning robot based on bi-directional theory Neural Networks 9 8 1281 1302 10.1016/S0893-6080(96)00043-3 (Pubitemid 26413052)
31. Ng, A. Y., & Jordan, M. (2000). Pegasus: A policy search method for large mdps and pomdps. In Proceedings of the international conference on uncertainty in artificial intelligence (UAI) (pp. 406-415), Palo Alto, CA.
32. Ng, A. Y., Kim, H. J., Jordan, M. I., & Sastry, S. (2004). Inverted autonomous helicopter flight via reinforcement learning. In Proceedings of the international symposium on experimental robotics (ISER). Cambridge: MIT Press.
33. Park, D. H., Hoffmann, H., Pastor, P., & Schaal, S. (2008). Movement reproduction and obstacle avoidance with dynamic movement primitives and potential fields. In IEEE international conference on humanoid robots (HUMANOIDS) (pp. 91-98).
34. PASCAL2 (2010). Challenges. http://pascallin2.ecs.soton.ac.uk/Challenges/ .
35. Peshkin, L. (2001). Reinforcement learning by policy search. PhD thesis, Brown University, Providence, RI.
36. Peters, J. (2007). Machine learning of motor skills for robotics. PhD thesis, University of Southern California, Los Angeles, CA, 90089, USA.
37. Peters, J., & Schaal, S. (2006). Policy gradient methods for robotics. In Proceedings of the IEEE/RSJ 2006 international conference on intelligent robots and systems (IROS) (pp. 2219-2225), Beijing, China. (Pubitemid 46928224)
38. Peters, J., & Schaal, S. (2007). Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the international conference on machine learning (ICML), Corvallis, OR, USA.
39. Peters, J., Vijayakumar, S., & Schaal, S. (2003). Reinforcement learning for humanoid robotics. In Proceedings of the IEEE-RAS international conference on humanoid robots (HUMANOIDS) (pp. 103-123), Karlsruhe, Germany.
40. Peters, J., Vijayakumar, S., & Schaal, S. (2005). Natural actor-critic. In Proceedings of the European conference on machine learning (ECML) (pp. 280-291), Porto, Portugal.
41. Rückstie, T., Felder, M., & Schmidhuber, J. (2008). State-dependent exploration for policy gradient methods. In Proceedings of the European conference on machine learning (ECML) (pp. 234-249), Antwerp, Belgium.
42. S. Sato T. Sakaguchi Y. Masutani F. Miyazaki 1993 Mastering of a task with interaction between a robot and its environment: "kendama" task Transactions of the Japan Society of Mechanical Engineers C 59 558 487 493
43. S. Schaal C. G. Atkeson S. Vijayakumar 2002 Scalable techniques from nonparameteric statistics for real-time robot learning Applied Intelligence 17 1 49 60 1003.68169 10.1023/A:1015727715131 (Pubitemid 34789897)
44. Schaal, S., Peters, J., Nakanishi, J., & Ijspeert, A. J. (2003). Control, planning, learning, and imitation with dynamic movement primitives. In Proceedings of the workshop on bilateral paradigms on humans and humanoids, IEEE international conference on intelligent robots and systems (IROS), Las Vegas, NV, October 27-31, 2003.
45. S. Schaal P. Mohajerian A. J. Ijspeert 2007 Dynamics systems vs. optimal control-a unifying view Progress in Brain Research 165 1 425 445 10.1016/S0079-6123(06)65027-9 (Pubitemid 47513886)
46. F. Sehnke C. Osendorfer T. Rückstie A. Graves J. Peters J. Schmidhuber 2010 Parameter-exploring policy gradients Neural Networks 21 4 551 559 10.1016/j.neunet.2009.12.004
47. Shone, T., Krudysz, G., & Brown, K. (2000). Dynamic manipulation of Kendama (Tech. rep.). Rensselaer Polytechnic Institute.
48. Strens, M., & Moore, A. (2001). Direct policy search using paired statistical tests. In Proceedings of the 18th international conference on machine learning (ICML).
49. Sumners, C. (1997). Toys in space: exploring science with the astronauts. New York: McGraw-Hill.
50. Sutton, R., & Barto, A. (1998). Reinforcement learning. Cambridge: MIT Press.
51. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the international machine learning conference (pp. 9-44).
52. Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems (NIPS) (Vol. 13, pp. 1057-1063), Denver, CO, USA.
53. K. Takenaka 1984 Dynamical control of manipulator with vision: "cup and ball" game demonstrated by robot Transactions of the Japan Society of Mechanical Engineers C 50 458 2046 2053 (Pubitemid 16635650)
54. Taylor, M. E., Whiteson, S., & Stone, P. (2007). Transfer via inter-task mappings in policy search reinforcement learning. In Proceedings of the sixth international joint conference on autonomous agents and multiagent systems (AAMAS).
55. Tedrake, R., Zhang, T. W., & Seung, H. S. (2004). Stochastic policy gradient reinforcement learning on a simple 3d biped. In Proceedings of the IEEE 2004 international conference on intelligent robots and systems (IROS) (pp. 2849-2854). (Pubitemid 40276027)
56. Theodorou, E. A., Buchli, J., & Schaal, S. (2010). Reinforcement learning of motor skills in high dimensions: a path integral approach. In Proceedings of IEEE international conference on robotics and automation (ICRA) (pp. 2397-2403).
57. Toussaint, M., & Goerick, C. (2007). Probabilistic inference for structured planning in robotics. In Proceedings of the IEEE/RSJ 2007 international conference on intelligent robots and systems (IROS), San Diego, CA, USA.
58. Van Der Maaten, L., Postma, E., & Van Den Herik, H. (2007). Dimensionality reduction: a comparative review. Preprint.
59. N. Vlassis M. Toussaint G. Kontes S. Piperidis 2009 Learning model-free robot control by a Monte Carlo EM algorithm Autonomous Robots 27 2 123 130 10.1007/s10514-009-9132-0
60. R. J. Williams 1992 Simple statistical gradient-following algorithms for connectionist reinforcement learning Machine Learning 8 229 256 0772.68076
61. Wulf, G. (2007). Attention and motor skill learning. Champaign: Human Kinetics.