Model-free reinforcement learning of impedance control in stochastic environments

IEEE Transactions on Autonomous Mental Development

Volumn 4, Issue 4, 2012, Pages 330-341

  Stulp, Freek ^a   Buchli, Jonas ^b   Ellmer, Alice ^c   Mistry, Michael ^a   Theodorou, Evangelos A ^a,c   Schaal, Stefan ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b ISTITUTO ITALIANO DI TECNOLOGIA   (Italy)

^c UNIVERSITY OF WASHINGTON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CO-CONTRACTION; ENVIRONMENT DYNAMICS; ESSENTIAL COMPONENT; FEED-FORWARD; FORCE FIELDS; HUMAN MOVEMENTS; IMPEDANCE CONTROL; MODEL FREE; PHYSICAL INTERACTIONS; POSITION ERRORS; SIMULATED ROBOT; SPECIFIC TASKS; STOCHASTIC ENVIRONMENT; STOCHASTIC FORCES; VARIABLE IMPEDANCE CONTROL;

COMPUTER SIMULATION; ENERGY UTILIZATION; LEARNING ALGORITHMS; REINFORCEMENT LEARNING; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

HUMAN ROBOT INTERACTION;

EID: 84871081166     PISSN: 19430604     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAMD.2012.2205924     Document Type: Article

References (29)

1. J. Buchli, F. Stulp, E. Theodorou, and S. Schaal, "Learning variable impedance control," Int. J. Robot. Res., vol. 30, no. 7, pp. 820-833, 2011.
2. C. Takahashi, D. Scheidt, and R. Reinkensmeyer, "Impedance control and internal model formation when reach in a randomly varying dynamical environment," J. Neurophys., vol. 86, pp. 1047-1051, 2001.
3. S. Dayanidhi, A. Hedberg, I. Hägg, N. Lilja, H. Forssberg, and F. J. Valero-Cuevas, "Dynamic control of fingertip forces: Development in childhood and decline with aging," presented at the Annu. Amer. Soc. Biomech. Conf., 2011.
4. D. W. Franklin, E. Burdet, K. P. Tee, R. Osu, C. M. Chew, T. E. Milner, and M. Kawato, "CNS learns stable, accurate, and efficient movements using a simple algorithm," J. Neurosci., vol. 28, no. 44, pp. 11165-11173, 2008.
5. G. Ganesh, A. Albu-Schäffer, M. Haruno, M. Kawato, and E. Burdet, "Biomimetic motor behavior for simultaneous adaptation of force, impedance and trajectory in interaction tasks," in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2010, pp. 2705-2711.
6. N. Hansen and A. Ostermeier, "Completely derandomized self-adaptation in evolution strategies," Evol. Comput., vol. 9, no. 2, pp. 159-195, 2001.
7. N. Hogan, "Impedance control-An approach to manipulation. I-Theory. II-Implementation. III-Applications," ASME Trans. J. Dynamic Syst., Meas., Contr., vol. 107, pp. 1-24, 1985.
8. A. J. Ijspeert, J. Nakanishi, and S. Schaal, "Movement imitation with nonlinear dynamical systems in humanoid robots," presented at the IEEE Int. Conf. Robot. Autom. (ICRA), 2002.
9. J. Izawa, T. Kondo, and K. Ito, "Biological robot arm motion through reinforcement learning," presented at the IEEE Int. Conf. Robot. Autom. (ICRA), 2002.
10. J. Izawa, T. Kondo, and K. Ito, "Biological arm motion through reinforcement learning," Biol. Cybern., vol. 91, no. 1, pp. 10-22, 2004.
11. J. Izawa, T. Rane, O. Donchin, and R. Shadmehr, "Motor adaptation as a process of reoptimization," J. Neurosci., vol. 278, pp. 2325-2332, 2008.
12. H. J. Kappen, "Path integrals and symmetry breaking for optimal control theory," J. Statist. Mech.: Theory Exper., vol. 2005, no. 11, p. 11011, 2005.
13. B.Kim, J. Park, S. Park, and S. Kang, "Impedance learning for robotic contact tasks using natural actor-critic algorithm," IEEE Trans. Syst., Man, Cybern., Part B: Cybern., vol. 40, no. 2, pp. 433-443, 2009.
14. M. Mistry, E. Theodorou, H. Hoffmann, and S. Schaal, "The dual role of uncertainty in force field learning," presented at the Abstracts 18th Annu. Meeting Neural Contr. Movement (NCM), 2008.
15. D. Mitrovic, S. Klanke, M. Howard, and S. Vijayakumar, "Exploiting sensorimotor stochasticity for learning control of variable impedance actuators," presented at the IEEE-RAS Int. Conf. Human. Robots (Humanoids), 2010.
16. S. Schaal, The SL Simulation and Real-Time Control Software Package Univ. Southern California, 2009, Unpublished.
17. R. Scheidt, B. Dingwell, and F.Mussa-Ivaldi, "Learning to move amid uncertainty," J. Neurophys., vol. 86, pp. 971-985, 2001.
18. L. P. Selen, D. W. Franklin, and D. M. Wolpert, "Impedance control reduces instability that arises from motor noise," J. Neurosci., vol. 7, no. 40, pp. 12606-12616, 2009.
19. R. Shadmehr and F. A. Mussa-Ivaldi, "Adaptive representation of dynamics during learning of a motor task," J. Neurosci., vol. 14, no. 5, pp. 3208-3224, 1994.
20. B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics-Modelling, Planning and Control. Berlin, Germany: Springer-Verlag, 2009.
21. R. F. Stengel, Optimal Control and Estimation. New York: Dover Publications, 1994.
22. F. Stulp, J. Buchli, A. Ellmer, M. Mistry, E. Theodorou, and S. Schaal, "Reinforcement learning of impedance control in stochastic force fields," presented at the Int. Conf. Develop. Learn. (ICDL), 2011.
23. F. Stulp, J. Buchli, E. Theodorou, and S. Schaal, "Reinforcement learning of full-body humanoid motor skills," in Proc. 10th IEEE-RAS Int. Conf. Human. Robots, 2010, pp. 405-410, Best paper finalist.
24. F. Stulp and S. Schaal, "Hierarchical reinforcement learning with motion primitives," presented at the 11th IEEE-RAS Int. Conf. Humanoid Robots, 2011.
25. F. Stulp, E. Theodorou, J. Buchli, and S. Schaal, "Learning to grasp under uncertainty," presented at the IEEE Int. Conf. Robot. Autom. (ICRA), 2011.
26. R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, and D. Precup, "Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction," presented at the 10th Int. Conf. Auton. Agents Multiagent Syst., 2011.
27. E. Theodorou, J. Buchli, and S. Schaal, "A generalized path integral control approach to reinforcement learning," J. Mach. Learn. Res., vol. 11, 2011.
28. S. Vijayakumar, A. D'Souza, and S. Schaal, "Incremental online learning in high dimensions," Neural Comput., vol. 17, no. 12, pp. 2602-2634, 2005.
29. D.M.Wolpert, Z. Ghahramani, and M. I. Jordan, "Are arm trajectories planned in kinematic or dynamic coordinates? An adaptation study," Exp. Brain Res., vol. 103, pp. 460-470, 1995, 10.1007/BF00241505.