Learned parametrized dynamic movement primitives with shared synergies for controlling robotic and musculoskeletal systems

Frontiers in Computational Neuroscience

Volumn , Issue OCT, 2013, Pages

  Rückert, Elmar ^a   d'Avella, Andrea ^b  

^a GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

^b IRCCS FONDAZIONE SANTA LUCIA   (Italy)

Author keywords

Dynamic movement primitives; Motor control; Muscle synergies; Musculoskeletal model; Reinforcement learning

Indexed keywords

BIOLOGICAL MOTOR CONTROL; COMPACT REPRESENTATION; DYNAMIC MOVEMENT PRIMITIVES; MACHINE LEARNING APPROACHES; MOTOR CONTROL; MOTOR SKILL LEARNING; MUSCLE SYNERGIES; MUSCULOSKELETAL MODEL;

COMPUTER SIMULATION; CONTINUOUS TIME SYSTEMS; KNOWLEDGE MANAGEMENT; REINFORCEMENT LEARNING; ROBOTICS; ROBOTS;

MUSCLE;

ARTICLE; COMPUTER SIMULATION; DYNAMIC MOVEMENT PRIMITIVE; FEEDBACK SYSTEM; LEARNING; MECHANICS; MOTOR CONTROL; MUSCLE EXCITATION; MUSCULOSKELETAL SYSTEM; REINFORCEMENT; ROBOTICS; TENDON; WALKING;

EID: 84886246837     PISSN: 16625188     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncom.2013.00138     Document Type: Article

References (49)

1. Alessandro, C., Carbajal, J., and d'Avella, A. (2012). "Synthesis and adaptation of effective motor synergies for the solution of reaching tasks," in From Animals to Animats (SAB 2012), Vol. 7426 Lecture Notes in Computer Science, eds T. Ziemke, C. Balkenius, and J. Hallam (Denmark: Odense), 33-43. doi: 10.1007/978-3-642-33093-3_4
2. Angel, R. W. (1975). Electromyographic patterns during ballistic movement of normal and spastic limbs. Brain Res. 99, 387-392. doi: 10.1016/0006-8993(75) 90042-6
3. Berardelli, A., Hallett, M., Rothwell, J. C., Agostino, R., Manfredi, M., Thompson, P. D., et al. (1996). Single-joint rapid arm movements in normal subjects and in patients with motor disorders. Brain 119, 661-674. doi: 10.1093/brain/119.2.661
4. Berniker, M., Jarc, A., Bizzi, E., and Tresch, M. C. (2009). Simplified and effective motor control based on muscle synergies to exploit mus- culoskeletal dynamics. Proc. Natl. Acad. Sci. U.S.A. 106, 7601-7606. doi: 10.1073/pnas. 0901512106
5. Bizzi, E., Cheung, V., d'Avella, A., Saltiel, P., and Tresch, M. (2008). Combining modules for movement. Brain Res. Rev. 57, 125-133. doi: 10.1016/j.brainresrev. 2007.08.004
6. Chadwick, E., Blana, D., van den Bogert, A., and Kirsch, R. (2009). A real-time, 3-d mus-culoskeletal model for dynamic simulation of arm movements. Biomed. Eng. IEEE Trans. 56, 941-948. doi: 10.1109/TBME.2008. 2005946
7. Chhabra, M., and Jacobs, R. A. (2006). Properties of synergies arising from a theory of optimal motor behavior. Neural Comput. 18, 2320-2342. doi: 10.1162/neco. 2006.18.10.2320
8. Chiovetto, E., Berret, B., and Pozzo, T. (2010). Tri-dimensional and triphasic muscle organization of whole-body pointing movements. Neuroscience 170, 1223-1238. doi: 10.1016/j.neuroscience.2010.07.006
9. Chiovetto, E., d'Avella, A., and Giese, M. (2013). "A unifying framework for the identification of kinematic and electromyographic motor primitives," in Abstracts of Neural Control of Movement Conference (NCM 2013), Puerto Rico.
10. d'Avella, A., and Bizzi, E. (2005). Shared and specific muscle synergies in natural motor behaviors. Proc. Natl. Acad. Sci. U.S.A. 102, 3076-3081. doi: 10.1073/pnas.0500199102
11. d'Avella, A., and Pai, D. K. (2010). Modularity for sensorimotor control: evidence and a new prediction. J. Motor Behav. 42, 361-369. doi: 10.1080/00222895.2010. 526453
12. d'Avella, A., Portone, A., Fernandez, L., and Lacquaniti, F. (2006). Control of fast-reaching movements by muscle synergy combinations. J. Neurosci. 26, 7791-7810. doi: 10.1523/JNEUROSCI.0830-06.2006
13. d'Avella, A., Saltiel, P., and Bizzi, E. (2003). Combinations of muscle synergies in the construction of a natural motor behavior. Nat. Neurosci. 6, 300-308. doi: 10.1038/nn1010
14. d'Avella, A., and Tresch, M. C. (2001). "Modularity in the motor system: decomposition of muscle patterns as combinations of time-varying synergies," in Advances of Neural Information Processing Systems, (NIPS 2001), eds T. G. Dietterich, S. Becker, and Z. Ghahramani (Vancouver, BC: MIT Press), 141-148.
15. Delp, S., Anderson, F., Arnold, A., Loan, P., Habib, A., John, C., et al. (2007). Opensim: open-source software to create and analyze dynamic simulations of movement. Biomed. Eng. IEEE Trans. 54, 1940-1950. doi: 10.1109/TBME. 2007.901024
16. Dominici, N., Ivanenko, Y. P., Cappellini, G., d'Avella, A., Mondì, V., Cicchese, M., et al. (2011). Locomotor primitives in newborn babies and their development. Science 334, 997-999. doi: 10.1126/science.1210617
17. Erdemir, A., McLean, S., Herzog, W., and van den Bogert, A. J. (2007). Model-based estimation of muscle forces exerted during movements. Clin. Biomech. 22, 131-154. doi: 10.1016/j.clinbiomech. 2006.09.005
18. Garner, B., and Pandy, M. (2001). Musculoskeletal model of the upper limb based on the visible human male dataset. Comput. Methods Biomech. Biomed. Eng. 4, 93-126. doi: 10.1080/10255840008 908000
19. Giese, M. A., Mukovskiy, A., Park, A.-N., Omlor, L., and Slotine, J.-J. E. (2009)."Real-time synthesis of body movements based on learned primitives," in Statistical and Geometrical Approaches to Visual Motion Analysis, Vol. 5604 Lecture Notes in Computer Science, eds D. Cremers, B. Rosenhahn, A. L. Yuille, and F. R. Schmidt (Berlin, Heidelberg, Springer), 107-127. doi: 10.1007/978-3-642-03061-1_6
20. Hallett, M., Shahani, B. T., and Young, R. R. (1975). EMG analysis of stereotyped voluntary movements in man. J. Neurol. Neurosurg. Psychiatry 38, 1154-1162. doi: 10.1136/jnnp.38.12.1154
21. Hansen, N., Muller, S., and Koumoutsakos, P. (2003). Reducing the time complexity of the deran-domized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11, 1-18. doi: 10.1162/10636560332 1828970
22. Holzbaur, K. R., Murray, W. M., and Delp, S. L. (2005). A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control. Ann. Biomed. Eng. 33, 829-840. doi: 10.1007/s10439-005-3320-7
23. Ijspeert, A., Nakanishi, J., Pastor, P., Hoffmann, H., and Schaal, S. (2013). Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput. 25, 328-373. doi: 10.1162/ NECO_a_00393
24. Ijspeert, A. J., Nakanishi, J., and Schaal, S. (2002). "Learning attractor landscapes for learning motor primitives," in Advances in Neural Information Processing Systems 15, (NIPS 2002), eds S. Becker, S. Thrun, and K. Obermayer (Vancouver, BC: MIT Press), 1523-1530.
25. Ivanenko, Y. P., Poppele, R. E., and Lacquaniti, F. (2004). Five basic muscle activation patterns account for muscle activity during human locomotion. J. Physiol. 556, 267-282. doi: 10.1113/jphysiol. 2003.057174
26. Kober, J., Oztop, E., and Peters, J. (2010). "Reinforcement learning to adjust robot movements to new situations," in Proceedings of the 2010 Robotics: Science and Systems Conference (RSS 2010), (Zaragoza), 2650-2655.
27. Kober, J., and Peters, J. (2011). Policy search for motor primitives in robotics. Mach. Learn. 84, 171-203. doi: 10.1007/s10994-010-5223-6
28. Li, W., and Todorov, E. (2004). "Iterative linear quadratic regulator design for nonlinear biological movement systems," in Proceedings of the 1st International Conference on Informatics in Control, Automation and Robotics, (ICINCO 2004), (Setúbal), 222-229.
29. Lockhart, D. B., and Ting, L. H. (2007). Optimal sensorimotor transformations for balance. Nat. Neurosci. 10, 1329-1336. doi: 10.1038/ nn1986
30. McKay, J. L., and Ting, L. H. (2012). Optimization of muscle activity for task-level goals predicts com plex changes in limb forces across biomechanical contexts. PLoS Comput. Biol. 8:e1002465. doi: 10.1371/journal.pcbi.1002465
31. Meier, F., Theodorou, E., Stulp, F., and Schaal, S. (2011). "Movement segmentation using a primitive library," in IEEE/RSJ International Conference on Intelligent Robots and Systems, (IROS 2011) (San Francisco, CA), 3407-3412.
32. Mülling, K., Kober, J., Kroemer, O., and Peters, J. (2013). Learning to select and generalize striking movements in robot table tennis. Int. J. Robot. Res. 32, 263-279. doi: 10.1177/0278364912472380
33. Nakanishi, J., Morimoto, J., Endo, G., Cheng, G., Schaal, S., and Kawato, M. (2004). Learning from demonstration and adaptation of biped locomotion. Robot. Auton. Syst. 47, 79-91. doi: 10.1016/j.robot.2004.03.003
34. Neptune, R. R., Clark, D. J., and Kautz, S. A. (2009). Modular control of human walking: a sim ulation study. J. Biomech. 42, 1282-1287. doi: 10.1016/j.jbiomech.2009.03.009
35. Overduin, A., d'Avella, A., Roh, J., and Bizzi, E. (2008). Modulation of muscle synergy recruitment in primate grasping. J. Neurosci. 28, 880-892. doi: 10.1523/ JNEUROSCI.2869-07.2008
36. Pastor, P., Hoffmann, H., Asfour, T., and Schaal, S. (2009). "Learning and generalization of motor skills by learning from demonstration," in International Conference on Robotics and Automation (ICRA 2009), (Kobe). doi: 10.1109/ROBOT.2009. 5152385
37. Peters, J., and Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural Netw. 21, 682-697. doi: 10.1016/j.neunet.2008.02.003
38. R̈ckert, E. A., Neumann, G., Toussaint, M., and Maass, W. (2013). Learned graphical models for probabilistic planning provide a new class of movement primitives. Front. Comput. Neurosci. 6:97. doi: 10.3389/fncom.2012.00097
39. Schaal, S., Peters, J., Nakanishi, J., and Ijspeert, A. J. (2003). "Learning movement primitives," in International Symposium on Robotics Research, (ISRR 2003), (Lucerne), 561-572.
40. Schutte, L., Rodgers, M., Zajac, F., and Glaser, R. (1993). Improving the efficacy of electrical stimulation-induced leg cycle ergometry: an analysis based on a dynamic musculoskeletal model. Rehabil. Eng. IEEE Trans. 1, 109-125. doi: 10.1109/86.242425
41. Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., and Schmidhuber, J. (2010). Parameter-exploring policy gradients. Neural Netw. 23, 551-559. doi: 10.1016/j.neunet.2009.12.004
42. Seth, A., Sherman, M., Reinbolt, J. A., and Delp, S. L. (2011). "Opensim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange," in Procedia International Union of Theoretical and Applied Mathematics (IUTAM 2011), 2, 212-232. doi: 10.1016/j.piutam. 2011.04.021
43. Sutton, R., and Barto, A. (1998). Reinforcement Learning: An Introduction. Boston, MA: MIT Press.
44. Theodorou, E., Buchli, J., and Schaal, S. (2010). "Reinforcement learning of motor skills in high dimensions: a path integral approach," in Robotics and Automation (ICRA), 2010 IEEE International Conference on, (Anchorage, AL), 2397-2403.
45. Westervelt, E. R., Buche, G., and Grizzle, J. W. (2004). Experimental validation of a framework for the design of controllers that induce stable walking in planar bipeds. Int. J. Robot. Res. 23, 559-582. doi: 10.1177/0278364904044410
46. Wierstra, D., Schaul, T., Peters, J., and Schmidhuber, J. (2008). "Episodic reinforcement learning by logistic reward-weighted regression," in Proceedings of the 18th International Conference on Artificial Neural Networks, Part I, (ICANN 2008), (Berlin), 407-416.
47. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229-256. doi: 10.1007/BF00992696
48. Winters, J. M., and Stark, L. (1985). Analysis of fundamental human movement patterns through the use of in-depth antagonistic muscle models. Biomed. Eng. IEEE Trans. BME-32, 826-839. doi: 10.1109/TBME.1985.325498
49. Zajac, F. E. (1989). Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Crit. Rev. Biomed. Eng. 17, 359-411.