Learning and generation of goal-directed arm reaching from scratch

Neural Networks

Volumn 22, Issue 4, 2009, Pages 348-361

  Kambara, Hiroyuki ^a,b   Kim, Kyoungsik ^a,b   Shin, Duk ^c   Sato, Makoto ^a   Koike, Yasuharu ^a,b  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b JAPAN SCIENCE AND TECHNOLOGY AGENCY   (Japan)

^c Rd Management Section   (Japan)

Author keywords

Feedback error learning; Internal model; Reaching; Reinforcement learning; Trajectory planning

Indexed keywords

ARM DYNAMICS; CENTRAL NERVOUS SYSTEMS; COMPUTATIONAL MODELS; CONTROL MECHANISMS; FEEDBACK-ERROR-LEARNING; HUMAN SUBJECTS; INTERNAL MODEL; MOTOR CONTROLS; MOTOR SKILL LEARNING; MOVEMENT TRAJECTORIES; PRIOR KNOWLEDGE; REACHING; REACHING CONTROLS; REACHING MOTIONS; REACHING MOVEMENTS; REACHING TASKS; SAGITTAL PLANES; TARGET POINTS; TRAJECTORY PLANNING; TRIAL-AND-ERROR MANNERS;

EDUCATION; FLIGHT CONTROL SYSTEMS; MOTORS; REINFORCEMENT; REINFORCEMENT LEARNING; TRAJECTORIES;

FEEDBACK;

ARM MOVEMENT; ARTICLE; ARTIFICIAL NEURAL NETWORK; LEARNING ALGORITHM; MATHEMATICAL ANALYSIS; MATHEMATICAL MODEL; MOTOR CONTROL; PRIORITY JOURNAL;

ALGORITHMS; ARM; BIOMECHANICS; COMPUTER SIMULATION; CUES; FEEDBACK; GOALS; HUMANS; JOINTS; LEARNING; MATHEMATICS; MOTOR SKILLS; MOVEMENT; ORIENTATION; PROPRIOCEPTION; PSYCHOMOTOR PERFORMANCE; RANGE OF MOTION, ARTICULAR; SPACE PERCEPTION; VOLITION;

EID: 67349130193     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2008.11.004     Document Type: Article

References (60)

1. Abbeel P., Coates A., Quigley M., and Ng A.Y. An application of reinforcement learning to aerobatic helicopter flight. In: Schölkopf B., Platt J., and Hofmann T. (Eds). Advaces in neural information processing system Vol. 19 (2007), MIT Press, Cambridge, MA 1-8
2. Abend W., Bizzi E., and Morasso P. Human arm trajectory formation. Brain 105 2 (1982) 331-348
3. Albus J. A theory of cerebellar function. Mathematical Bioscience 10 1-2 (1971) 25-61
4. Andersson G., and Armstrong D.M. Complex spikes in purkinje cells in the lateral vermis (b zone) of the cat cerebellum during locomotion. The Journal of Physiology 385 1 (1987) 107-134
5. Atkeson C.G., and Hollerback M. Kinematic features of unrestrained vertical arm movements. The Journal of Neuroscience 5 9 (1985) 2318-2330
6. Bard C., Turrell Y., Fleury M., Teasdale N., Lamarre L., and Martin O. Deafferentation and pointing with visual double-step perturbations. Experimental Brain Research 125 4 (1999) 410-416
7. Barto A.G. Adaptive critics and the basal ganglia. In: Houk C.J., Davis J.L., and G. B.D. (Eds). Models of information processing in the basal ganglia (1995), MIT Press, Cambridge, MA 215-232 (Chapter 11)
8. Bugmann G. Normalized gaussian radial basis function networks. Neurocomputing 20 1-3 (1998) 97-110
9. Coltz J.D., Johnson M.T.V., and Ebner T.J. Population code for tracking velocity based on cerebellar purkinje cell simple spike firing in monkeys. Neuroscience Letters 296 1 (2000) 1-4
10. De Wolf S., Slijper H., and Latash L.M. Anticipatory postural adjustments during self-paced and reaction-time movements. Experimental Brain Research 121 1 (1998) 7-19
11. Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex. Neural Networks 12 7-8 (1999) 961-974
12. Doya K. Reinforcement learning in continuous time and space. Neural Computation 12 1 (2000) 219-245
13. Feldman A.G. Functional tuning of the nervous system during control of movement or maintenace of a steady posture-3. Mechanographic analysis of the execution by man of the simplest motor tasks. Biophysics 11 (1966) 766-775
14. Flanagan J., Vetter P., Johansson R.S., and Wolpert D.M. Prediction precedes control in motor learning. Current Biology 13 2 (2003) 146-150
15. Flanagan J.R., and Wing A.M. The role of internal models in motion planning and control: Evidence from grip force adjustments during movements of hand-held loads. The Journal of Neuroscience 17 4 (1997) 1519-1528
16. Flash T. The control of hand equilibrium trajectories in multi-joint arm movements. Biological Cybernetics 57 4-5 (1987) 257-274
17. Flash T., and Hogan N. The coordination of arm movements: An experimentally confirmed mathmatical model. The Journal of Neuroscience 5 7 (1985) 1688-1703
18. Fu Q.-G., Flament D., Coltz J.D., and Ebner T.J. Relationship of cerebellar purkinje cell simple spike discharge to movement kinematics in the monkey. Journal of Neurophysiology 78 1 (1997) 478-491
19. Gellman R., Gibson A.R., and Houk J.C. Inferior olivary neurons in the awake cat: Detection of contact and passive body displacement. Journal of Neurophysiology 54 1 (1985) 40-60
20. Gribble P.L., and Ostry D.J. Compensation for interaction torques during single- and multijoints limb movement. Journal of Neurophysiology 82 5 (1999) 2310-2326
21. Gribble P.L., Ostry D.J., Sanguineti V., and Laboissiere R. Are complex control signals required for human arm movement?. Journal of Neurophysiology 79 3 (1998) 1409-1424
22. Hamilton A.F.C., and Wolpert D.M. Controlling the statistics of action: Obstacle avoidance. Journal of Neurophysiology 87 5 (2002) 2434-2440
23. Harris C.M., and Wolpert D.M. Signal-dependent noise determines motor planning. Nature 394 6695 (1998) 780-794
24. Haruno M., and Kawato M. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fmri examination in stimulus-action-reward association learning. Neural Networks 19 8 (2006) 1242-1254
25. Hogan N. An organizing principle for a class of voluntary movements. The Journal of Neuroscience 4 11 (1984) 2745-2754
26. Ito M. Long-term depression. Annual Review of Neuroscience 12 (1989) 85-102
27. Izawa J., Kondo T., and Ito K. Biological arm motion through reinforcement learning. Biological Cybernetics 91 1 (2004) 10-22
28. Joel D., Niv Y., and Ruppin E. Actor-critic models of the basal gaglia: New anatomical and computational perspectives. Neural Networks 15 4-6 (2002) 535-547
29. Julier S., Uhlmann J., and Durrant-Whyte H.F. A new method for the nonlinear transformation of means and covariances in filtering and estimators. IEEE Transaction on Automatic Control 45 3 (2004) 477-482
30. Kambara, H., Kim, J., Sato, M., & Koike, Y. (2004). Learning arm's posture control using reinforcement learning and feedback-error-learning. In Proceedings of the 26th annual international conferenece of the IEEE engineering in medicine and biology society (pp. 486-489)
31. Katayama M., and Kawato M. Learning trajectory and force control of an aritificial muscle arm by parallel-hierachical neural network model. In: Lippmann R.P., Moody J.E., and Touretzky D.S. (Eds). Advaces in neural information processing system: Vol. 3 (1991), Morgan Kauffmann, San Mateo, CA 436-442 (Chapter 8)
32. Kawato M. Internal models for motor control and trajectory planning. Current Opinion in Neurobiology 9 6 (1999) 718-727
33. Kawato M., Furukawa K., and Suzuki R. A hierachical neural-network model for control and learning of voluntary movement. Biological Cybernetics 53 2 (1987) 56-66
34. Kawato M., and Gomi H. A computational model of four regions of the cerebellum based on feedback-error learning. Biological Cybernetics 68 2 (1992) 95-103
35. Kelly R.M., and Strick P.L. Cerebellar loops with motor cortex and prefrontal cortex of a nonhuman primate. The Journal of Neuroscience 23 23 (2003) 8432-8444
36. Koike Y., and Kawato M. Estimation of dynamic joint torques and trajectory formation from surface electromyography signals using a neural network model. Biological Cybernectics 73 4 (1995) 291-300
37. Konczak J., and Dichgans J. The development toward stereotypic arm kinematics during reaching in the first 3 years of life. Experimental Brain Research 117 2 (1997) 346-354
38. Mannard A., and Stein R.B. Determination of the frequency response of isometric solues muscle in the cat using random nerve stimulation. The Journal of Physiology 229 2 (1973) 275-296
39. Marr D. A theory of cerebellar cortex. The Journal of Physiology 202 2 (1969) 437-470
40. Miall R.C., Keating J.G., Malkmus M., and Thach W.T. Simple spike activity predicts occurrence of complex spikes in cerebellar purkinje cells. Nature Neuroscience 1 1 (1998) 13-15
41. Miall R.C., and Wolpert D.M. Is the cerebellum a smith predictor?. Journal of Motor Behavior 25 3 (1993) 203-216
42. Miall R.C., and Wolpert D.M. Forward models for physiological motor control. Neural Networks 9 8 (1996) 1265-1279
43. Miyamoto H., Kawato M., Setoyoama T., and Suzuki R. Feedback-error-learning neural network for trajectory control of robotic manipulator. Neural Networks 1 3 (1988) 251-265
44. Miyamoto H., Nakano E., Wolpert D.M., and Kawato M. Tops (task optimization in the presence of signal-dependent noise) model. Systems and Computers in Japan 35 11 (2004) 48-58
45. Nakano E., Imamizu H., Osu R., Uno Y., Gomi H., Yoshioka T., et al. Quantative examinations of internal representations for arm trajectory planning: Minimum commanded torque change model. Journal of Neurophysiology 81 5 (1999) 2140-2155
46. Ozkaya N., and Nordin M. Fundamentals of biomechanics: Equilibrium, motion, and deformation. 2nd ed. (1999), Springer, New York, NY
47. Sakaguchi Y., and Ikeda S. Motor planning and sparse motor command representation. Neurocomputing 70 10-12 (2007) 1748-1752
48. Schultz W., Dayan P., and Montague P.R. A neural substrate of prediction and reward. Science 273 5306 (1997) 1593-1599
49. Schwighofer N., Spoelstra J., Arbib M.A., and Kawato M. Role of the cerebellum in reaching movements in humans. ii. A neural model of the intermediate cerebellum. European Journal of Neuroscience 10 1 (1998) 95-105
50. Smith M.A., Brandt J., and Shadmehr R. Motor disorder in huntington's disease begins as a dysfunction in error feedback control. Nature 403 6769 (2000) 544-549
51. Stroeve S. Learning combined feedback and feedforward control of a musculoskeletal system. Biological Cybernetics 75 1 (1996) 73-83
52. Suri R.E. Td models of reward predictive responses in dopamine neurons. Neural Networks 15 4-6 (2002) 523-533
53. Sutton R., and Barto A. Reinforcement learning: An introduction (1998), MIT Press, Cambridge, MA
54. Todorov E., and Jordan M. Optimal feedback control as a theory of motor coordination. Nature Neuroscience 5 11 (2002) 1226-1235
55. Uno Y., Kawato M., and Suzuki R. Formation and control of optimal trajectory in human multijoint arm movement. Biological Cybernetics 61 2 (1989) 89-101
56. Wada Y., and Kawato M. A neural network model for arm trajectory formation using forward and inverse dynamics models. Neural Networks 6 7 (1993) 919-932
57. Wolpert D.M., Ghahramani Z., and Jordan M.I. An internal model for sensorimotor integration. Science 269 5232 (1995) 1880-1882
58. Wolpert D.M., Miall R.C., and Kawato M. Internal models in the cerebellum. Trends in Cognitive Sciences 2 9 (1998) 338-347
59. Yamamoto K., Kawato M., Kotosaka S., and Kitazawa S. Encoding of movement dynamics by purkinje cell simple spike activity during fast arm movements under resistive and assistive force fields. Journal of Neurophysiology 97 2 (2007) 1588-1599
60. Zaal F.T.J.M., Daigle K., Gottlieb L., and Thelen E. An unlearned principle for controlling natural movement. Journal of Neurophysiology 82 1 (1999) 255-259