Exploiting Redundancy for Flexible Behavior: Unsupervised Learning in a Modular Sensorimotor Control Architecture

Psychological Review

Volumn 114, Issue 4, 2007, Pages 1015-1046

  Butz, Martin V ^a   Herbort, Oliver ^a   Hoffmann, Joachim ^a  

^a UNIVERSITY OF WÜRZBURG   (Germany)

Author keywords

computational model; population encoding; redundancy resolution; sensorimotor control; unsupervised motor learning

Indexed keywords

BIOMECHANICS; HUMAN; LEARNING; MOTOR PERFORMANCE; PSYCHOLOGICAL MODEL; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT;

BIOMECHANICAL PHENOMENA; HUMANS; LEARNING; MODELS, PSYCHOLOGICAL; MOTOR SKILLS; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY);

EID: 34249668981     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/0033-295X.114.4.1015     Document Type: Article

References (130)

1. Abbott L.F., and Nelson S.B. Synaptic plasticity: Taming the beast. Nature Neuroscience 3 (2000) 1178-1183
2. Aflalo T.N., and Graziano M.S.A. Possible origins of the complex topographic organization of motor cortex: Reduction of a multidimensional space onto a two-dimensional array. Journal of Neuroscience 26 (2006) 6288-6297
3. Atkeson C.G., Moore A.W., and Schaal S. Locally weighted learning. Artificial Intelligence Review 11 (1997) 11-73
4. Baraduc P., Guigon E., and Burnod Y. Where does the population vector of motor cortical cells point during reaching movements?. In: Kearns M., Solla S., and Cohn D. (Eds). Advances in neural information processing systems 11 (1999), MIT Press, Cambridge, MA 83-89
5. Baraduc P., Guigon E., and Burnod Y. Recoding arm position to learn visuomotor transformation. Cerebral Cortex 11 (2001) 906-917
6. Barto A.G., Bradtke S.J., and Singh S.P. Learning to act using real-time dynamic programming. Artificial Intelligence 72 (1995) 81-138
7. Barto A.G., Fagg A.H., Sitkoff N., and Houk J.C. A cerebellar model of timing and prediction in the control of reaching. Neural Computation 11 (1999) 565-594
8. Bastian A., Schöner G., and Riehle A. Preshaping and continuous evolution of motor cortical representations during movement preparation. European Journal of Neuroscience 18 (2003) 2047-2058
9. Bellman R. Dynamic programming (1957), Princeton University Press, Princeton, NJ
10. Bernstein N.A. The co-ordination and regulation of movements (1967), Pergamon Press, Oxford, England
11. Berthier N.E. Learning to reach: A mathematical model. Developmental Psychology 32 (1996) 811-823
12. Berthier N.E., Rosenstein M.T., and Barto A.G. Approximate optimal control as a model for motor learning. Psychological Review 112 (2005) 329-346
13. Berthier N.E., Singh S.P., Barto A.G., and Houk J.C. A cortico-cerebellar model that learns to generate distributed motor commands to control a kinematic arm. Advances in Neural Information Processing Systems 4 (1992) 611-618
14. Berthier N.E., Singh S.P., Barto A.G., and Houk J.C. Distributed representation of limb motor programs in arrays of adjustable pattern generators. Journal of Cognitive Neuroscience 5 (1993) 56-78
15. Bock O., and Arnold K. Motor control prior to movement onset: Preparatory mechanisms for pointing at visual targets. Experimental Brain Research 90 (1992) 209-216
16. Bullock D., Cisek P., and Grossberg S. Cortical networks for control of voluntary arm movements under variable force conditions. Cerebral Cortex 8 (1998) 48-62
17. Bullock D., Grossberg S., and Guenther F.H. A self-organizing neural model of motor equivalent reaching and tool use by a multijoint arm. Journal of Cognitive Neuroscience 5 (1993) 408-435
18. Butz M.V. Biasing exploration in an anticipatory learning classifier system. In: Lanzi P.L., Stolzmann W., and Wilson S.W. (Eds). Advances in learning classifier systems: Fourth International Workshop, 2001 (2002), Springer-Verlag, Berlin, Germany 3-22
19. Butz M.V. Kernel-based, ellipsoidal conditions in the real-valued XCS classifier system. GECCO 2005: Genetic and Evolutionary Computation Conference (2005) 1835-1842
20. Butz M.V., Lanzi P.L., and Wilson S.W. Hyper-ellipsoidal conditions in XCS: Rotation, linear approximation, and solution structure. GECCO 2006: Genetic and Evolutionary Computation Conference (2006) 1457-1464
21. Connolly C.I., and Grupen R.A. The applications of harmonic functions to robotics. Journal of Robotic Systems 10 (1993) 931-946
22. Craig J.J. Introduction to robotics: Mechanics and control. 3rd ed. (2005), Pearson Education, Upper Saddle River, NJ
23. Cruse H. The evolution of cognition-A hypothesis. Cognitve Science 27 (2003) 135-155
24. Cruse H., Brüwer M., and Dean J. Control of three- and four-joint arm movement: Strategies for a manipulator with redundant degrees of freedom. Journal of Motor Behavior 25 (1993) 131-139
25. Cruse H., and Steinkühler U. Solution of the direct and inverse kinematic problems by a common algorithm based on the mean of multiple computations. Biological Cybernetics 69 (1993) 341-351
26. Cruse, H., Steinkühler, U., & Burkamp, C. (1998). MMC-A recurrent neural network which can be used as manipulable body model. From Animals to Animats 5: Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior, 381-389.
27. Dehaene S., Naccache L., Le Clec'H G., Koechlin E., Mueller M., Dehaene-Lambertz G., et al. Imaging unconscious semantic priming. Nature 395 (1998, October 8) 597-600
28. Deneve S., and Pouget A. Bayesian multisensory integration and cross-modal spatial links. Journal of Physiology-Paris 98 (2004) 249-258
29. Doya K. What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex?. Neural Networks 12 7-8 (1999) 961-974
30. Doya K. Complementary roles of basal ganglia and cerebellum in learning and motor control. Current Opinion in Neurobiology 10 (2000) 732-739
31. D'Souza A., Vijayakumar S., and Schaal S. Learning inverse kinematics. IEEE International Conference on Intelligent Robots and Systems (IROS 2001) 1 (2001) 298-303
32. Elsner B., and Hommel B. Effect anticipation and action control. Journal of Experimental Psychology: Human Perception and Performance 27 (2001) 229-240
33. Engelbrecht S.E. Minimum principles in motor control. Journal of Mathematical Psychology 45 (2001) 497-542
34. Erlhagen W., and Schöner G. Dynamic field theory of movement preparation. Psychological Review 109 (2002) 545-572
35. Fischer M.H., Rosenbau D.A., and Vaughan J. Speed and sequential effects in reaching. Journal of Experimental Psychology: Human Perception and Performance 23 (1997) 404-428
36. Flament D., Shapiro M.B., Kempf T., and Corcos D.M. Time course and temporal order of changes in movement kinematics during learning of fast and accurate elbow flexions. Experimental Brain Research 129 (1999) 441-450
37. Flash T., and Hogan N. The coordination of arm movements: An experimentally confirmed mathematical model. Journal of Neuroscience 5 (1985) 1688-1703
38. Flash T., and Sejnowski T.J. Computational approaches to motor control. Current Opinion in Neurobiology 11 (2001) 655-662
39. Fritzke B. A growing neural gas network learns topologies. In: Tesauro G., Touretzky D.S., and Leen T.K. (Eds). Advances in neural information processing systems 7 (1995), MIT Press, Cambridge, MA 625-632
40. Georgopoulus A.P. Current issues in directional motor control. Trends in Neuroscience 18 (1995) 506-510
41. Georgopoulos A.P., Caminiti R., Kalaska J.F., and Massey J.T. Spatial coding of movement: A hypothesis concerning the coding of movement direction by motor cortical populations. Experimental Brain Research 49 (1983) 327-336
42. Gottlieb G.L., Corcos D.M., Jaric S., and Agarwal G.C. Practice improves even the simplest movements. Experimental Brain Research 73 (1988) 436-440
43. Graziano M. The organization of behavioral repertoire in motor cortex. Annual Review Neuroscience 29 (2006) 105-134
44. Grush R. The emulation theory of representation: Motor control, imagery, and perception. Behavioral and Brain Sciences 27 (2004) 377-396
45. Harris C.M., and Wolpert D.M. Signal-dependent noise determines motor planning. Nature 394 (1998, August 20) 780-784
46. Haruno M., Wolpert D.M., and Kawato M. MOSAIC model for sensorimotor learning and control. Neural Computation 13 (2001) 2201-2220
47. Haruno M., Wolpert D.M., and Kawato M. Hierarchical MOSAIC for movement generation. In: Ono T., Matsumoto G., Llinas R.R., Berthoz A., Norgren R., Nishijo H., and Tamura R. (Eds). Excepta Medica International Congress Series 1250 (2003), Elsevier, Amsterdam 575-590
48. Haykin S. Neural networks: A comprehensive foundation. 2nd ed. (1999), Prentice Hall, Upper Saddle River, NJ
49. Haykin S. Adaptive filter theory. 4th ed. (2002), Prentice Hall, Upper Saddle River, NJ
50. Hebb D.O. The organization of behavior (1949), Wiley, New York
51. Herbart J.F. Psychologie als Wissenschaft neu gegründet auf Erfahrung, Metaphysik und Mathematik. Zweiter, analytischer Teil [Psychology as a newly founded science based on research, metaphysics, and mathematics. Second, analytical part] (1825), August Wilhem Unzer, Königsberg, Germany
52. Herbort, O. (2005). A computational model of hierarchical anticipatory control for reaching. Unpublished Diplomarbeit thesis, University of Würzburg.
53. Herbort O., Butz M.V., and Hoffmann J. Towards an adaptive hierarchical anticipatory behavioral control system. In: Castelfranchi C., Balkenius C., Butz M.V., and Ortony A. (Eds). From reactive to anticipatory cognitive embodied systems: Papers from the AAAI fall symposium (2005), AAAI Press, Menlo Park, CA 83-90
54. Herbort, O., Butz, M. V., & Hoffmann, J. (2005b). Towards the advantages of hierarchical anticipatory behavioral control. In K. Opwis & I.-K. Penner (Eds.), Proceedings of the KogWis05. The German Cognitive Science Conference (pp. 77-82). Basel, Germany: Schwabe.
55. Hoffmann J. Vorhersage und Erkenntnis: Die Funktion von Antizipationen in der menschlichen Verhaltenssteuerung und Wahrnehmung. [Anticipation and cognition: The function of anticipations in human behavioral control and perception.] (1993), Hogrefe, Göttingen, Germany
56. Hoffmann J., Stöcker C., and Kunde W. Anticipatory control of actions. International Journal of Sport and Exercise Psychology 2 (2004) 346-361
57. Holmes N.P., and Spence C. The body schema and multisensory representation(s) of peripersonal space. Cognitive Processing 5 (2004) 94-105
58. Hubel D.H. Evolution of ideas on the primary visual cortex, 1955-1978: A biased historical account. In: Lindsten J. (Ed). Nobel lectures in physiology or medicine, 1981-1990 (1993), World Scientific Publishing, Singapore 24-56
59. Ito M., Noda K., Hoshino Y., and Tani J. Dynamic and interactive generation of object handling behaviors by a small humanoid robot using a dynamic neural network model. Neural Networks 19 (2006) 323-337
60. Jackson A., Mavoori J., and Fetz E.E. Long-term motor cortex plasticity induced by an electronic neural implant. Nature 444 (2006, November 2) 56-60
61. James W. The principles of psychology (1890), Holt, New York
62. Jaric S., Corcos D.M., and Latash M.L. Effects of practice on final position reproduction. Experimental Brain Research 91 (1992) 129-134
63. Jordan M.I., and Rumelhart D.E. Forward models: Supervised learning with a distal teacher. Cognitive Science 16 (1992) 307-354
64. Körding K.P., Ku S.P., and Wolpert D.M. Bayesian integration in force estimation. Journal of Neurophysiology 92 (2004) 3161-3165
65. Körding K.P., and Wolpert D.M. Bayesian integration in sensorimotor learning. Nature 427 (2004, January 15) 244-247
66. Karniel A., and Inbar G.F. A model for learning human reaching movements. Biological Cybernetics 77 (1997) 173-183
67. Kawato M. Feedback-error-learning neural network for supervised learning. In: Eckmiller R. (Ed). Advanced neural computers (1990), North-Holland, Amsterdam 365-372
68. Kawato M. Internal models for motor control and trajectory planning. Current Opinion in Neurobiology 9 (1999) 718-727
69. Kawato M., Furukawa K., and Suzuki R. A hierarchical neural-network model for control and learning of voluntary movement. Biological Cybernetics 57 (1987) 169-185
70. Kawato M., and Gomi H. A computational model of four regions of the cerebellum based on feedback-error learning. Biological Cybernetics 68 (1992) 95-103
71. Knill D.C., and Pouget A. The Bayesian brain: The role of uncertainty in neural coding and computation. Trends in Neurosciences 27 (2004) 712-719
72. Konczak J., Borutta M., and Dichgans J. The development of goal-directed reaching in infants. II. Learning to produce task-adequate patterns of joint torque. Experimental Brain Research 113 (1997) 465-474
73. Konczak J., and Dichgans J. The development toward stereotypic arm kinematics during reaching in the first 3 years of life. Experimental Brain Research 117 (1997) 346-354
74. Kositsky M., and Barto A.G. The emergence of movement units through learning with noisy efferent signals and delayed sensory feedback. Neurocomputing, 44- 46 (2002) 889-895
75. Krakauer J.W., and Shadmehr R. Consolidation of motor memory. Trends in Neurosciences 29 (2006) 58-64
76. Kunde W., Kiesel A., and Hoffmann J. Conscious control over the content of unconscious cognition. Cognition 88 (2003) 223-242
77. Kunde W., Koch I., and Hoffmann J. Anticipated action effects affect the selection, initiation, and execution of actions. The Quarterly Journal of Experimental Psychology: Human Experimental Psychology 57 A (2004) 87-106
78. Kuperstein M. Neural model of adaptive hand-eye coordination for single postures. Science 239 (1988, March 11) 1308-1311
79. Kuperstein M. Infant neural controller for adaptive sensory-motor coordination. Neural Networks 4 (1991) 131-145
80. Latash M.L., Scholz J.P., and Schöner G. Motor control strategies revealed in the structure of motor variability. Exercise and Sport Sciences Reviews 30 (2002) 26-31
81. Lavrysen A., Helsen W.F., Tremblay L., Elliott D., Adam J.J., Feys P., et al. The control of sequential aiming movements: The influence of practice and manual asymmetries on the one-target advantage. Cortex 39 (2003) 307-325
82. Lozano-Perez T. Automatic planing of manipulator transfer movements. IEEE Transactions on Systems, Man, and Cybernetics 11 (1981) 681-698
83. Lozano-Perez T. A simple motion planning algorithm for general robot manipulators. IEEE Journal of Robotics and Automation, RA-3 (1987) 224-238
84. Ludwig D.A. EMG changes during the acquisition of a motor skill. American Journal of Physical Medicine 61 5 (1982) 229-243
85. Maravita A., Spence C., and Driver J. Multisensory integration and the body schema: Close to hand and within reach. Current Biology 13 (2003) 531-539
86. Martinetz T.M., Berkovitsch S.G., and Schulten K.J. "Neural-gas" network for vector quantization and its application to time-series prediction. IEEE Transactions on Neural Networks 4 (1993) 558-569
87. Mehta B., and Schaal S. Forward models in visuomotor control. Journal of Neurophysiology 88 (2002) 942-953
88. Mel B.W. A connectionist model may shed light on neural mechanisms for visually guided reaching. Journal of Cognitive Neuroscience 3 (1991) 273-292
89. Meulenbroek R.G.J., Rosenbaum D.A., Jansen C., Vaughan J., and Vogt S. Multijoint grasping movements. Simulated and observed effects of object location, object size, and initial aperture. Experimental Brain Research 138 (2001) 219-234
90. Moore A.W., and Atkeson C. Prioritized sweeping: Reinforcement learning with less data and less real time. Machine Learning 13 (1993) 103-130
91. Morasso P., Sanguineti V., and Spada G. A computational theory of targeting movements based on force fields and topology representing networks. Neurocomputing 15 (1997) 411-434
92. Mountcastle V.B. An organizing principle for cerebral function: The unit model and the distributed system. In: Edelman G.M., and Mountcastle V.V. (Eds). The mindful brain (1978), MIT Press, Cambridge, MA 7-50
93. Ognibene, D., Rega, A., & Baldassarre, G. (2006). A model of reaching integrating continuous reinforcement learning, accumulator models, and direct inverse modeling. From Animals to Animats 9: Proceedings of the Ninth International Conference on the Simulation of Adaptive Behavior (SAB-2006), 381-393.
94. Olsson, L. A. (2006). Information self-structuring for developmental robotics: Organization, adaptation, and integration. Unpublished doctoral dissertation, School of Computer Science, Faculty of Engineering and Information Sciences, University of Hertfordshire.
95. O'Regan J.K., and Noë A. A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences 24 (2001) 939-1031
96. Paninski L., Shoham S., Fellows M.R., Hatsopoulos N.G., and Donoghue J.P. Superlinear population encoding of dynamic hand trajectory in primary motor cortex. Journal of Neuroscience 24 (2004) 8551-8561
97. Pfeifer R., and Gomez G. Interacting with the real world: Design principles for intelligent systems. Artificial Life and Robotics 9 (2004) 1-6
98. Poggio T., and Bizzi E. Generalization in vision and motor control. Nature 431 (2004, October 14) 768-774
99. Prinz W. Perception and action planning. European Journal of Cognitive Psychology 9 (1997) 129-154
100. Rizzolatti G., Camarda R., Fogassi L., Gentilucci M., Luppino G., and Matelli M. Functional organization of inferior area 6 in the macaque monkey. Experimental Brain Research 71 (1988) 491-507
101. Rizzolatti G., Fadiga L., Fogassi L., and Gallese V. Enhanced: The space around us. Science 277 (1997, July 11) 190-191
102. Robertson E.M., and Miall R.C. Multi-joint limbs permit a flexible response to unpredictable events. Experimental Brain Research 117 (1997) 148-152
103. Rosenbaum D.A., Engelbrecht S.E., Bushe M.M., and Loukopoulos L.D. Knowledge model for selecting and producing reaching movements. Journal of Motor Behavior 25 (1993) 217-227
104. Rosenbaum D.A., Loukopoulos L.D., Meulenbroek R.G.J., Vaughan J., and Engelbrecht S.E. Planning reaches by evaluating stored postures. Psychological Review 102 (1995) 28-67
105. Rosenbaum D.A., Meulenbroek R.G.J., Vaughan J., and Jansen C. Posture-based motion planning: Applications to grasping. Psychological Review 108 (2001) 709-734
106. Roweis S., and Saul L. Nonlinear dimensionality reduction by locally linear embedding. Science 290 (2000, December 22) 2323-2326
107. Russel S., and Norvig P. Artificial intelligence-A modern approach (1995), Prentice Hall, London
108. Schmidt T. The finger in flight: Real-time motor control by visually masked color stimuli. Psychological Science 13 (2002) 112-118
109. Schwartz A.B., Moran D.W., and Reina G.A. Differential representation of perception and action in the frontal cortex. Science 303 (2004, January 16) 380-383
110. Schweighofer N., Arbib M.A., and Kawato M. Role of the cerebellum in reaching movements in humans. I. Distributed inverse dynamics control. European Journal of Neuroscience 10 (1998) 86-94
111. Schweighofer N., Spoelstra J., Arbib M.A., and Kawato M. Role of the cerebellum in reaching movements in humans. II. A neural model of the intermediate cerebellum. European Journal of Neuroscience 10 (1998) 95-105
112. Soechting J.F., Buneo C.A., Herrmann U., and Flanders M. Moving effortlessly in three dimensions: Does Donders' law apply to arm movement?. Journal of Neuroscience 15 (1995) 6271-6280
113. Souccar, K., & Grupen, R. A. (1996). Distributed motion control for multiple robotic manipulators. Proceedings of the 1996 IEEE Conference on Robotics and Automation, 3434-3439.
114. Stringer S.M., Rolls E.T., Trappenberg T.P., and de Araujo I.E.T. Self-organizing continuous attractor networks and motor function. Neural Networks 16 (2003) 161-182
115. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning, 216-224.
116. Sutton R.S., and Barto A.G. Reinforcement learning: An introduction (1998), MIT Press, Cambridge, MA
117. Tanaka Y., Tsuji T., Sanguineti V., and Morasso P.G. Bio-mimetic trajectory generation using a neural time-base generator. Journal of Robotic Systems 22 (2005) 625-637
118. Tenenbaum J.B., de Silva V., and Langford J.C. A global geometric framework for nonlinear dimensionality reduction. Science 290 (2000, December 22) 2319-2323
119. Todorov E. Optimality principles in sensorimotor control. Nature Reviews Neuroscience 7 (2004) 907-915
120. Todorov E., and Jordan M.I. Optimal feedback control as a theory of motor coordination. Nature Neuroscience 5 (2002) 1226-1235
121. Toussaint M. A sensorimotor map: Modulating lateral interactions for anticipation and planning. Neural Computation 18 (2006) 1132-1155
122. Tversky A. Elimination by aspects: A theory of choice. Psychological Review 79 (1972) 281-299
123. Vaughan, J., Rosenbaum, D. A., & Meulenbroek, R. G. J. (2006, June 3). Modeling reaching and manipulating in 2- and 3-D workspaces: The posture-based model. Proceedings of the Fifth International Conference on Development and Learning [CD-ROM]. Bloomington, IN.
124. von Hofsten C. On the development of perception and action. In: Valsiner J., and Connolly K.J. (Eds). Handbook of developmental psychology (2003), Sage, London 114-140
125. von Hofsten C. An action perspective on motor development. Trends in Cognitive Science 8 (2004) 266-272
126. Vorberg, D., Mattler, U., Heinecke, A., Schmidt, T., & Schwarzbach, J. (2003). Different time courses for visual perception and action priming. Proceedings of the National Academy of Sciences, USA, 100, 6275-6280.
127. Weigelt M., Kunde W., and Prinz W. End-state comfort in bimanual object manipulation. Experimental Psychology 53 (2006) 143-148
128. Whitney D.E. Resolved motion rate control of manipulators and human prostheses. IEEE Transactions on Man-Machine Systems 10 (1969) 47-53
129. Wolpert D.M., and Ghahramani Z. Computational principles of movement neuroscience. Nature Neuroscience 3 (2000) 1212-1217
130. Wolpert D.M., and Kawato M. Multiple paired forward and inverse models for motor control. Neural Networks 11 (1998) 1317-1329