Effect on movement selection of an evolving sensory representation: A multiple controller model of skill acquisition

Brain Research

Volumn 1299, Issue , 2009, Pages 55-73

  Shah, Ashvin ^a   Barto, Andrew G ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b University of Massachusetts Amherst   (United States)

Author keywords

Basal ganglia; Computational model; Decision making; Reinforcement learning; Skill acquisition; Uncertainty

Indexed keywords

ARTICLE; BASAL GANGLION; DECISION MAKING; HUMAN; LEARNING; MATHEMATICAL MODEL; NONHUMAN; PREFRONTAL CORTEX; PRIORITY JOURNAL; REINFORCEMENT; SKILL; UNCERTAINTY; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER SIMULATION; ENVIRONMENT; MOVEMENT (PHYSIOLOGY); NERVE CELL; NERVE CELL NETWORK; NERVE CELL PLASTICITY; PHYSIOLOGY;

COMPUTER SIMULATION; DECISION MAKING; ENVIRONMENT; HUMANS; LEARNING; MODELS, NEUROLOGICAL; MOVEMENT; NERVE NET; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEURONS; PRACTICE (PSYCHOLOGY);

EID: 70449698526     PISSN: 00068993     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.brainres.2009.07.006     Document Type: Article

References (89)

1. Abbs J., Gracco V., and Cole K. Control of multimovement coordination: sensorimotor mechanisms in speech motor programming. J. Mot. Behav. 16 (1984) 195-231
2. Baader A., Kasennikov O., and Wiesendanger M. Coordination of bowing and fingering in violin playing. Cogn. Brain Res. 23 (2005) 436-443
3. Battaglia P., and Schrater P. Humans trade off viewing time and movement duration to improve visuomotor accuracy in a fast reaching task. J. Neurosci. 27 (2007) 6984-6994
4. Bayley P., Frascino J., and Squire L. Robust habit learning in the absence of awareness and independent of the medial temporal lobe. Nature 436 (2005) 550-553
5. Bolam J., Hanley J., Booth P., and Bevan M. Synaptic organisation of the basal ganglia. J. Anat. 196 (2000) 527-542
6. Brafman R., and Tennenholtz M. R-max a general polynomial time algorithm for near optimal reinforcement learning. J. Mach. Learn. Res. 3 (2002) 213-231
7. Britten K., Shadlen M., Newsome W., and Movshon J. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurophysiol. 12 (1992) 4745-4765
8. Centonze D., Picconi B., Gubellini P., Bernardi G., and Calabresi P. Dopaminergic control of synaptic plasticity in the dorsal striatum. Eur. J. Neurosci. 13 (2001) 1071-1077
9. Chrisman L. Reinforcement Learning with Perceptual Aliasing: The Perceptual Distinctions Approach. In Proceedings of the Tenth National Conference on Artificial Intelligence (1992), Morgan Kaufmann, San Jose, CA
10. Corkin S. Acquisition of motor skill after bilateral medial temporal-lobe excision. Neuropsychologia 6 (1968) 225-264
11. Corkin S. What's new with the amnesiac patient H.M.?. Nat. Rev. Neurosci. 3 (2002) 153-160
12. Daw N., Niv Y., and Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8 (2005) 1704-1711
13. Dickinson A. Actions and habits: the development of behavioral autonomy. Philos. Trans. R. Soc. Lond., B Biol. Sci. 308 (1985) 67-78
14. DiZio P., and Lackner J. Motor adaptation to coriolis force perturbations of reaching movements: endpoint but not trajectory adaptation transfers to non-exposed arm. J. Neurophysiol. 74 (1995) 1787-1792
15. Doya K. Reinforcement learning: computational theory and biological mechanisms. HFSP Journal. 1 (2007) 30-40
16. Doyon J., and Benali H. Reorganization and plasticity in the adult brain during learning of motor skills. Current Opinion in Neurobiology 15 (2005) 161-167
17. Engel K.C., Flanders M., and Soechting J.F. Anticipatory and sequential motor control in piano playing. Exp. Brain Res. 113 (1997) 189-199
18. Fagg A., Zelevinsky L., Barto A., and Houk J.C. Using crude corrective movements to learn accurate motor programs for reaching. Presented at NIPS Workshop on Can Artificial Cerebellar Models Compete to Control Robots? (1997) Breckenridge, CO
19. Fagg, A., Zelevinsky, L., Barto, A. G., and Houk, J. C., 1997b. Cerebellar learning for control of a two-link arm in muscle space. In Proceedings of the IEEE Conference on Robotics and Automation, pp. 2638-2644.
20. Fagg A., Barto A.G., and Houk J.C. Learning to Reach via Corrective Movements. In Proceedings of the Tenth Yale Workshop on Adaptive and Learning Systems (1998) 179-185 New Haven, CT
21. Frank M., Loughry B., and O'Reilly R. Interactions between the frontal cortex and basal ganglia in working memory: a computational model. Cogn. Affect. Behav. Neurosci. 1 (2001) 137-160
22. Glimcher P. Decisions, decisions, decisions: choosing a biological science of choice. Neuron 36 (2002) 323-332
23. Glimcher P. The neurobiology of visual-saccadic decision making. Annu. Rev. Neurosci. 26 (2003) 133-179
24. Graybiel A.M. The basal ganglia and chunking of action repertoires. Neurobiol. Learn. Mem. 70 (1998) 119-136
25. Graybiel A.M. The basal ganglia: learning new tricks and loving it. Curr. Opin. Neurobiol. 15 (2005) 638-644
26. Harris C.M., and Wolpert D.M. Signal dependent noise determines motor planning. Nature 394 (1998) 780-784
27. Hart P., Nilsson N., and Raphael B. A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern. SSC-4 (1968) 100-107
28. Hikosaka O., Rand M.K., Miyachi S., and Miyashita K. Learning of sequential movements in the monkey: process of learning and retention of memory. J. Neurophysiol. 74 (1995) 1652-1661
29. Hikosaka O., Nakahara H., Rand M.K., Sakai K., Lu X., Nakamura K., Miyachi S., and Doya K. Parallel neural networks for learning sequential procedures. Trends Neurosci. 22 (1999) 464-471
30. Houk J., and Wise S. Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling action. Cereb. Cortex 5 (1995) 95-110
31. Houk J.C., Adams J., and Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk J.C., Davis J.L., and Beiser D.G. (Eds). Models of Information Processing in the Basal Ganglia (1995), MIT Press, Cambridge, MA 249-270
32. Hudson T., Maloney L., and Landy M. Movement planning with probabilistic target information. J. Neurophysiol. 98 (2007) 3034-3046
33. Jeannerod M. Intersegmental coordination during reaching at natural visual objects. In: Long J., and Baddeley A. (Eds). Attention and Performance IX (1981), Erlbaum, Hillsdale, NJ 153-168
34. Jerde T., Soechting J., and Flanders M. Coarticulation in fluent finger spelling. J. Neurosci. 23 (2003) 2383-2393
35. Jog M.S., Kubota Y., Connolly C.I., Hillegaart V., and Graybiel A.M. Building neural representations of habits. Science 286 (1999) 1745-1749
36. Kaelbling L., Littman M., and Cassandra A. Planning and acting in partially observable stochastic domains. Artif. Intell. 101 (1998) 99-134
37. Kalman R. A new approach to linear filtering and prediction problems. Transaction of the ASME Journal of Basic Engineering 82 (1960) 35-45
38. Kawato M. Feedback-error-learning neural network for supervised motor learning. In: Eckmiller R. (Ed). Advanced Neural Computers (1990), Elsevier, North-Holland 365-372
39. Kawato M., and Gomi H. The cerebellum and vor/okr learning models. Trends Neurosci. 15 (1992) 445-453
40. Kawato M., Furukawa K., and Suzuki R. A hierarchical neural-network model for control and learning of voluntary movement. Biol. Cybern. 57 (1987) 169-185
41. Kent R.D., and Minifie F.D. Coarticulation in recent speech production models. J. Phon. 5 (1977) 115-117
42. Klein-Breteler M., Hondzinski J., and Flanders M. Drawing sequences of segments in 3d: kinetic influences on arm configuration. J. Neurophysiol. 89 (2003) 3253-3263
43. Knowlton B.J., Squire L.R., and Gluck M. Probabilistic category learning in amnesia. Learning and memory. 1 106-120 (1994)
44. Knowlton B.J., Mangels J.A., and Squire L.R. A neostriatal habit learning system in humans. Science 273 (1996) 1399-1402
45. Kording K., and Wolpert D. Bayesian integration in sensorimotor learning. Nature 427 (2004) 244-247
46. Kording K., and Wolpert D. Bayesian decision theory in sensorimotor control. Trends Cogn. Sci. 10 (2006) 319-326
47. Lackner J., and DiZio P. Rapid adaptation to coriolis force perturbations of arm trajectory. J. Neurophysiol. 72 (1994) 299-313
48. Lackner J., and DiZio P. Adaptation in a rotating artificial gravity environment. Brain Res. Rev. 28 (1998) 194-202
49. Lackner J., and DiZio P. Adaptation to coriolis force perturbation of movement trajectory: role of prioprioceptive and cutaneous somatosensory feedback. Adv. Exp. Med. Biol. 508 (2002) 69-78
50. Ledoux J. The Emotional Brain (1998), Simon and Schuster, New York, NY
51. Littman M., Cassandra A., and Kaelbling L. Learning Policies for Partially Observable Environments: Scaling Up. In Proceedings of the Twelfth International Conference on Machine Learning (1995), Morgan Kaufmann, San Francisco, CA 362-370
52. Matsuzaka Y., Picard N., and Strick P. Skill representation in the primary motor cortex after long-term practice. J. Neurophysiol. 97 (2007) 1819-1832
53. Messier J., Adamovich S., Berkinblit M., Tunik E., and Poizner H. Influence of movement speed on accuracy and coordination of reaching movements to memorized targets in three dimensional space in a deafferented subject. Exp. Brain Res. 150 (2003) 399-419
54. Miller E.K. The prefrontal cortex and cognitive control. Nat. Rev. Neurosci. 1 (2000) 59-65
55. Miller E.K., and Cohen J.D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24 (2001) 167-202
56. Milner B. Les troubles de la m'emoire accompagnant les l'esions hippocampiques bilat'erales. Physiologie del l'Hippocampe, Colloqeus Internationaux 107 (1962) 257-272
57. Milner B., Squire L., and Kendel E. Cognitive neuroscience and the study of memory. Neuron 20 (1998) 445-468
58. Mink J. The basal ganglia: focused selection and inhibition of competing motor programs. Prog. Neurobiol. 50 (1996) 381-425
59. Morris G., Nevet A., Arkadir D., Vaada E., and Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9 (2006) 1057-1063
60. Mushiake H., Saito N., Sakamoto K., Itoyama Y., and Tanji J. Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 50 (2006) 631-641
61. Nakahara H., Doya K., and Hikosaka O. Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences - a computational approach. J. Cogn. Neurosci. 13 (2001) 626-647
62. Niv Y., Daw N., and Dayan P. Choice values. Nat. Neurosci. 9 (2006) 987-988
63. Opris I., and Bruce C. Neural circuitry of judgment and decision mechanisms. Brain Res. Rev. 48 (2005) 509-526
64. Packard M., and Knowlton B. Learning and memory functions of the basal ganglia. Annu. Rev. Neurosci. 25 (2002) 563-593
65. Pasupathy A., and Miller E.K. Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 433 (2005) 873-876
66. Platt M., and Glimcher P. Neural correlates of decision variables in parietal cortex. Nature 400 (1999) 233-238
67. Puttemans V., Wenderoth N., and Swinnen S. Changes in brain activation during the acquisition of a multifrequency bimanual coordination task: from the cognitive stage to advanced levels of automaticity. J. Neurosci. 25 (2005) 4270-4278
68. Rao A., and Gordon A. Contribution of tactile information to accuracy in pointing movements. Exp. Brain Res. 138 (2001) 438-445
69. Rosenstein M. Learning to Exploit Dynamics for Robot Motor Coordination (2003), University of Massachusetts Amherst, PhD thesis
70. Rosenstein M., and Barto A. Supervised actor-critic reinforcement learning. In: Si J., Barto A., Powell W., and Wunsch D. (Eds). Handbook of Learning and Approximate Dynamic Programming. IEEE Press Series on Computational Intelligence (2004), Wiley-IEEE Press, Piscataway, NJ 359-380 chapter 14
71. Rummery G., and Niranjan M. On-line Q-Learning Using Connectionist Systems. Technical Report CUED/F-INFENG/TR 166 (1994), Engineering Department, Cambridge University, Cambridge, England
72. Samejima K., Ueda Y., Doya K., and Kimura M. Representation of action-specific reward values in the striatum. Science 310 (2005) 1337-1340
73. Schlegel T., and Schuster S. Small circuits for large tasks: high-speed decision-making in archerfish. Science 319 (2008) 104-106
74. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80 (1998) 1-27
75. Seger C., and Cincotta C. Dynamics of frontal, striatal, and hippocampal systems during rule learning. Cereb. Cortex 16 (2006) 1546-1555
76. Shah A. Biologically-based Functional Mechanisms of Motor Skill Acquisition (2008), University of Massachusetts Amherst, PhD thesis
77. Shah, A., Barto, A., and Fagg, A., 2006. Biologically-based functional mechanisms of coarticulation. Poster presentation at the Sixteenth Annual Neural Control of Movement Conference, Key Biscayne, FL, May 2-7. Poster available online at: http://www-all.cs.umass.edu/pubs/2006/shah_bf_NCM06.pdf.
78. Sutton R.S., and Barto A.G. Reinforcement Learning (1998), An Introduction. MIT Press, Cambridge, MA
79. Tanji J., and Hoshi E. Role of the lateral prefrontal cortex in executive behavioral control. Physiol. Rev. 88 (2008) 37-57
80. Tassinari H., Hudson T., and Landy M. Combining priors and noisy visual cues in a rapid pointing task. J. Neurosci. 26 (2006) 10154-10163
81. Thorndike E.L. Animal Intelligence (1911), Hafner, Darien, CT
82. Tunik E., Poizner H., Levin M., Adamovich S., Messier J., Lamarre Y., and Feldman A. Arm-trunk coordination in the absence of proprioception. Exp. Brain Res. 153 (2003) 343-355
83. Waelti P., Dickinson A., and Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412 (2001) 43-48
84. Wickens J., Reynolds J., and Hyland B. Neural mechanisms of reward-related motor learning. Current Opinion in Neurobiology. 13 (2003) 685-690
85. Wilson C., and Oorschot D. Neural dynamics and surround inhibition in the neostriatum: a possible connection. In: Miller R., and Wickens J. (Eds). Brain Dynamics and the Striatal Complex (2000), Harwood, Amsterdam 141-149
86. Wolpert D. Probabilistic models in human sensorimotor control. Hum. Mov. Sci. 27 (2007) 511-524
87. Yin H., and Knowlton B. The role of the basal ganglia in habit formation. Nat. Rev. Neurosci. 7 (2006) 464-476
88. Yoshida W., and Ishii S. Resolution of uncertainty in prefrontal cortex. Neuron 50 (2006) 781-789
89. Zheng T., and Wilson C. The implications of corticostriatal axonal arborizations. J. Neurophysiol. 87 (2002) 1007-1017