Reinforcement learning of two-joint virtual arm reaching in a computer model of sensorimotor cortex

Neural Computation

Volumn 25, Issue 12, 2013, Pages 3263-3293

  Neymotin, Samuel A ^a,b   Chadderdon, George L ^b   Kerr, Cliff C ^b,c   Francis, Joseph T ^d,e,f   Lytton, William W ^d,e,f,g  

^a YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b SUNY DOWNSTATE MEDICAL CENTER   (United States)

^c UNIVERSITY OF SYDNEY   (Australia)

^d Program in Neural and Behavioral Science   (United States)

^e Downstate Health Sciences University   (United States)

^f POLYTECHNIC UNIVERSITY   (United States)

^g KINGS COUNTY HOSPITAL CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARM; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMAN; INNERVATION; LETTER; PHYSIOLOGY; PSYCHOMOTOR PERFORMANCE; SOMATOSENSORY CORTEX;

ARM; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMANS; PSYCHOMOTOR PERFORMANCE; SOMATOSENSORY CORTEX;

EID: 84887535941     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00521     Document Type: Letter

References (75)

1. Afshar, A., Santhanam, G., Yu, B., Ryu, S., Sahani,M.,&Shenoy, K. (2011). Single-trial neural correlates of arm movement preparation. Neuron, 71(3), 555-564.
2. Almassy, N., Edelman, G., & Sporns, O. (1998). Behavioral constraints in the development of neuronal properties: A cortical model embedded in a real-world device. Cereb. Cortex, 8(4), 346-361.
3. Bannister, A. (2005). Inter-and intra-laminar connections of pyramidal cells in the neocortex. Neuroscience Research, 53(2), 95-103.
4. Berthier, N. (2011). The syntax of human infant reaching. In 8th International Conference on Complex Systems (pp. 1477-1487).
5. Berthier, N., Clifton, R., McCall, D., & Robin, D. (1999). Proximodistal structure of early reaching in human infants. Exp. Brain Res., 127(3), 259-269.
6. Carnevale, N., & Hines, M. (2006). The NEURON book. Cambridge: Cambridge University Press.
7. Chadderdon, G., Neymotin, S., Kerr, C., & Lytton,W. (2012). Reinforcement learning of targeted movement in a spiking neuronal model of motor cortex. PLoS One, 7(10), e47251.
8. Cools, R. (2006). Dopaminergic modulation of cognitive function-implications for l-dopa treatment in parkinson's disease. Neurosci. Biobehav. Rev., 30(1), 1-23.
9. Corbetta, D., & Snapp-Childs, W. (2009). Seeing and touching: The role of sensorymotor experience on the development of infant reaching. Infant Behav. Dev., 32(1), 44-58.
10. De Schutter, E. (2008). Why are computational neuroscience and systems biology so separate? PLoS Comput Biol, 4(5), e1000078.
11. Dyhrfjeld-Johnsen, J., Santhakumar, V., Morgan, R., Huerta, R., Tsimring, L., & Soltesz, I. (2007). Topological determinants of epileptogenesis in large-scale structural and functional models of the dentate gyrus derived fromexperimental data. J. Neurophysiol., 97(2), 1566-1587.
12. Edelman, G. (1987). Neural Darwinism: The theory of neuronal group selection. New York: Basic Books.
13. Edelman, G. (2006). The embodiment of mind. Daedalus, 135(3), 23-32.
14. Engel, A., Konig, P., Kreiter, A., Gray, C., & Singer, W. (1991). Temporal coding by coherent oscillations as a potential solution to the binding problem: Physiological evidence. In H. Schuster (Ed.), Nonlinear dynamics and neural networks. Weinheim: VCH Verlagsgesellschaft.
15. Evans, R.,Morera-Herreras, T., Cui, Y., Du, K., Sheehan, T., Kotaleski, J., et al. (2012). The effects of NMDA subunit composition on calcium influx and spike timingdependent plasticity in striatal medium spiny neurons. PLoS Comput. Bio., 8(4), e1002493.
16. Farries, M., & Fairhall, A. (2007). Reinforcement learning with modulated spike timing-dependent synaptic plasticity. J. Neurophysiol., 98(6), 3648-3665.
17. Fenton, A., Lytton, W., Barry, J., Lenck-Santini, P., Zinyuk, L., Kubík, S., et al. (2010). Attention-like modulation of hippocampus place cell discharge. J. Neurosci., 30(13), 4613-4625.
18. Florian, R. (2007). Reinforcement learning through modulation of spike-timingdependent synaptic plasticity. Neural Comput., 19(6), 1468-1502.
19. Frank, M., & O'Reilly, R. (2006). A mechanistic account of striatal dopamine function in human cognition: Psychopharmacological studies with cabergoline and haloperidol. Behav. Neurosci., 120(3), 497.
20. Frank, M., Seeberger, L., & O'Reilly, R. (2004). By carrot or by stick: Cognitive reinforcement learning in Parkinsonism. Science, 306(5703), 1940-1943.
21. Gourevitch, B., & Eggermont, J. (2007). Evaluating information transfer between auditory cortical neurons. J. Neurophysiol., 97(3), 2533-2543.
22. Graybiel, A., Aosaki, T., Flaherty, A., & Kimura, M. (1994). The basal ganglia and adaptive motor control. Science, 265(5180), 1826-1831.
23. Hikosaka, O., Nakamura, K., Sakai, K., & Nakahara, H. (2002). Central mechanisms of motor skill learning. Current Opin. Neurobiol., 12(2), 217-222.
24. Hosp, J., Pekanovic, A., Rioult-Pedotti, M., & Luft, A. (2011). Dopaminergic projections from midbrain to primary motor cortex mediate motor skill learning. J. Neurosci., 31(7), 2481-2487.
25. Houk, J., &Wise, S. (1995). Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: Their role in planning and controlling action. Cereb. Cortex, 5(2), 95-110.
26. Izhikevich, E. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex, 17, 2443-2452.
27. Jones, S., Kerr, C., Wan, Q., Pritchett, D., Hämäläinen, M., & Moore, C. (2010). Cued spatial attention drives functionally relevant modulation of the mu rhythm in primary somatosensory cortex. J. Neurosci., 30(41), 13760-13765.
28. Kelemen, E., & Fenton, A. (2010). Dynamic grouping of hippocampal neural activity during cognitive control of two spatial frames. PLoS Biol., 8(6), e1000403.
29. Kerr, C., Neymotin, S., Chadderdon, G., Fietkiewicz, C., Francis, J., & Lytton, W. (2012). Electrostimulation as a prosthesis for repair of information flow in a computer model of neocortex. IEEE Trans. Neural Syst. Rehabil. Eng., 20, 153-160.
30. Kerr, C., Van Albada, S., Neymotin, S., Chadderdon, G., Robinson, P., & Lytton, W. (2013). Cortical information flow in Parkinson's disease: A composite network/field model. Front Comput. Neurosci., 7, 39.
31. Krichmar, J., & Edelman, G. (2005). Brain-based devices for the study of nervous systems and the development of intelligent machines. Artif. Life, 11(1-2), 63-77.
32. Kubikova, L., & Kostál, L. (2010). Dopaminergic system in birdsong learning and maintenance. J. Chem. Neuroanat, 39(2), 112-123.
33. Le Novère, N. (2007). The long journey to a systems biology of neuronal function. BMC Syst. Biol., 1(1), 28.
34. Luft, A., & Schwarz, S. (2009). Dopaminergic signals in primary motor cortex. Int. J. Dev. Neurosci., 27(5), 415-421.
35. Lungarella, M., & Sporns, O. (2006). Mapping information flow in sensorimotor networks. PLoS Comput. Biol., 2(10), e144.
36. Lytton, W. (2008). Computer modelling of epilepsy. Nature Rev. Neurosci., 9(8), 626-637.
37. Lytton, W., Neymotin, S., & Hines, M. (2008). The virtual slice setup. J. Neurosci Methods, 171(2), 309-315.
38. Lytton, W., & Omurtag, A. (2007). Tonic-clonic transitions in computer simulation. J. Clin. Neurophys., 24, 175-181.
39. Lytton,W., Omurtag, A., Neymotin, S., & Hines,M. (2008). Just-in-time connectivity for large spiking networks. Neural Comput., 20(11), 2745-2756.
40. Lytton,W., & Stewart,M. (2005). A rule-based firing model for neural networks. Int. J. Bioelectromagnetism, 7, 47-50.
41. Lytton, W., & Stewart, M. (2006). Rule-based firing for network simulations. Neurocomputing, 69(10-12), 1160-1164.
42. Mahmoudi, B., & Sanchez, J. (2011). A symbiotic brain-machine interface through value-based decision making. PloS One, 6(3), e14760.
43. Marsh, B., Tarigoppula, A., & Francis, J. (2011). Correlates of reward expectation in the primary motor cortex: Developing an actor-critic model in macaques for a brain computer interface. Society for Neuroscience Abstracts, 41.
44. Mo, J., Schroeder, C., & Ding, M. (2011). Attentional modulation of alpha oscillations in macaque inferotemporal cortex. J. Neurosci., 31(3), 878-882.
45. Molina-Luna, K., Pekanovic, A., Röhrich, S., Hertler, B., Schubring-Giese,M., Rioult-Pedotti,M., et al. (2009). Dopamine in motor cortex is necessary for skill learning and synaptic plasticity. PLoS One, 4(9), e7082.
46. Neymotin, S., Jacobs, K., Fenton, A., & Lytton,W. (2011). Synaptic information transfer in computer models of neocortical columns. J. Comput. Neurosci., 30(1), 69-84.
47. Neymotin, S., Kerr, C., Francis, J., & Lytton,W. (2011). Training oscillatory dynamics with spike-timing-dependent plasticity in a computer model of neocortex. In 2011 IEEE Signal Processing in Medicine and Biology Symposium (SPMB) (pp. 1-6) Piscataway, NJ: IEEE.
48. Neymotin, S., Lazarewicz, M., Sherif, M., Contreras, D., Finkel, L., & Lytton, W. (2011). Ketamine disrupts theta modulation of gamma in a computer model of hippocampus. J. Neurosci., 31(32), 11733-11743.
49. Neymotin, S., Lee, H., Park, E., Fenton, A., & Lytton,W. (2011). Emergence of physiological oscillation frequencies in a computer model of neocortex. Front Comput. Neurosci., 5, 19.
50. Pastalkova, E., Serrano, P., Pinkhasova, D., Wallace, E., Fenton, A., & Sacktor, T. (2006). Storage of spatial information by the maintenance mechanism of LTP. Science, 313(5790), 1141-1144.
51. Peterson, B., Healy, M., Nadkarni, P., Miller, P., & Shepherd, G. (1996). ModelDB: An environment for running and storing computational models and their results applied to neuroscience. J. Am. Med. Inform. Assoc., 3(6), 389-398.
52. Potjans, W., Morrison, A., & Diesmann, M. (2009). A spiking neural network model of an actor-critic learning agent. Neural Comput., 21(2), 301-339.
53. Qiu, S., Anderson, C., Levitt, P., & Shepherd, G. (2011). Circuit-specific intracortical hyperconnectivity in mice with deletion of the autism-associated met receptor tyrosine kinase. J. Neurosci., 31(15), 5855-5864.
54. Reid, R. (2012). From functional architecture to functional connectomics. Neuron, 75(2), 209-217.
55. Reynolds, J., & Wickens, J. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw, 15(4-6), 507-521.
56. Roberts, P., & Bell, C. (2002). Spike timing dependent synaptic plasticity in biological systems. Biol. Cybern., 87(5), 392-403.
57. Rowan, M., & Neymotin, S. (2013). Synaptic scaling balances learning in a spiking model of neocortex. Springer LNCS, 7824, 20-29.
58. Sanes, J. (2003). Neocortical mechanisms in motor learning. Curr. Opin. Neurobiol., 13(2), 225-231.
59. Seung, H. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 40(6), 1063-1073.
60. Shadmehr, R., & Krakauer, J. W. (2008). A computational neuroanatomy for motor control. Exp. Brain Res., 185(3), 359-381.
61. Shadmehr, R.,&Wise, S. (2005). The computational neurobiology of reaching and pointing: A foundation for motor learning. Cambridge, MA: MIT Press.
62. Shen,W., Flajolet, M., Greengard, P., & Surmeier, D. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science, 321(5890), 848-851.
63. Shepherd, G. (2004). The synaptic organization of the brain. New York: Oxford University Press.
64. Sober, S., & Brainard, M. (2009). Adult birdsong is actively maintained by error correction. Nat. Neurosci., 12(7), 927-931.
65. Song, S., Miller, K.,&Abbott, L. (2000). Competitive hebbian learning through spiketiming-dependent synaptic plasticity. Nat. Neurosci., 3, 919-926.
66. Sporns, O., Tononi, G., & Kotter, R. (2005). The human connectome: A structural description of the human brain. PLoS Comput. Biol., 1(4), e42.
67. Thomson, A., & Bannister, A. (2003). Interlaminar connections in the neocortex. Cereb. Cortex, 13(1), 5-14.
68. Thomson, A., West, D., Wang, Y., & Bannister, A. (2002). Synaptic connections and small circuits involving excitatory and inhibitory neurons in layers 2-5 of adult rat and cat neocortex: triple intracellular recordings and biocytin labelling in vitro. Cereb. Cortex, 12, 936-953.
69. Thorndike, E. (1911). Animal intelligence. New York: Macmillan.
70. Tiesinga, P., & Sejnowski, T. (2004). Rapid temporal modulation of synchrony by competition in cortical interneuron networks. Neural Comput., 16(2), 251-275.
71. Uhlhaas, P., & Singer,W. (2006). Neural synchrony in brain disorders: Relevance for cognitive dysfunctions and pathophysiology. Neuron, 52, 155-168.
72. Von der Malsburg, C., & Schneider, W. (1986). A neural cocktail-party processor. Biol. Cybern., 54, 29-40.
73. von Hofsten, C. (1979). Development of visually directed reaching: The approach phase. Department of Psychology, University of Uppsala.
74. Von Kraus, L., Sacktor, T., & Francis, J. (2010). Erasing sensorimotor memories via PKMζ inhibition. PloS One, 5(6), e11125.
75. Webb, B. (2000). What does robotics offer animal behaviour? Animal Behav., 60(5), 545-558.