Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex

PLoS ONE

Volumn 7, Issue 10, 2012, Pages

  Chadderdon, George L ^a   Neymotin, Samuel A ^a,b   Kerr, Cliff C ^a,c   Lytton, William W ^a,d,e  

^a SUNY DOWNSTATE MEDICAL CENTER   (United States)

^b YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c UNIVERSITY OF SYDNEY   (Australia)

^d Downstate Health Sciences University   (United States)

^e KINGS COUNTY HOSPITAL CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

4 AMINOBUTYRIC ACID; ALPHA AMINO 3 HYDROXY 5 METHYL 4 ISOXAZOLEPROPIONIC ACID; N METHYL DEXTRO ASPARTIC ACID;

ARTICLE; CELL MOTION; CONTROLLED STUDY; EXCITATORY POSTSYNAPTIC POTENTIAL; EXTENSOR MUSCLE; FLEXOR MUSCLE; FOREARM; INHIBITORY POSTSYNAPTIC POTENTIAL; LEARNING; LOCOMOTION; MOTOR CORTEX; NERVE CELL PLASTICITY; PROBABILITY; PROPRIOCEPTION; REINFORCEMENT; SENSORIMOTOR FUNCTION; SIGNAL NOISE RATIO; SPIKE WAVE; STOCHASTIC MODEL; VIRTUAL REALITY;

ACTION POTENTIALS; ANIMALS; COMPUTER SIMULATION; DOPAMINE; HUMANS; MODELS, NEUROLOGICAL; MOTOR CORTEX; MOVEMENT; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEURONS; REINFORCEMENT (PSYCHOLOGY); REWARD; STOCHASTIC PROCESSES; SYNAPSES; SYNAPTIC TRANSMISSION;

EID: 84867671556     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0047251     Document Type: Article

References (60)

1. Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: Neural Networks, 1989. IJCNN., International Joint Conference on. IEEE, 593-605.
2. Rumelhart D, McClelland J (1986) Parallel distributed processing: Psychological and biological models, volume 2. The MIT Press.
3. Kohonen T, (1990) The self-organizing map. Proceedings of the IEEE 78: 1464-1480.
4. Tesauro G, (1995) Temporal difference learning and TD-Gammon. Communications of the ACM 38: 58-68.
5. Farries M, Fairhall A, (2007) Reinforcement learning with modulated spike timing-dependent synaptic plasticity. Journal of Neurophysiology 98: 3648-3665.
6. Florian R, (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation 19: 1468-1502.
7. Izhikevich E, (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443-2452.
8. Potjans W, Morrison A, Diesmann M, (2009) A spiking neural network model of an actor-critic learning agent. Neural Computation 21: 301-339.
9. Seung H, (2003) Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40: 1063-1073.
10. Dan Y, Poo M, (2004) Spike timing-dependent plasticity of neural circuits. Neuron 44: 23-30.
11. Roberts P, Bell C, (2002) Spike timing dependent synaptic plasticity in biological systems. Biol Cybern 87: 392-403.
12. Song S, Miller K, Abbott L, (2000) Competitive hebbian learning through spike-timing-dependent synaptic plasticity. Nat Neurosci 3: 919-926.
13. Singer W (2003) Synchronization, binding and expectancy. The handbook of brain theory and neural networks: 1136-1143.
14. Thorpe S, Fize D, Marlot C, et al. (1996) Speed of processing in the human visual system. Nature 381: 520-522.
15. VanRullen R, Guyonneau R, Thorpe S, (2005) Spike times make sense. Trends Neurosci 28: 1-4.
16. Baker S, Kilner J, Pinches E, Lemon R, (1999) The role of synchrony and oscillations in the motor output. Experimental Brain Research 128: 109-117.
17. Thorndike E (1911) Animal intelligence. New York: Macmillan.
18. Sutton R, Barto A (1998) Reinforcement learning: An introduction, volume 1. Cambridge Univ Press.
19. Sanchez J, Tarigoppula A, Choi J, Marsh B, Chhatbar P, et al. (2011) Control of a center-out reaching task using a reinforcement learning brain-machine interface. In: Neural Engineering (NER), 2011 5th International IEEE/EMBS Conference on. IEEE, 525-528.
20. Chadderdon G (2009) A neurocomputational model of the functional role of dopamine in stimulus-response task learning and performance. Ph.D. thesis, Indiana University. URL http://pqdtopen.proquest.com/#viewpdf?dispub=3355003.
21. Joel D, Niv Y, Ruppin E, (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks 15: 535-547.
22. Demiris Y, Dearden A (2005) From motor babbling to hierarchical learning by imitation: a robot developmental pathway.
23. Der R, Martius G, (2006) From motor babbling to purposive actions: Emerging self-exploration in a dynamical systems approach to early robot development. From Animals to Animats 9: 406-421.
24. Wanjerkhede S, Bapi R, (2007) Modeling the sub-cellular signaling pathways involved in reinforcement learning at the striatum. Progress in Brain Research 168: 193-206.
25. Magee J, Johnston D, (1995) Synaptic activation of voltage-gated channels in the dendrites of hippocampal pyramidal neurons. Science 268: 301-304.
26. Takechi H, Eilers J, Konnerth A, (1998) A new class of synaptic response involving calcium release in dendritic spines. Nature 396: 757.
27. Hollerman J, Schultz W, (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1: 304-309.
28. Schultz W, (1998) Predictive reward signal of dopamine neurons. Journal of Neurophysiology 80: 1-27.
29. Ungless M, Magill P, Bolam J, (2004) Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303: 2040-2042.
30. Hosp J, Pekanovic A, Rioult-Pedotti M, Luft A, (2011) Dopaminergic projections from midbrain to primary motor cortex mediate motor skill learning. The Journal of Neuroscience 31: 2481-2487.
31. Luft A, Schwarz S, (2009) Dopaminergic signals in primary motor cortex. International Journal of Developmental Neuroscience 27: 415-421.
32. Molina-Luna K, Pekanovic A, Röhrich S, Hertler B, Schubring-Giese M, et al. (2009) Dopamine in motor cortex is necessary for skill learning and synaptic plasticity. PLoS ONE 4: e7082.
33. Almassy N, Edelman G, Sporns O, (1998) Behavioral constraints in the development of neuronal properties: a cortical model embedded in a real-world device. Cerebral Cortex 8: 346-361.
34. Sporns O, Alexander W, (2002) Neuromodulation and plasticity in an autonomous robot. Neural Networks 15: 761-774.
35. Lytton W, Stewart M, (2005) A rule-based firing model for neural networks. Int J Bioelectromagnetism 7: 47-50.
36. Lytton W, Stewart M, (2006) Rule-based firing for network simulations. Neurocomputing 69: 1160-1164.
37. Lytton W, Omurtag A, (2007) Tonic-clonic transitions in computer simulation. J Clin Neurophysiol 24: 175-181.
38. Lytton W, Neymotin S, Hines M, (2008b) The virtual slice setup. J Neurosci Methods 171: 309-315.
39. Lytton W, Omurtag A, Neymotin S, Hines M, (2008) Just-in-time connectivity for large spiking networks. Neural Comput 20: 2745-2756.
40. Neymotin S, Lee H, Park E, Fenton A, Lytton W, (2011b) Emergence of physiological oscillation frequencies in a computer model of neocortex. Front Comput Neurosci 5: 19.
41. Carnevale N, Hines M (2006) The NEURON Book. New York: Cambridge University Press.
42. Reynolds J, Wickens J, (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks 15: 507-521.
43. Shen W, Flajolet M, Greengard P, Surmeier D, (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science's STKE 321: 848.
44. Faure A, Haberland U, Condé F, El Massioui N, (2005) Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. The Journal of Neuroscience 25: 2771-2780.
45. Robbins T, Giardini V, Jones G, Reading P, Sahakian B, (1990) Effects of dopamine depletion from the caudate-putamen and nucleus accumbens septi on the acquisition and performance of a conditional discrimination task. Behavioural Brain Research 38: 243-261.
46. O'Neill M, Brown V, (2007) The effect of striatal dopamine depletion and the adenosine A2A antagonist KW-6002 on reversal learning in rats. Neurobiology of Learning and Memory 88: 75-81.
47. Parkinson J, Dalley J, Cardinal R, Bamford A, Fehnert B, et al. (2002) Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive pavlovian approach behaviour: implications for mesoaccumbens dopamine function. Behavioural Brain Research 137: 149-163.
48. Smith-Roe S, Kelley A, (2000) Coincident activation of NMDA and dopamine D1Receptors within the nucleus accumbens core is required for appetitive instrumental learning. The Journal of Neuroscience 20: 7737-7742.
49. Cools R, (2006) Dopaminergic modulation of cognitive function-implications for L-DOPA treatment in parkinson's disease. Neuroscience & Biobehavioral Reviews 30: 1-23.
50. Frank M, O'Reilly R, (2006) A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behavioral Neuroscience 120: 497.
51. Frank M, Seeberger L, O'Reilly R, (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306: 1940-1943.
52. Sober S, Brainard M, (2009) Adult birdsong is actively maintained by error correction. Nature Neurosci 12: 927-931.
53. Edelman G (1987) Neural Darwinism: The theory of neuronal group selection. New York: Basic Books New York.
54. Kao M, Doupe A, Brainard M, (2005) Contributions of an avian basal ganglia-forebrain circuit to real-time modulation of song. Nature 433: 638-643.
55. Tumer E, Brainard M, (2007) Performance variability enables adaptive plasticity of crystallized adult birdsong. Nature 450: 1240-1244.
56. Shima K, Tanji J, (1998) Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282: 1335-1338.
57. Bush G, Vogt B, Holmes J, Dale A, Greve D, et al. (2002) Dorsal anterior cingulate cortex: a role in reward-based decision making. Proceedings of the National Academy of Sciences 99: 523.
58. Daw N, O'Doherty J, Dayan P, Seymour B, Dolan R, (2006) Cortical substrates for exploratory decisions in humans. Nature 441: 876-879.
59. Koechlin E, Hyafil A, (2007) Anterior prefrontal function and the limits of human decision-making. Science 318: 594-598.
60. Mufson E, Pandya D, (1984) Some observations on the course and composition of the cingulum bundle in the rhesus monkey. The Journal of Comparative Neurology 225: 31-43.