Solving the distal reward problem through linkage of STDP and dopamine signaling

Cerebral Cortex

Volumn 17, Issue 10, 2007, Pages 2443-2452

  Izhikevich, Eugene M ^a  

^a NEUROSCIENCES INSTITUTE   (United States)

Author keywords

Classical conditioning; Dopamine; Instrumental conditioning; Reward; Simulation; Spike timing dependent plasticity (STDP)

Indexed keywords

DOPAMINE;

ARTICLE; DOPAMINERGIC NERVE CELL; INSTRUMENTAL CONDITIONING; LONG TERM POTENTIATION; NERVE CELL EXCITABILITY; NERVE CELL NETWORK; NERVE CELL PLASTICITY; NEUROMODULATION; PRIORITY JOURNAL; REINFORCEMENT; REWARD; THEORETICAL STUDY;

ANIMALS; CONDITIONING, CLASSICAL; DOPAMINE; HUMANS; KINETICS; NEURONAL PLASTICITY; PROBLEM SOLVING; REWARD; SIGNAL TRANSDUCTION;

EID: 34948906745     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhl152     Document Type: Article

References (61)

1. Ahissar E, Vaadia E, Ahissar M, Bergman H, Arieli A, Abeles M. 1992. Hebbian plasticity in the cortex of the adult monkey depends on the behavioral context. Science. 257:1412-1415.
2. Au-Young SM, Shen H, Yang CR. 1999. Medial prefrontal cortical output neurons to the ventral tegmental area VTA and their responses to burst-patterned stimulation of the VTA: neuroanatomical and in vivo electrophysiological analyses. Synapse. 34:245-255.
3. Barad M, Bourchouladze R, Winder DG, Golan H, Kandel ER. 1998. Rolipram, a type IV-specific phosphodiesterase inhibitor, facilitates the establishment of long-lasting long-term potentiation and improves memory. Proc Natl Acad Sci USA. 95:15020-15025.
4. Barto AG, Sutton RS, Anderson CW. 1983. Neuronlike elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern. 13:835-846.
5. Bi GQ, Poo MM. 1998. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and post-synaptic cell type. J Neurosci. 18:10464-10472.
6. Calabresi P, Gubellini P, Centonze D, Picconi B, Bernardi G, Chergui K, Svenningsson P, Fienberg AA, Greengard P. 2000. Dopamine and cAMP-regulated phosphoprotein 32 kDa controls both striatal long-term depression and long-term potentiation, opposing forms of synaptic plasticity. J Neurosci. 20(22):8443-8451.
7. Cass WA, Gerhardt GA. 1995. In vivo assessment of dopamine uptake in rat medial prefrontal cortex: comparison with dorsal striatum and nucleus accumbens. J Neurochem. 65:201-207.
8. Centonze D, Gubellini P, Picconi B, Giacomini P, Calabresi P, Bernardi G. 1999. Unilateral dopamine denervation blocks corticostriatal LTP. J Neurophysiol. 82:3575-3579.
9. Choi S, Lovinger DM. 1997. Decreased probability of neurotransmitter release underlies striatal long-term depression and postnatal development of corticostriatal synapses. Proc Natl Acad Sci USA. 94:2665-2670.
10. Connors BW, Gutnick MJ. 1990. Intrinsic firing patterns of diverse neocortical neurons. Trends Neurosci. 13:99-104.
11. Dayan P, Abbott LF. 2001. Theoretical neuroscience: computational and mathematical modeling of neural systems. Cambridge (MA): The MIT Press.
12. Drew PJ, Abbott LF. 2006. Extending the effects of spike-timingdependent plasticity to behavioral timescales. Proc Natl Acad Sci USA. 103:8876-8881.
13. Frey U, Morris RG. 1997. Synaptic tagging and long-term potentiation. Nature. 385:533-536.
14. Frey U, Schroeder H, Matthies H. 1990. Dopaminergic antagonists prevent long-term maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Res. 522:69-75.
15. Fusi S, Drew PJ, Abbott LF. 2005. Cascade models of synaptically stored memories. Neuron. 45:599-611.
16. Garris PA, Ciolkowski IL, Pastore P, Wightman RM. 1994. Efflux of dopamine from the synaptic cleft in the nucleus accumbens of the rat brain. J Neurosci. 14:6084-6093.
17. Gerstner W, Kempter R, van Hemmen JL, Wagner H. 1996. A neuronal learning rule for sub-millisecond temporal coding. Nature. 383:76-78.
18. Gurden H, Takita M, Jay TM. 2000. Essential role of D1 but not D2 receptors in the NMDA receptor-dependent long-term potentiation at hippocampal-prefrontal cortex synapses in vivo. J Neurosci. 20:RC106.
19. Hasselmo ME. 2005. A model of prefrontal cortical mechanisms for goal-directed behavior. J Cogn Neurosci. 17:1-14.
20. Houk JC, Adams JL, Barto AG. 1995. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk JC, Davis JL, Beiser DG, editors. Models of information processing in the basal ganglia. Cambridge (MA): The MIT Press. p. 249-270.
21. Houk JC, Davis JL, Beiser DG. 1995. Models of information processing in the basal ganglia. Cambridge (MA): The MIT Press.
22. Hull CL. 1943. Principles of behavior. New York: Appleton-Century.
23. Impey S, Mark M, Villacres EC, Poser S, Chavkin C, Storm DR. 1996. Induction of CRE-mediated gene expression by stimuli that generate long-lasting LTP in area CA1 of the hippocampus. Neuron. 16:973-982.
24. Izhikevich EM. 2006. Polychronization: computation with spikes. Neural Comput. 18:245-282.
25. Izhikevich EM, Gally JA, Edelman GM. 2004. Spike-timing dynamics of neuronal groups. Cereb Cortex. 14:933-944.
26. Jay TM, Burette F, Laroche S. 1996. Plasticity of the hippocampalprefrontal cortex synapses. J Physiol Paris. 90:361-366.
27. Kempter R, Gerstner W, van Hemmen JL. 1999a. Hebbian learning and spiking neurons. Phys Rev E. 59:4498-4514.
28. Kempter R, Gerstner W, van Hemmen JL. 1999b. Spike-Based Compared to Rate-Based Hebbian Learning NIPS Conference; 1998 December; Denver. In: Kearns MS, Solla SA, Cohn DA, editors. Advances in neural information processing systems 11. Cambridge (MA): The MIT Press. p. 125-131.
29. Klopf AH. 1982. The hedonistic neuron. Washington (DC): Hemisphere.
30. Koene RA, Hasselmo ME. 2005. An integrate-and-fire model of prefrontal cortex neuronal activity during performance of goal-directed decision making. Cereb Cortex. 15(12):1964-1981.
31. Lauwereyns J, Watanabe K, Coe B, Hikosaka O. 2002. A neural correlate of response bias in monkey caudate nucleus. Nature. 418:413-417.
32. Levy WB, Steward O. 1983. Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus. Neurosci. 8:791-797.
33. Lisman J. 1989. A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proc Natl Acad Sci USA. 86:9574-9578.
34. Ljungberg T, Apicella P, Schultz W. 1992. Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol. 67:145-163.
35. Markram H, Lubke J, Frotscher M, Sakmann B. 1997. Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science. 275:213-215.
36. Minsky ML. 1963. Steps toward artificial intelligence. In: Feigenbaum EA, Feldman J, editors. Computers and thought. New York: McGraw-Hill. p. 406-450.
37. Montague PR, Dayan P, Sejnowski TJ. 1996. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci. 16:1936-1947.
38. Montague PR, McClure SM, Balfwin PR, Phillips PEM, Budygin EA, Stuber GD, Kilpatrick MR, Wightman RM. 2004. Dynamic gain control of dopamine delivery in freely moving animals. J Neurosci. 24:1754-1759.
39. Otani S, Daniel H, Roisin MP, Crepel F. 2003. Dopaminergic modulation of long-term synaptic plasticity in rat prefrontal neurons. Cereb Cortex. 13:1251-1256.
40. Otmakhova NA, Lisman JE. 1996. D1/D5 dopamine receptor activation increases the magnitude of early long-term potentiation at CA1 hippocampal synapses. J Neurosci. 16:7478-7486.
41. Otmakhova NA, Lisman JE. 1998. D1/D5 dopamine receptor inhibit depotentiation at CA1 synapses via cAMP-dependent mechanism. J Neurosci. 18:1270-1279.
42. Pan W-X, Schmidt R, Wickens JR, Hyland BI. 2005. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward learning network. J Neurosci. 25:6235-6242.
43. Pavlov IP. 1927. Conditioned reflexes. Oxford: Oxford University Press.
44. Pasupathy A, Miller BK. 2005. Different time courses of learning-related activity in the prefrental cortex and basal ganglia. Nature. 433:873-876.
45. Rao RPN, Sejnowski TJ. 2001. Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Comput. 12:2221-2237.
46. Schultz W. 1998. Predictive reward signal of dopamine neurons. J Neurophysiol. 80:1-27.
47. Schultz W. 2002. Getting formal with dopamine and reward. Neuron. 36:241-263.
48. Schultz W. Forthcoming 2007a. Reward. Scholarpedia.
49. Schultz W. Forthcoming 2007b. Reward signals. Scholarpedia.
50. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science. 275:1593-1599.
51. Seamans JK, Yang CR. 2004. The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog Neurobiol. 74:1-57.
52. Seung HS. 2003. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron. 40:1063-1073.
53. Song S, Miller KD, Abbott LF. 2000. Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nat Neurosci. 3:919-926.
54. Suri RE, Schultz W. 2001. Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13:841-862.
55. Sutton RS. 1988. Learning to predict by the methods of temporal differences. Mach Learn. 3:9-44.
56. Sutton RS, Barto AG. 1998. Reinforcement learning: an introduction. Cambridge (MA): The MIT Press.
57. Swadlow HA. 1990. Efferent neurons and suspected interneurons in S-1 forelimb representation of the awake rabbit: receptive fields and axonal properties. J Neurophysiol. 63:1477-1498.
58. Swadlow HA. 1994. Efferent neurons and suspected interneurons in motor cortex of the awake rabbit: axonal properties, sensory receptive fields, and subthreshold synaptic inputs. J Neurophysiol. 71:437-453.
59. Watanabe M. 1996. Reward expectancy in primate prefrontal neurons. Nature. 382:629-632.
60. Wightman RM, Zimmerman JB. 1990. Control of dopaine extracellular concentration in rat striatum by impulse flow and update. Brain Res Brain Res Rev. 15:135-144.
61. Worgotter F, Porr B. 2005. Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms. Neural Comput. 15:245-319.