Learning reward timing in cortex through reward dependent expression of synaptic plasticity

Proceedings of the National Academy of Sciences of the United States of America

Volumn 106, Issue 16, 2009, Pages 6826-6831

  Gavornik, Jeffrey P ^a,b   Shuler, Marshall G Hussain ^c   Loewenstein, Yonatan ^d   Bear, Mark F ^e   Shouval, Harel Z ^a  

^a UNIVERSITY OF TEXAS MEDICAL SCHOOL   (United States)

^b The University of Texas at Austin   (United States)

^c JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

^d HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^e HOWARD HUGHES MEDICAL INSTITUTE   (United States)

Author keywords

Reinforcment learning; Visual cortex

Indexed keywords

ARTICLE; BRAIN CORTEX; BRAIN REGION; CELL ACTIVITY; HUMAN; LEARNING; NERVE CELL; NERVE CELL PLASTICITY; PREDICTION; PRIORITY JOURNAL; REWARD; SENSORY CORTEX; VISUAL CORTEX; ACTION POTENTIAL; BIOLOGICAL MODEL; NERVE CELL NETWORK; PHYSIOLOGY; STATISTICS; TIME;

ACTION POTENTIALS; CEREBRAL CORTEX; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NERVE NET; NEURONAL PLASTICITY; NEURONS; REWARD; STOCHASTIC PROCESSES; TIME FACTORS;

EID: 66149143188     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.0901835106     Document Type: Article

References (55)

1. Mauk MD, Buonomano DV (2004) The neural basis of temporal processing. Annu Rev Neurosci 27:307-340.
2. Lewis PA, Miall RC (2006) Remembering the time: A continuous clock. Trends Cogn Sci 10:401-406.
3. Staddon JE (2005) Interval timing: Memory, not a clock. Trends Cogn Sci 9:312-314.
4. Meck WH (2005) Neuropsychology of timing and time perception. Brain Cogn 58:1-8.
5. Rammsayer TH (1999) Neuropharmacological evidence for different timing mechanisms in humans. Q J Exp Psychol B 52:273-286.
6. Buonomano DV, Karmarkar UR (2002) How do we tell time? Neuroscientist 8:42-51.
7. Merchant H, Zarco W, Prado L (2008) Do we have a common mechanism for measuring time in the hundreds of millisecond range? Evidence from multiple-interval timing tasks. J Neurophysiol 99:939-949.
8. Moshitch D, Las L, Ulanovsky N, Bar-Yosef O, Nelken I (2006) Responses of neurons in primary auditory cortex (A1) to pure tones in the halothane-anesthetized cat. J Neu-rophysiol 95:3756-3769.
9. Shuler MG, Bear MF (2006) Reward timing in the primary visual cortex. Science 311:1606-1609.
10. Super H, Spekreijse H, Lamme VA (2001) A neural correlate of working memory in the monkey primary visual cortex. Science 293:120-124.
11. Pantoja J, et al. (2007) Neuronal activity in the primary somatosensory thalamocortical loop is modulated by reward contingency during tactile discrimination. J Neurosci 27:10608-10620.
12. Pleger B, Blankenburg F, Ruff CC, Driver J, Dolan RJ (2008) Reward facilitates tactile judgments and modulates hemodynamic responses in human primary somatosensory cortex. J Neurosci 28:8161-8168.
13. Weinberger NM (2007) Associative representational plasticity in the auditory cortex: A synthesis of two disciplines. Learn Mem 14:1-16.
14. Johnson RR, Burkhalter A (1996) Microcircuitry of forward and feedback connections within rat visual cortex. J Comp Neurol 368:383-398.
15. Budd JM (1998) Extrastriate feedback to primary visual cortex in primates: A quantitative analysis of connectivity. Proc Biol Sci 265:1037-1044.
16. Dayan P, Abbott LF (2001) Theoretical Neuroscience: Computational Modeling of Neural Systems (MIT Press, Cambridge, MA).
17. Machens CK, Romo R, Brody CD (2005) Flexible control of mutual inhibition: A neural model of two-interval discrimination. Science 307:1121-1124.
18. Matell MS, King GR, Meck WH (2004) Differential modulation of clock speed by the administration of intermittent versus continuous cocaine. Behav Neurosci 118:150-156.
19. Rescorla RA, Wagner AR (1972) Classical Conditioning II: Current Research and Theory , eds Black AH, Prokasy WF (Appleton-Century-Crofts, New York), pp 64-69.
20. Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).
21. Frey U, Morris RG (1997) Synaptic tagging and long-term potentiation. Nature 385:533-536.
22. Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17:2443-2452.
23. Loewenstein Y (2008) Robustness of learning that is based on covariance-driven synaptic plasticity. PLoS Comput Biol 4:e1000007.
24. Loewenstein Y, Seung HS (2006) Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc Natl Acad Sci USA 103:15224-15229.
25. Brody CD, Romo R, Kepecs A (2003) Basic mechanisms for graded persistent activity: Discrete attractors, continuous attractors, and dynamic representations. Curr Opin Neurobiol 13:204-211.
26. Seung HS (1996) How the brain keeps the eyes still. Proc Natl Acad Sci USA 93:13339-13344.
27. Wang XJ (2001) Synaptic reverberation underlying mnemonic persistent activity. Trends Neurosci 24:455-463.
28. Amit DJ, Brunel N (1997) Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cereb Cortex 7:237-252.
29. Watanabe M (1996) Reward expectancy in primate prefrontal neurons. Nature 382:629-632.
30. Miller P, Brody CD, Romo R, Wang XJ (2003) A recurrent network model of somato-sensory parametric working memory in the prefrontal cortex. Cereb Cortex 13:1208-1218.
31. Reutimann J, Yakovlev V, Fusi S, Senn W (2004) Climbing neuronal activity as an event-based cortical representation of time. J Neurosci 24:3295-3303.
32. Barbieri F, Brunel N (2007) Irregular persistent activity induced by synaptic excitatory feedback. Front Comput Neurosci 1:5.
33. Wang XJ (1999) Synaptic basis of cortical persistent activity: The importance of NMDA receptors to working memory. J Neurosci 19:9587-9603.
34. Karmarkar UR, Buonomano DV (2007) Timing in the absence of clocks: Encoding time in neural network states. Neuron 53:427-438.
35. Maass W, Natschlager T, Markram H (2002) Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Comput 14:2531-2560.
36. White OL, Lee DD, Sompolinsky H (2004) Short-term memory in orthogonal neural networks. Phys Rev Lett 92:148102.
37. Bernander O, Douglas RJ, Martin KA, Koch C (1991) Synaptic background activity influences spatiotemporal integration in single pyramidal cells. Proc Natl Acad Sci USA 88:11569-11573.
38. Tsodyks M, Kenet T, Grinvald A, Arieli A (1999) Linking spontaneous activity of single cortical neurons and the underlying functional architecture. Science 286:1943-1946.
39. Bear MF, Singer W (1986) Modulation of visual cortical plasticity by acetylcholine and noradrenaline. Nature 320:172-176.
40. Kilgard MP, Merzenich MM (1998) Cortical map reorganization enabled by nucleus basalis activity. Science 279:1714-1718.
41. Kirkwood A, Rozas C, Kirkwood J, Perez F, Bear MF (1999) Modulation of long-term synaptic depression in visual cortex by acetylcholine and norepinephrine. J Neurosci 19:1599-1609.
42. Seol GH, et al. (2007) Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron 55:919-929.
43. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1-27.
44. Gibbon J, Malapani C, Dale CL, Gallistel C (1997) Toward a neurobiology of temporal cognition: Advances and challenges. Curr Opin Neurobiol 7:170-184.
45. Meck WH (1996) Neuropharmacology of timing and time perception. Brain Res Cogn Brain Res 3:227-242.
46. Papadopoulos GC, Parnavelas JG, Buijs RM (1989) Light and electron microscopic immunocytochemical analysis of the dopamine innervation of the rat visual cortex. J Neurocytol 18:303-310.
47. Sullivan MJ, et al. (2003) Lesions of the diagonal band of broca enhance drinking in the rat. J Neuroendocrinol 15:907-915.
48. Whishaw IQ, O'Connor WT, Dunnett SB (1985) Disruption of central cholinergic systems in the rat by basal forebrain lesions or atropine: Effects on feeding, sensorimotor behaviour, locomotor activity and spatial navigation. Behav Brain Res 17:103-115.
49. Mechawar N, Cozzari C, Descarries L (2000) Cholinergic innervation in adult rat cerebral cortex: A quantitative immunocytochemical description. J Comp Neurol 428:305-318.
50. Wenk H, Bigl V, Meyer U (1980) Cholinergic projections from magnocellular nuclei of the basal forebrain to cortical areas in rats. Brain Res 2:295-316.
51. Ben-Chaim Y, et al. (2006) Movement of ''gating charge'' is coupled to ligand binding in a G-protein-coupled receptor. Nature 444:106-109.
52. Hu H, et al. (2007) Emotion enhances learning via norepinephrine regulation of AMPA-receptor trafficking. Cell 131:160-173.
53. Schultz W (1999) The reward signal of midbrain dopamine neurons. News Physiol Sci 14:249-255.
54. Driska SP, Stein PG, Porter R (1989) Myosin dephosphorylation during rapid relaxation of hog carotid artery smooth muscle. Am J Physiol 256:C315-321.
55. King MM, et al. (1984) Mammalian brain phosphoproteins as substrates for calcineurin. J Biol Chem 259:8080-8083.