Twenty-Five Lessons from Computational Neuromodulation

Neuron

Volumn 76, Issue 1, 2012, Pages 240-256

  Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ACETYLCHOLINE; DOPAMINE; NORADRENALIN; SEROTONIN;

ALGORITHM; AVOIDANCE BEHAVIOR; BAYES THEOREM; COMPUTER SIMULATION; CONDITIONING; DECISION MAKING; DOPAMINERGIC SYSTEM; HUMAN; LEARNING; MOTIVATION; NERVE CELL PLASTICITY; NEUROANATOMY; NEUROMODULATION; PREFRONTAL CORTEX; PRIORITY JOURNAL; PROTEIN SYNTHESIS; PUNISHMENT; REINFORCEMENT; REVIEW; REWARD; THREAT; UNCERTAINTY; WORKING MEMORY;

ALGORITHMS; ANIMALS; BRAIN; DECISION MAKING; HUMANS; MODELS, NEUROLOGICAL; NEUROTRANSMITTER AGENTS;

EID: 84867124100     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2012.09.027     Document Type: Review

References (237)

1. Adcock R.A., Thangavel A., Whitfield-Gabrieli S., Knutson B., Gabrieli J.D.E. Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron 2006, 50:507-517.
2. Agnati L.F., Leo G., Zanardi A., Genedani S., Rivera A., Fuxe K., Guidolin D. Volume transmission and wiring transmission from cellular to molecular networks: history and perspectives. Acta Physiol. (Oxf.) 2006, 187:329-344.
3. Alex K.D., Pehek E.A. Pharmacologic mechanisms of serotonergic regulation of dopamine neurotransmission. Pharmacol. Ther. 2007, 113:296-320.
4. Aosaki T., Tsubokawa H., Ishida A., Watanabe K., Graybiel A.M., Kimura M. Responses of tonically active neurons in the primate's striatum undergo systematic changes during behavioral sensorimotor conditioning. J. Neurosci. 1994, 14:3969-3984.
5. Aosaki T., Kimura M., Graybiel A.M. Temporal and spatial characteristics of tonically active neurons of the primate's striatum. J. Neurophysiol. 1995, 73:1234-1252.
6. Arnsten A.F.T. Catecholamine influences on dorsolateral prefrontal cortical networks. Biol. Psychiatry 2011, 69:e89-e99.
7. Aston-Jones G., Cohen J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 2005, 28:403-450.
8. Aston-Jones G., Chiang C., Alexinsky T. Discharge of noradrenergic locus coeruleus neurons in behaving rats and monkeys suggests a role in vigilance. Prog. Brain Res. 1991, 88:501-520.
9. Aston-Jones G., Rajkowski J., Kubiak P., Alexinsky T. Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task. J. Neurosci. 1994, 14:4467-4480.
10. Aston-Jones G., Rajkowski J., Kubiak P. Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 1997, 80:697-715.
11. Aston-Jones G., Smith R.J., Sartor G.C., Moorman D.E., Massi L., Tahsili-Fahadan P., Richardson K.A. Lateral hypothalamic orexin/hypocretin neurons: A role in reward-seeking and addiction. Brain Res. 2010, 1314:74-90.
12. Balleine B.W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 2005, 86:717-730.
13. Balleine B. Sensation, incentive learning, and the motivational control of goal-directed action. Neurobiology of Sensation and Reward, Chapter 13 2011, CRC Press, Boca Raton, FL. J. Gottfried (Ed.).
14. Bari A., Mar A.C., Theobald D.E., Elands S.A., Oganya K.C.N.A., Eagle D.M., Robbins T.W. Prefrontal and monoaminergic contributions to stop-signal task performance in rats. J. Neurosci. 2011, 31:9254-9263.
15. Barto A. Adaptive critics and the basal ganglia. Models of Information Processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge, MA. J. Houk, J. Davis, D. Beiser (Eds.).
16. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
17. Behrens T.E.J., Woolrich M.W., Walton M.E., Rushworth M.F.S. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
18. Beninger R.J. The role of dopamine in locomotor activity and learning. Brain Res. 1983, 287:173-196.
19. Bentley P., Husain M., Dolan R.J. Effects of cholinergic enhancement on visual stimulation, spatial attention, and spatial working memory. Neuron 2004, 41:969-982.
20. Berridge K.C. Motivation concepts in behavioral neuroscience. Physiol. Behav. 2004, 81:179-209.
21. Berridge C.W. Noradrenergic modulation of arousal. Brain Res. Brain Res. Rev. 2008, 58:1-17.
22. Berridge K.C., Robinson T.E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?. Brain Res. Brain Res. Rev. 1998, 28:309-369.
23. Bland S.T., Hargrave D., Pepin J.L., Amat J., Watkins L.R., Maier S.F. Stressor controllability modulates stress-induced dopamine and serotonin efflux and morphine-induced serotonin efflux in the medial prefrontal cortex. Neuropsychopharmacology 2003, 28:1589-1596.
24. Bland S.T., Twining C., Watkins L.R., Maier S.F. Stressor controllability modulates stress-induced serotonin but not dopamine efflux in the nucleus accumbens shell. Synapse 2003, 49:206-208.
25. Blier P. The pharmacology of putative early-onset antidepressant strategies. Eur. Neuropsychopharmacol. 2003, 13:57-66.
26. Bogacz R., Brown E., Moehlis J., Holmes P., Cohen J.D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 2006, 113:700-765.
27. Bolles R.C. Species-specific defense reactions and avoidance learning. Psychol. Rev. 1970, 77:32-48.
28. Boureau Y.-L., Dayan P. Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 2011, 36:74-97.
29. Bouret S., Sara S.J. Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur. J. Neurosci. 2004, 20:791-802.
30. Bouret S., Sara S.J. Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci. 2005, 28:574-582.
31. Bremner J.D., Krystal J.H., Southwick S.M., Charney D.S. Noradrenergic mechanisms in stress and anxiety: I. Preclinical studies. Synapse 1996, 23:28-38.
32. Bremner J.D., Krystal J.H., Southwick S.M., Charney D.S. Noradrenergic mechanisms in stress and anxiety: II. Clinical studies. Synapse 1996, 23:39-51.
33. Briand L.A., Gritton H., Howe W.M., Young D.A., Sarter M. Modulators in concert for cognition: modulator interactions in the prefrontal cortex. Prog. Neurobiol. 2007, 83:69-91.
34. Brischoux F., Chakraborty S., Brierley D.I., Ungless M.A. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc. Natl. Acad. Sci. USA 2009, 106:4894-4899.
35. Bromberg-Martin E.S., Matsumoto M., Hikosaka O. Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 2010, 68:815-834.
36. Brown E., Gao J., Holmes P., Bogacz R., Gilzenrat M., Cohen J. Simple neural networks that optimize decisions. Int. J. Bifurcat. Chaos 2005, 15:803-826.
37. Buzsáki G. Theta oscillations in the hippocampus. Neuron 2002, 33:325-340.
38. Cabib S., Puglisi-Allegra S. Opposite responses of mesolimbic dopamine system to controllable and uncontrollable aversive experiences. J. Neurosci. 1994, 14:3333-3340.
39. Cabib S., Puglisi-Allegra S. The mesoaccumbens dopamine in coping with stress. Neurosci. Biobehav. Rev. 2012, 36:79-89.
40. Cardinal R.N. Neural systems implicated in delayed and probabilistic reinforcement. Neural Netw. 2006, 19:1277-1301.
41. Clark L., Cools R., Robbins T.W. The neuropsychology of ventral prefrontal cortex: decision making and reversal learning. Brain Cogn. 2004, 55:41-53.
42. Clarke H.F., Hill G.J., Robbins T.W., Roberts A.C. Dopamine, but not serotonin, regulates reversal learning in the marmoset caudate nucleus. J. Neurosci. 2011, 31:4290-4297.
43. Clayton E.C., Rajkowski J., Cohen J.D., Aston-Jones G. Phasic activation of monkey locus ceruleus neurons by simple decisions in a forced-choice task. J. Neurosci. 2004, 24:9914-9920.
44. Cohen M.X., Frank M.J. Neurocomputational models of basal ganglia function in learning, memory and choice. Behav. Brain Res. 2009, 199:141-156.
45. Cohen J.D., Servan-Schreiber D. A theory of dopamine function and its role in cognitive deficits in schizophrenia. Schizophr. Bull. 1993, 19:85-104.
46. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482:85-88.
47. Collins A., Koechlin E. Reasoning, learning, and creativity: frontal lobe function and human decision making. PLoS Biol. 2012, 10:e1001293.
48. Cools R. Dopaminergic control of the striatum for high-level cognition. Curr. Opin. Neurobiol. 2011, 21:402-407.
49. Cools R., Nakamura K., Daw N.D. Serotonin and dopamine: unifying affective, activational, and decision functions. Neuropsychopharmacology 2011, 36:98-113.
50. Cooper J., Bloom F., Roth R. The Biochemical Basis of Neuropharmacology 2002, OUP, New York. 8th Edition.
51. Crockett M.J., Clark L., Robbins T.W. Reconciling the role of serotonin in behavioral inhibition and aversion: acute tryptophan depletion abolishes punishment-induced inhibition in humans. J. Neurosci. 2009, 29:11993-11999.
52. Crockett M.J., Clark L., Apergis-Schoute A.M., Morein-Zamir S., Robbins T.W. Serotonin modulates the effects of pavlovian aversive predictions on response vigor. Neuropsychopharmacology 2012, 37:2244-2252.
53. Damasio A. Descartes' Error: Emotion, Reason, and the Human Brain 1994, GP Putnam, New York, NY.
54. Daw N.D., Kakade S., Dayan P. Opponent interactions between serotonin and dopamine. Neural Netw. 2002, 15:603-616.
55. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
56. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
57. Dayan P. How to set the switches on this thing. Curr. Opin. Neurobiol. 2012, Published online June 15, 2012. 10.1016/j.conb.2012.05.011.
58. Dayan P. Instrumental vigour in punishment and reward. Eur. J. Neurosci. 2012, 35:1152-1168.
59. Dayan P., Huys Q.J.M. Serotonin, inhibition, and negative mood. PLoS Comput. Biol. 2008, 4:e4.
60. Dayan P., Yu A.J. Phasic norepinephrine: a neural interrupt signal for unexpected events. Network 2006, 17:335-350.
61. Dayan P., Kakade S., Montague P.R. Learning and selective attention. Nat. Neurosci. 2000, 3(Suppl):1218-1223.
62. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Netw. 2006, 19:1153-1160.
63. De Dreu C.K.W. Oxytocin modulates cooperation within and competition between groups: an integrative review and research agenda. Horm. Behav. 2012, 61:419-428.
64. de Wit S., Standing H.R., Devito E.E., Robinson O.J., Ridderinkhof K.R., Robbins T.W., Sahakian B.J. Reliance on habits at the expense of goal-directed control following dopamine precursor depletion. Psychopharmacology (Berl.) 2012, 219:621-631.
65. Deakin J.F.W. Roles of brain serotonergic neurons in escape, avoidance and other behaviors. J. Psychopharmacol. 1983, 43:563-577.
66. Deneve S. Bayesian spiking neurons II: learning. Neural Comput. 2008, 20:118-145.
67. Devauges V., Sara S.J. Activation of the noradrenergic system facilitates an attentional shift in the rat. Behav. Brain Res. 1990, 39:19-28.
68. Dickinson A., Balleine B. The role of learning in the operation of motivational systems. Stevens' Handbook of Experimental Psychology 2002, Volume 3:497-534. Wiley, New York. R. Gallistel (Ed.).
69. Dickinson A., Smith J., Mirenowicz J. Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav. Neurosci. 2000, 114:468-483.
70. Doya K. Metalearning, neuromodulation, and emotion. Affective Minds 2000, 1:101-104.
71. Doya K. Metalearning and neuromodulation. Neural Netw. 2002, 15:495-506.
72. Doya K., Dayan P., Hasselmo M. Special issue: computational models of neuromodulation. Neural Netw. 2002, 16:5-6.
73. Bayesian brain: Probabilistic approaches to neural coding 2007, The MIT Press, Cambridge, MA. K. Doya, S. Ishii, A. Pouget, R. Rao (Eds.).
74. Durstewitz D., Seamans J.K., Sejnowski T.J. Dopamine-mediated stabilization of delay-period activity in a network model of prefrontal cortex. J. Neurophysiol. 2000, 83:1733-1750.
75. Ellison G. Chemical systems of the brain and evolution. Brain, Behaviour and Evolution 1979, 78-98. Methuen, London. D. Oakley, H. Plotkin (Eds.).
76. Estes W. Discriminative conditioning. I. a discriminative property of conditioned anticipation. J. Exp. Psychol. 1943, 32:150-155.
77. Fermin A., Yoshida T., Ito M., Yoshimoto J., Doya K. Evidence for model-based action planning in a sequential finger movement task. J. Mot. Behav. 2010, 42:371-379.
78. Fiser J., Berkes P., Orbán G., Lengyel M. Statistically optimal perception and learning: from behavior to neural representations. Trends Cogn. Sci. 2010, 14:119-130.
79. Fletcher P.J. Effects of combined or separate 5,7-dihydroxytryptamine lesions of the dorsal and median raphe nuclei on responding maintained by a DRL 20s schedule of food reinforcement. Brain Res. 1995, 675:45-54.
80. Fletcher P.J., Korth K.M., Chambers J.W. Selective destruction of brain serotonin neurons by 5,7-dihydroxytryptamine increases responding for a conditioned reward. Psychopharmacology (Berl.) 1999, 147:291-299.
81. Floresco S.B., Magyar O. Mesocortical dopamine modulation of executive functions: beyond working memory. Psychopharmacology (Berl.) 2006, 188:567-585.
82. Frank M.J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 2005, 17:51-72.
83. Frank M.J., O'Reilly R.C. A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav. Neurosci. 2006, 120:497-517.
84. Frank M.J., Seeberger L.C., O'reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
85. Frank M.J., Moustafa A.A., Haughey H.M., Curran T., Hutchison K.E. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. USA 2007, 104:16311-16316.
86. Frank M.J., Doll B.B., Oas-Terpstra J., Moreno F. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci. 2009, 12:1062-1068.
87. Gallistel C.R., Gibbon J. Time, rate, and conditioning. Psychol. Rev. 2000, 107:289-344.
88. Gan J.O., Walton M.E., Phillips P.E.M. Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat. Neurosci. 2010, 13:25-27.
89. Garris P.A., Kilpatrick M., Bunin M.A., Michael D., Walker Q.D., Wightman R.M. Dissociation of dopamine release in the nucleus accumbens from intracranial self-stimulation. Nature 1999, 398:67-69.
90. Gerfen C.R., Engber T.M., Mahan L.C., Susel Z., Chase T.N., Monsma F.J., Sibley D.R. D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. Science 1990, 250:1429-1432.
91. Gil Z., Connors B.W., Amitai Y. Differential regulation of neocortical synapses by neuromodulators and activity. Neuron 1997, 19:679-686.
92. Gilzenrat M.S., Nieuwenhuis S., Jepma M., Cohen J.D. Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cogn. Affect. Behav. Neurosci. 2010, 10:252-269.
93. Giorgi O., Lecca D., Piras G., Driscoll P., Corda M.G. Dissociation between mesocortical dopamine release and fear-related behaviours in two psychogenetically selected lines of rats that differ in coping strategies to aversive conditions. Eur. J. Neurosci. 2003, 17:2716-2726.
94. Glimcher P. Foundations of Neuroeconomic Analysis 2010, Oxford Univ Press, New York, NY.
95. Goard M., Dan Y. Basal forebrain activation enhances cortical coding of natural scenes. Nat. Neurosci. 2009, 12:1444-1449.
96. Gold J.I., Shadlen M.N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 2002, 36:299-308.
97. Goto Y., Grace A.A. Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat. Neurosci. 2005, 8:805-812.
98. Gottlieb J., Balan P. Attention as a decision in information space. Trends Cogn. Sci. 2010, 14:240-248.
99. Grace A.A. Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience 1991, 41:1-24.
100. Gray J.A., McNaughton N. The Neuropsychology of Anxiety 2003, OUP, New York. Second Edition.
101. Grottick A.J., Fletcher P.J., Higgins G.A. Studies to investigate the role of 5-HT(2C) receptors on cocaine- and food-maintained behavior. J. Pharmacol. Exp. Ther. 2000, 295:1183-1191.
102. Guitart-Masip M., Beierholm U.R., Dolan R., Duzel E., Dayan P. Vigor in the face of fluctuating rates of reward: an experimental examination. J. Cogn. Neurosci. 2011, 23:3933-3938.
103. Guitart-Masip M., Chowdhury R., Sharot T., Dayan P., Duzel E., Dolan R.J. Action controls dopaminergic enhancement of reward representations. Proc. Natl. Acad. Sci. USA 2012, 109:7511-7516.
104. Harmer C.J., Goodwin G.M., Cowen P.J. Why do antidepressants take so long to work? A cognitive neuropsychological model of antidepressant drug action. Br. J. Psychiatry 2009, 195:102-108.
105. Harris-Warrick R.M. Neuromodulation and flexibility in central pattern generator networks. Curr. Opin. Neurobiol. 2011, 21:685-692.
106. Hasselmo M.E. The role of acetylcholine in learning and memory. Curr. Opin. Neurobiol. 2006, 16:710-715.
107. Hasselmo M.E., Bower J.M. Acetylcholine and memory. Trends Neurosci. 1993, 16:218-222.
108. Hasselmo M.E., Sarter M. Modes and models of forebrain cholinergic neuromodulation of cognition. Neuropsychopharmacology 2011, 36:52-73.
109. Hasselmo M.E., Bodelón C., Wyble B.P. A proposed function for hippocampal theta rhythm: separate phases of encoding and retrieval enhance reversal of prior learning. Neural Comput. 2002, 14:793-817.
110. Hazy T.E., Frank M.J., O'Reilly R.C. Banishing the homunculus: making working memory work. Neuroscience 2006, 139:105-118.
111. Herrero J.L., Roberts M.J., Delicato L.S., Gieselmann M.A., Dayan P., Thiele A. Acetylcholine contributes through muscarinic receptors to attentional modulation in V1. Nature 2008, 454:1110-1114.
112. Hershberger W.A. An approach through the looking-glass. Anim. Learn. Behav. 1986, 14:443-451.
113. Hervé D., Simon H., Blanc G., Lisoprawski A., Le Moal M., Glowinski J., Tassin J.P. Increased utilization of dopamine in the nucleus accumbens but not in the cerebral cortex after dorsal raphe lesion in the rat. Neurosci. Lett. 1979, 15:127-133.
114. Higley M.J., Soler-Llavina G.J., Sabatini B.L. Cholinergic modulation of multivesicular release regulates striatal synaptic potency and integration. Nat. Neurosci. 2009, 12:1121-1128.
115. Hitchcott P.K., Quinn J.J., Taylor J.R. Bidirectional modulation of goal-directed actions by prefrontal cortical dopamine. Cereb. Cortex 2007, 17:2820-2827.
116. Holland P.C., Gallagher M. Amygdala circuitry in attentional and representational processes. Trends Cogn. Sci. 1999, 3:65-73.
117. Homberg J.R. Serotonin and decision making processes. Neurosci. Biobehav. Rev. 2012, 36:218-236.
118. Honey G., Bullmore E. Human pharmacological MRI. Trends Pharmacol. Sci. 2004, 25:366-374.
119. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 2000, 96:651-656.
120. Hoyer D., Hannon J.P., Martin G.R. Molecular, pharmacological and functional diversity of 5-HT receptors. Pharmacol. Biochem. Behav. 2002, 71:533-554.
121. Huys Q.J.M., Dayan P. A Bayesian formulation of behavioral control. Cognition 2009, 113:314-328.
122. Huys Q.J.M., Moutoussis M., Williams J. Are computational models of any use to psychiatry?. Neural Netw. 2011, 24:544-551.
123. Huys Q.J.M., Eshel N., O'Nions E., Sheridan L., Dayan P., Roiser J.P. Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 2012, 8:e1002410.
124. Hyland B.I., Reynolds J.N.J., Hay J., Perk C.G., Miller R. Firing modes of midbrain dopamine cells in the freely moving rat. Neuroscience 2002, 114:475-492.
125. Ikemoto S., Panksepp J. The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res. Brain Res. Rev. 1999, 31:6-41.
126. Johnson J., Li W., Li J., Klopf A. A computational model of learned avoidance behavior in a one-way avoidance experiment. Adapt. Behav. 2001, 9:91.
127. Kaelbling L., Littman M., Cassandra A. Planning and acting in partially observable stochastic domains. Artif. Intell. 1998, 101:99-134.
128. Kakade S., Dayan P. Dopamine: generalization and bonuses. Neural Netw. 2002, 15:549-559.
129. Kaneko S., Hikida T., Watanabe D., Ichinose H., Nagatsu T., Kreitman R.J., Pastan I., Nakanishi S. Synaptic integration mediated by striatal cholinergic interneurons in basal ganglia function. Science 2000, 289:633-637.
130. Katz P.S. Neural mechanisms underlying the evolvability of behaviour. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2011, 366:2086-2099.
131. Keay K.A., Bandler R. Parallel circuits mediating distinct emotional coping reactions to different types of stress. Neurosci. Biobehav. Rev. 2001, 25:669-678.
132. Kehagia A.A., Murray G.K., Robbins T.W. Learning and cognitive flexibility: frontostriatal function and monoaminergic modulation. Curr. Opin. Neurobiol. 2010, 20:199-204.
133. Kimberg D.Y., D'Esposito M., Farah M.J. Effects of bromocriptine on human subjects depend on working memory capacity. Neuroreport 1997, 8:3581-3585.
134. Kimura F., Fukuda M., Tsumoto T. Acetylcholine suppresses the spread of excitation in the visual cortex revealed by optical recording: possible differential effect depending on the source of input. Eur. J. Neurosci. 1999, 11:3597-3609.
135. Kishida K.T., Sandberg S.G., Lohrenz T., Comair Y.G., Sáez I., Phillips P.E.M., Montague P.R. Sub-second dopamine detection in human striatum. PLoS ONE 2011, 6:e23291.
136. Klink R., Alonso A. Muscarinic modulation of the oscillatory and repetitive firing properties of entorhinal cortex layer II neurons. J. Neurophysiol. 1997, 77:1813-1828.
137. Koechlin E., Summerfield C. An information theoretical approach to prefrontal executive function. Trends Cogn. Sci. 2007, 11:229-235.
138. Konorski J. Integrative Activity of the Brain: An Interdisciplinary Approach 1967, University of Chicago Press, Chicago.
139. Körding K. Decision theory: what "should" the nervous system do?. Science 2007, 318:606-610.
140. Kravitz A.V., Tye L.D., Kreitzer A.C. Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 2012, 15:816-818.
141. Lammel S., Hetzel A., Häckel O., Jones I., Liss B., Roeper J. Unique properties of mesoprefrontal neurons within a dual mesocorticolimbic dopamine system. Neuron 2008, 57:760-773.
142. Lammel S., Ion D.I., Roeper J., Malenka R.C. Projection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli. Neuron 2011, 70:855-862.
143. Lavin A., Nogueira L., Lapish C.C., Wightman R.M., Phillips P.E.M., Seamans J.K. Mesocortical dopamine neurons operate in distinct temporal domains using multimodal signaling. J. Neurosci. 2005, 25:5013-5023.
144. Lisman J., Grace A.A., Duzel E. A neoHebbian framework for episodic memory; role of dopamine-dependent late LTP. Trends Neurosci. 2011, 34:536-547.
145. Lo C.-C., Wang X.-J. Cortico-basal ganglia circuit mechanism for a decision threshold in reaction time tasks. Nat. Neurosci. 2006, 9:956-963.
146. Loewenstein, G.F., and O'Donoghue, T. (2004). Animal spirits: affective and deliberative processes in economic behavior. Working Papers 04-14, Cornell University, Center for Analytic Economics.
147. Ma W.J., Pouget A. Linking neurons to behavior in multisensory perception: a computational review. Brain Res. 2008, 1242:4-12.
148. Ma W.J., Beck J.M., Latham P.E., Pouget A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 2006, 9:1432-1438.
149. Maia T.V. Two-factor theory, the actor-critic model, and conditioned avoidance. Learn. Behav. 2010, 38:50-67.
150. Maia T.V., Frank M.J. From reinforcement learning models to psychiatric and neurological disorders. Nat. Neurosci. 2011, 14:154-162.
151. Maier S.F., Watkins L.R. Stressor controllability and learned helplessness: the roles of the dorsal raphe nucleus, serotonin, and corticotropin-releasing factor. Neurosci. Biobehav. Rev. 2005, 29:829-841.
152. Marder E., Thirumalai V. Cellular, synaptic and network effects of neuromodulation. Neural Netw. 2002, 15:479-493.
153. Marrocco R.T., Lane R.F., McClurkin J.W., Blaha C.D., Alkire M.F. Release of cortical catecholamines by visual stimulation requires activity in thalamocortical afferents of monkey and cat. J. Neurosci. 1987, 7:2756-2767.
154. Mazzoni P., Hristova A., Krakauer J.W. Why don't we move faster? Parkinson's disease, movement vigor, and implicit motivation. J. Neurosci. 2007, 27:7105-7116.
155. McClure S.M., Daw N.D., Montague P.R. A computational substrate for incentive salience. Trends Neurosci. 2003, 26:423-428.
156. McGaughy J., Sarter M. Behavioral vigilance in rats: task validation and effects of age, amphetamine, and benzodiazepine receptor ligands. Psychopharmacology (Berl.) 1995, 117:340-357.
157. McNally G.P., Johansen J.P., Blair H.T. Placing prediction into the fear circuit. Trends Neurosci. 2011, 34:283-292.
158. McNaughton N., Corr P.J. A two-dimensional neuropsychology of defense: fear/anxiety and defensive distance. Neurosci. Biobehav. Rev. 2004, 28:285-305.
159. Mirenowicz J., Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 1996, 379:449-451.
160. Miyazaki K., Miyazaki K.W., Doya K. Activation of dorsal raphe serotonin neurons underlies waiting for delayed rewards. J. Neurosci. 2011, 31:469-479.
161. Miyazaki K., Miyazaki K.W., Doya K. The role of serotonin in the regulation of patience and impulsivity. Mol. Neurobiol. 2012, 45:213-224.
162. Mobini S., Chiang T.J., Al-Ruwaitea A.S., Ho M.Y., Bradshaw C.M., Szabadi E. Effect of central 5-hydroxytryptamine depletion on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl.) 2000, 149:313-318.
163. Mongillo G., Barak O., Tsodyks M. Synaptic theory of working memory. Science 2008, 319:1543-1546.
164. Montague R. Why Choose This Book?: How We Make Decisions 2006, EP Dutton, New York.
165. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16:1936-1947.
166. Montague P.R., Hyman S.E., Cohen J.D. Computational roles for dopamine in behavioural control. Nature 2004, 431:760-767.
167. Montague P.R., Dolan R.J., Friston K.J., Dayan P. Computational psychiatry. Trends Cogn. Sci. 2012, 16:72-80.
168. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
169. Moutoussis M., Bentall R.P., Williams J., Dayan P. A temporal difference account of avoidance learning. Network 2008, 19:137-160.
170. Mowrer O. On the dual nature of learning: a reinterpretation of conditioning and problem-solving. Harv. Educ. Rev. 1947, 17:102-150.
171. Murschall A., Hauber W. Inactivation of the ventral tegmental area abolished the general excitatory influence of Pavlovian cues on instrumental performance. Learn. Mem. 2006, 13:123-126.
172. Nakahara H., Itoh H., Kawagoe R., Takikawa Y., Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 2004, 41:269-280.
173. Nakamura K., Hikosaka O. Role of dopamine in the primate caudate nucleus in reward modulation of saccades. J. Neurosci. 2006, 26:5360-5369.
174. Nassar M.R., Wilson R.C., Heasly B., Gold J.I. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 2010, 30:12366-12378.
175. Nieuwenhuizen A.G., Rutters F. The hypothalamic-pituitary-adrenal-axis in the regulation of energy balance. Physiol. Behav. 2008, 94:169-177.
176. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 2007, 191:507-520.
177. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 2006, 18:283-328.
178. O'Reilly R.C., Noelle D.C., Braver T.S., Cohen J.D. Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control. Cereb. Cortex 2002, 12:246-257.
179. Ostlund S.B., Wassum K.M., Murphy N.P., Balleine B.W., Maidment N.T. Extracellular dopamine levels in striatal subregions track shifts in motivation and response cost during instrumental conditioning. J. Neurosci. 2011, 31:200-207.
180. Palmiter R.D. Dopamine signaling in the dorsal striatum is essential for motivated behaviors: lessons from dopamine-deficient mice. Ann. N Y Acad. Sci. 2008, 1129:35-46.
181. Panksepp J. Affective Neuroscience 1998, OUP, New York.
182. Parikh V., Kozak R., Martinez V., Sarter M. Prefrontal acetylcholine release controls cue detection on multiple timescales. Neuron 2007, 56:141-154.
183. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 1980, 87:532-552.
184. Peciña S., Smith K.S., Berridge K.C. Hedonic hot spots in the brain. Neuroscientist 2006, 12:500-511.
185. Peppiatt C.M., Howarth C., Mobbs P., Attwell D. Bidirectional control of CNS capillary diameter by pericytes. Nature 2006, 443:700-704.
186. Peshkin, L., Meuleau, N., and Kaelbling, L. (2001). Learning policies with external memory. arXiv, arXiv:cs/0103003. http://arxiv.org/abs/cs/0103003.
187. Phillips J.M., McAlonan K., Robb W.G., Brown V.J. Cholinergic neurotransmission influences covert orientation of visuospatial attention in the rat. Psychopharmacology (Berl.) 2000, 150:112-116.
188. Posner M., Nissen M., Ogden W. Attended and unattended processing modes: the role of set for spatial location. Modes of Perceiving and Processing Information 1978, 137. Lawrence Erlbaum Associates, Hillsdale, NJ. H. Pick, I. Saltzman (Eds.).
189. Puglisi-Allegra S., Imperato A., Angelucci L., Cabib S. Acute stress induces time-dependent responses in dopamine mesolimbic system. Brain Res. 1991, 554:217-222.
190. Puig M.V., Gulledge A.T. Serotonin and prefrontal cortex function: neurons, networks, and circuits. Mol. Neurobiol. 2011, 44:449-464.
191. Raby C.R., Alexis D.M., Dickinson A., Clayton N.S. Planning for the future by western scrub-jays. Nature 2007, 445:919-921.
192. Rajkowski J., Majczynski H., Clayton E., Aston-Jones G. Activation of monkey locus coeruleus neurons varies with difficulty and performance in a target detection task. J. Neurophysiol. 2004, 92:361-371.
193. Ratcliff R., Smith P.L. A comparison of sequential sampling models for two-choice reaction time. Psychol. Rev. 2004, 111:333-367.
194. Redgrave P., Prescott T.J., Gurney K. Is the short-latency dopamine response too short to signal reward error?. Trends Neurosci. 1999, 22:146-151.
195. Robbins T.W., Arnsten A.F.T. The neuropsychopharmacology of fronto-executive function: monoaminergic modulation. Annu. Rev. Neurosci. 2009, 32:267-287.
196. Roberts A.C. The importance of serotonin for orbitofrontal function. Biol. Psychiatry 2011, 69:1185-1191.
197. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 2007, 10:1615-1624.
198. Salamone J.D., Correa M., Farrar A.M., Nunes E.J., Pardo M. Dopamine, behavioral economics, and effort. Front. Behav. Neurosci. 2009, 3:13.
199. Sarter M., Parikh V., Howe W.M. nAChR agonist-induced cognition enhancement: integration of cognitive and neuronal mechanisms. Biochem. Pharmacol. 2009, 78:658-667.
200. Satoh T., Nakai S., Sato T., Kimura M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 2003, 23:9913-9923.
201. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
202. Schweighofer N., Bertin M., Shishida K., Okamoto Y., Tanaka S.C., Yamawaki S., Doya K. Low-serotonin levels increase delayed reward discounting in humans. J. Neurosci. 2008, 28:4528-4532.
203. Schweimer J.V., Ungless M.A. Phasic responses in dorsal raphe serotonin neurons to noxious stimuli. Neuroscience 2010, 171:1209-1215.
204. Servan-Schreiber D., Printz H., Cohen J.D. A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 1990, 249:892-895.
205. Siegel J.M. Hypocretin (orexin): role in normal behavior and neuropathology. Annu. Rev. Psychol. 2004, 55:125-148.
206. Smith Y., Bevan M.D., Shink E., Bolam J.P. Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience 1998, 86:353-387.
207. Smith A., Li M., Becker S., Kapur S. Dopamine, prediction error and associative learning: a model-based account. Network 2006, 17:61-84.
208. Stuber G.D., Hnasko T.S., Britt J.P., Edwards R.H., Bonci A. Dopaminergic terminals in the nucleus accumbens but not the dorsal striatum corelease glutamate. J. Neurosci. 2010, 30:8229-8233.
209. Surmeier D.J., Ding J., Day M., Wang Z., Shen W. D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci. 2007, 30:228-235.
210. Surmeier D.J., Shen W., Day M., Gertler T., Chan S., Tian X., Plotkin J.L. The role of dopamine in modulating the structure and function of striatal circuits. Prog. Brain Res. 2010, 183:149-167.
211. Sutton R. Learning to predict by the methods of temporal differences. Mach. Learn. 1988, 3:9-44.
212. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
213. Talmi D., Seymour B., Dayan P., Dolan R.J. Human pavlovian-instrumental transfer. J. Neurosci. 2008, 28:360-368.
214. Tanaka S.C., Schweighofer N., Asahi S., Shishida K., Okamoto Y., Yamawaki S., Doya K. Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE 2007, 2:e1333.
215. Thiel C.M., Fink G.R. Effects of the cholinergic agonist nicotine on reorienting of visual spatial attention and top-down attentional control. Neuroscience 2008, 152:381-390.
216. Thorndike E. Animal Intelligence 1911, MacMillan, New York.
217. Todd M.T., Niv Y., Cohen J.D. Learning to use working memory in partially observable environments through dopaminergic reinforcement. Advances in Neural Information Processing Systems 21 2009, 1689-1696. MIT Press, Cambridge, MA. D. Koller, D. Schuurmans, Y. Bengio, L. Bottou (Eds.).
218. Tolman E.C. Cognitive maps in rats and men. Psychol. Rev. 1948, 55:189-208.
219. Tops M., Russo S., Boksem M.A.S., Tucker D.M. Serotonin: modulator of a drive to withdraw. Brain Cogn. 2009, 71:427-436.
220. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 2009, 29:2225-2232.
221. Tritsch N.X., Sabatini B.L. Dopaminergic Modulation of Synaptic Transmission in Cortex and Striatum. Neuron 2012, 76:33-50. this issue.
222. Ungless M.A., Magill P.J., Bolam J.P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 2004, 303:2040-2042.
223. Usher M., McClelland J.L. The time course of perceptual choice: the leaky, competing accumulator model. Psychol. Rev. 2001, 108:550-592.
224. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J. Neurosci. 2007, 27:4019-4026.
225. Williams G.V., Goldman-Rakic P.S. Modulation of memory fields by dopamine D1 receptors in prefrontal cortex. Nature 1995, 376:572-575.
226. Williams D.R., Williams H. Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J. Exp. Anal. Behav. 1969, 12:511-520.
227. Witte E.A., Marrocco R.T. Alteration of brain noradrenergic activity in rhesus monkeys affects the alerting component of covert orienting. Psychopharmacology (Berl.) 1997, 132:315-323.
228. Wittmann B.C., Schott B.H., Guderian S., Frey J.U., Heinze H.J., Düzel E. Reward-related FMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent long-term memory formation. Neuron 2005, 45:459-467.
229. Wolf O.T. The influence of stress hormones on emotional memory: relevance for psychopathology. Acta Psychol. (Amst.) 2008, 127:513-531.
230. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 15:786-791.
231. Wunderlich K., Smittenaar P., Dolan R.J. Dopamine enhances model-based over model-free choice behavior. Neuron 2012, 75:418-424.
232. Yerkes R., Dodson J. The relation of strength of stimulus to rapidity of habit-formation. J. Comp. Neurol. Psychol. 1908, 18:459-482.
233. Yu A.J., Dayan P. Inference, attention, and decision in a bayesian neural architecture. Advances in Neural Information Processing Systems 17 2005, 1577-1584. MIT Press, Cambridge, MA. L.K. Saul, Y. Weiss, L. Bottou (Eds.).
234. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46:681-692.
235. Zaborszky L. The modular organization of brain systems. Basal forebrain: the last frontier. Prog. Brain Res. 2002, 136:359-372.
236. Zaghloul K.A., Blanco J.A., Weidemann C.T., McGill K., Jaggi J.L., Baltuch G.H., Kahana M.J. Human substantia nigra neurons encode unexpected financial rewards. Science 2009, 323:1496-1499.
237. Zemel R.S., Dayan P., Pouget A. Probabilistic interpretation of population codes. Neural Comput. 1998, 10:403-430.