Go and no-go learning in reward and punishment: Interactions between affect and effect

NeuroImage

Volumn 62, Issue 1, 2012, Pages 154-166

  Guitart Masip, Marc ^a,b   Huys, Quentin J M ^b,c,d   Fuentemilla, Lluis ^e   Dayan, Peter ^c   Duzel, Emrah ^a,f   Dolan, Raymond J ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

^d ROYAL FREE AND UNIVERSITY COLLEGE MEDICAL SCHOOL   (United Kingdom)

^e UNIVERSITY OF BARCELONA   (Spain)

^f OTTO VON GUERICKE UNIVERSITY   (Germany)

Author keywords

Action; Instrumental; Learning; Pavlovian; Striatum

Indexed keywords

ADULT; ARTICLE; BOLD SIGNAL; BRAIN FUNCTION; CONDITIONING; CONTROLLED STUDY; CORPUS STRIATUM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GO LEARNING; HUMAN; INFERIOR FRONTAL GYRUS; LEARNING; MALE; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; MENTAL PERFORMANCE; MENTAL TASK; NERVE CELL; NO GO LEARNING; PRIORITY JOURNAL; PUNISHMENT; REINFORCEMENT; REWARD; SUBSTANTIA NIGRA; VENTRAL TEGMENTUM; VOLUNTEER;

ADOLESCENT; ADULT; AFFECT; BRAIN MAPPING; CORPUS STRIATUM; DECISION MAKING; FEMALE; HUMANS; INHIBITION (PSYCHOLOGY); MALE; NEURONAL PLASTICITY; PUNISHMENT; REWARD; YOUNG ADULT;

EID: 84861621282     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2012.04.024     Document Type: Article

References (75)

1. Aron A.R., Poldrack R.A. Cortical and subcortical contributions to stop signal response inhibition: role of the subthalamic nucleus. J. Neurosci. 2006, 26:2424-2433.
2. Atallah H.E., Lopez-Paniagua D., Rudy J.W., O'Reilly R.C. Separate neural substrates for skill learning and performance in the ventral and dorsal striatum. Nat. Neurosci. 2007, 10:126-131.
3. Balleine B.W., Liljeholm M., Ostlund S.B. The integrative function of the basal ganglia in instrumental conditioning. Behav. Brain Res. 2009, 199:43-52.
4. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
5. Bayer H.M., Lau B., Glimcher P.W. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 2007, 98:1428-1439.
6. Behrens T.E., Hunt L.T., Woolrich M.W., Rushworth M.F. Associative learning of social value. Nature 2008, 456:245-249.
7. Beninger R.J., Phillips A.G. The effects of pimozide during pairing on the transfer of classical conditioning to an operant discrimination. Pharmacol. Biochem. Behav. 1981, 14:101-105.
8. Berridge K.C., Robinson T.E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?. Brain Res. Brain Res. Rev. 1998, 28:309-369.
9. Blanchard D.C., Blanchard R.J. Ethoexperimental approaches to the biology of emotion. Annu. Rev. Psychol. 1988, 39:43-68.
10. Boureau Y.L., Dayan P. Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 2011, 36:74-97.
11. Breland K., Breland M. The misbehavior of organisms. Am. Psychol. 1961, 16:681-684.
12. Brett M., Anton J.-L., Valabregue R., Poline J.-B. Region of interest analysis using an SPM toolbox. 8th International Conference on Functional Mapping of the Human Brain 2002, Sendai, Japan.
13. Bunzeck N., Duzel E. Absolute coding of stimulus novelty in the human substantia nigra/VTA. Neuron 2006, 51:369-379.
14. Cardinal R.N., Parkinson J.A., Hall J., Everitt B.J. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 2002, 26:321-352.
15. Cools R., Frank M.J., Gibbs S.E., Miyakawa A., Jagust W., D'Esposito M. Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration. J. Neurosci. 2009, 29:1538-1543.
16. Cools R., Nakamura K., Daw N.D. Serotonin and dopamine: unifying affective, activational, and decision functions. Neuropsychopharmacology 2011, 36:98-113.
17. Crockett M.J., Clark L., Tabibnia G., Lieberman M.D., Robbins T.W. Serotonin modulates behavioral reactions to unfairness. Science 2008, 320:1739.
18. Crockett M.J., Clark L., Robbins T.W. Reconciling the role of serotonin in behavioral inhibition and aversion: acute tryptophan depletion abolishes punishment-induced inhibition in humans. J. Neurosci. 2009, 29:11993-11999.
19. Darvas M., Fadok J.P., Palmiter R.D. Requirement of dopamine signaling in the amygdala and striatum for learning and maintenance of a conditioned avoidance response. Learn. Mem. 2011, 18:136-143.
20. Daw N.D., Doya K. The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 2006, 16:199-204.
21. Daw N.D., Kakade S., Dayan P. Opponent interactions between serotonin and dopamine. Neural Netw. 2002, 15:603-616.
22. Dayan P., Daw N.D. Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 2008, 8:429-453.
23. Dayan P., Huys Q.J. Serotonin in affective control. Annu. Rev. Neurosci. 2009, 32:95-126.
24. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Netw. 2006, 19:1153-1160.
25. Deakin J.F.W., Graeff F.G. 5-HT and mechanisms of defense. J. Psychopharmacol. 1991, 5:305-316.
26. Denrell J. Adaptive learning and risk taking. Psychol. Rev. 2007, 114:177-187.
27. Dickinson A., Balleine B. The Role of Learning in Motivation 2002, Wiley, New York. 3rd ed.
28. Duzel E., Bunzeck N., Guitart-Masip M., Wittmann B., Schott B.H., Tobler P.N. Functional imaging of the human dopaminergic midbrain. Trends Neurosci. 2009, 32:321-328.
29. Everitt B.J., Belin D., Economidou D., Pelloux Y., Dalley J.W., Robbins T.W. Review. Neural mechanisms underlying the vulnerability to develop compulsive drug-seeking habits and addiction. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2008, 363:3125-3135.
30. Flagel S.B., Robinson T.E., Clark J.J., Clinton S.M., Watson S.J., Seeman P., Phillips P.E., Akil H. An animal model of genetic vulnerability to behavioral disinhibition and responsiveness to reward-related cues: implications for addiction. Neuropsychopharmacology 2010, 35:388-400.
31. Flagel S.B., Clark J.J., Robinson T.E., Mayo L., Czuj A., Willuhn I., Akers C.A., Clinton S.M., Phillips P.E., Akil H. A selective role for dopamine in stimulus-reward learning. Nature 2011, 469:53-57.
32. Fleming S.M., Thomas C.L., Dolan R.J. Overcoming status quo bias in the human brain. Proc. Natl. Acad. Sci. U. S. A. 2010, 107:6005-6009.
33. Frank M.J., Fossella J.A. Neurogenetics and pharmacology of learning, motivation, and cognition. Neuropsychopharmacology 2011, 36:133-152.
34. Frank M.J., Seeberger L.C., O'Reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
35. Friston K.J., Fletcher P., Josephs O., Holmes A., Rugg M.D., Turner R. Event-related fMRI: characterizing differential responses. NeuroImage 1998, 7:30-40.
36. Gerfen C.R. The neostriatal mosaic: multiple levels of compartmental organization. Trends Neurosci. 1992, 15:133-139.
37. Glascher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
38. Gray J.A., McNaughton M. The Neuropsychology of Anxiety: An Inquiry into the Function of the Septohippocampal System 2000, Oxford University Press, Oxford. 2nd ed.
39. Guitart-Masip M., Talmi D., Dolan R. Conditioned associations and economic decision biases. NeuroImage 2010, 53:206-214.
40. Guitart-Masip M., Fuentemilla L., Bach D.R., Huys Q.J., Dayan P., Dolan R.J., Duzel E. Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain. J. Neurosci. 2011, 31:7867-7875.
41. Hikida T., Kimura K., Wada N., Funabiki K., Nakanishi S. Distinct roles of synaptic transmission in direct and indirect striatal pathways to reward and aversive behavior. Neuron 2010, 66:896-907.
42. Hutton C., Bork A., Josephs O., Deichmann R., Ashburner J., Turner R. Image distortion correction in fMRI: a quantitative evaluation. NeuroImage 2002, 16:217-240.
43. Huys Q.J., Cools R., Gölzer M., Friedel E., Heinz A., Dolan R.J., Dayan P. Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding. PLoS Comput. Biol. 2001, 7:e1002028.
44. Kass R., Raftery A. Bayes factors. J. Am. Stat. Assoc. 1995, 90.
45. Klein A., Andersson J., Ardekani B.A., Ashburner J., Avants B., Chiang M.C., Christensen G.E., Collins D.L., Gee J., Hellier P., Song J.H., Jenkinson M., Lepage C., Rueckert D., Thompson P., Vercauteren T., Woods R.P., Mann J.J., Parsey R.V. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. NeuroImage 2009, 46:786-802.
46. Klein-Flugge M.C., Hunt L.T., Bach D.R., Dolan R.J., Behrens T.E. Dissociable reward and timing signals in human midbrain and ventral striatum. Neuron 2011, 72:654-664.
47. Li J., Daw N.D. Signals in human striatum are appropriate for policy update rather than value prediction. J. Neurosci. 2011, 31:5504-5511.
48. MacKay D.J. Information Theory, Inference and Learning Algorithms 2003, Cambridge University Press, Cambridge, UK.
49. Maia T.V. Two-factor theory, the actor-critic model, and conditioned avoidance. Learn. Behav. 2010, 38:50-67.
50. March J.G. Learning to be risk averse. Psychol. Rev. 1996, 103:309-319.
51. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
52. McCullough L.D., Sokolowski J.D., Salamone J.D. A neurochemical and behavioral investigation of the involvement of nucleus accumbens dopamine in instrumental avoidance. Neuroscience 1993, 52:919-925.
53. Moutoussis M., Bentall R.P., Williams J., Dayan P. A temporal difference account of avoidance learning. Network 2008, 19:137-160.
54. Mowrer O.H. On the dual nature of learning: a reinterpretation of conditioning and problem solving. Harv. Educ. Rev. 1947, 17:102-148.
55. Ng A.Y., Harada D., Russell S. Policy invariance under reward transformations: theory and application to reward shaping. 16th International Conference on Machine Learning 1999, 278-287.
56. Niv Y., Joel D., Meilijson I., Ruppin E. Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviors. Adapt. Behav. 2002, 10:5-24.
57. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl) 2007, 191:507-520.
58. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
59. O'Doherty J., Dayan P., Schultz J., Deichmann R., Friston K., Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452-454.
60. Parkinson J.A., Olmstead M.C., Burns L.H., Robbins T.W., Everitt B.J. Dissociation in effects of lesions of the nucleus accumbens core and shell on appetitive Pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by d-amphetamine. J. Neurosci. 1999, 19:2401-2411.
61. Parkinson J.A., Dalley J.W., Cardinal R.N., Bamford A., Fehnert B., Lachenal G., Rudarakanchana N., Halkerston K.M., Robbins T.W., Everitt B.J. Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function. Behav. Brain Res. 2002, 137:149-163.
62. Pessiglione M., Seymour B., Flandin G., Dolan R.J., Frith C.D. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006, 442:1042-1045.
63. Robbins T.W. Shifting and stopping: fronto-striatal substrates, neurochemical modulation and clinical implications. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362:917-932.
64. Salamone J.D., Correa M., Farrar A., Mingote S.M. Effort-related functions of nucleus accumbens dopamine and associated forebrain circuits. Psychopharmacology (Berl) 2007, 191:461-482.
65. Schonberg T., Daw N.D., Joel D., O'Doherty J.P. Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J. Neurosci. 2007, 27:12860-12867.
66. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
67. Soubrie P. Reconciling the role of central serotonin neurons in human and animal behavior. Behav. Brain Sci. 1986, 9:319-364.
68. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, The MIT Press, Cambridge, Massachusetts.
69. Talmi D., Seymour B., Dayan P., Dolan R.J. Human Pavlovian-instrumental transfer. J. Neurosci. 2008, 28:360-368.
70. Talmi D., Dayan P., Kiebel S.J., Frith C.D., Dolan R.J. How humans integrate the prospects of pain and reward during choice. J. Neurosci. 2009, 29:14617-14626.
71. Uslaner J.M., Dell'Orco J.M., Pevzner A., Robinson T.E. The influence of subthalamic nucleus lesions on sign-tracking to stimuli paired with food and drug rewards: facilitation of incentive salience attribution?. Neuropsychopharmacology 2008, 33:2352-2361.
72. Weiskopf N., Helms G. Multi-parameter Mapping of the Human Brain at 1mm Resolution in Less than 20minutes. ISMRM 16, Toronto, Canada 2008.
73. Weiskopf N., Hutton C., Josephs O., Deichmann R. Optimal EPI parameters for reduction of susceptibility-induced BOLD sensitivity losses: a whole-brain analysis at 3T and 1.5T. NeuroImage 2006, 33:493-504.
74. Wickens J.R., Budd C.S., Hyland B.I., Arbuthnott G.W. Striatal contributions to reward and decision making: making sense of regional variations in a reiterated processing matrix. Ann. N. Y. Acad. Sci. 2007, 1104:192-212.
75. Williams D.R., Williams H. Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J. Exp. Anal. Behav. 1969, 12:511-520.