Dopamine and performance in a reinforcement learning task: Evidence from Parkinson's disease

Brain

Volumn 135, Issue 6, 2012, Pages 1871-1883

  Shiner, Tamara ^a   Seymour, Ben ^a   Wunderlich, Klaus ^a   Hill, Ciaran ^a   Bhatia, Kailash P ^b   Dayan, Peter ^c   Dolan, Raymond J ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

dopamine; functional MRI; learning; Parkinson's disease

Indexed keywords

CARBIDOPA PLUS LEVODOPA;

ADULT; AGED; ARTICLE; BRAIN NERVE CELL; CLINICAL ARTICLE; DRUG HALF LIFE; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; LEARNING TEST; MALE; NUCLEUS ACCUMBENS; PARKINSON DISEASE; PRIORITY JOURNAL; REINFORCEMENT; VENTROMEDIAL PREFRONTAL CORTEX;

EID: 84861556604     PISSN: 00068950     EISSN: 14602156     Source Type: Journal    
DOI: 10.1093/brain/aws083     Document Type: Article

References (90)

1. Ahlenius S, Engel J, Zoller M. Effects of apomorphine and haloperidol on exploratory behavior and latent learning in mice. Physiol Psychol 1977; 5: 290-4.
2. Ashburner J, Friston KJ. Unified segmentation. Neuroimage 2005; 26: 839-51.
3. Bardgett ME, Depenbrock M, Downs N, Points M, Green L. Dopamine modulates effort-based decision making in rats. Behav Neurosci 2009; 123: 242-51.
4. Bayer HM, Glimcher PW. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005; 47: 129-41.
5. Beeler JA, Daw N, Frazier CR, Zhuang X. Tonic dopamine modulates exploitation of reward learning. Front Behav Neurosci 2010; 4: 170.
6. Beninger RJ. The role of dopamine in locomotor activity and learning. Brain Res 1983; 287: 173-96.
7. Beninger RJ, Phillips AG. The effects of pimozide during pairing on the transfer of classical conditioning to an operant discrimination. Pharmacol Biochem Behav 1981; 14: 101-5.
8. Berridge KC. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology 2007; 191: 391-431.
9. Berridge KC. 'Liking' and 'wanting' food rewards: brain substrates and roles in eating disorders. Physiol Behav 2009; 97: 537-550.
10. Berridge KC, Robinson TE. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev 1998; 28: 309-69.
11. Bodi N, Keri S, Nagy H, Moustafa A, Myers CE, Daw N, et al. Reward-learning and the novelty-seeking personality: a between-and within-subjects study of the effects of dopamine agonists on young Parkinson's patients. Brain 2009; 132: 2385-95.
12. Boorman ED, Behrens TE, Woolrich MW, Rushworth MF. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 2009; 62: 733-43.
13. Boureau YL, Dayan P. Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 2011; 36: 74-97.
14. Cagniard B, Balsam PD, Brunner D, Zhuang X. Mice with chronically elevated dopamine exhibit enhanced motivation, but not learning, for a food reward. Neuropsychopharmacology 2006; 31: 1362-70.
15. Cannon CM, Palmiter RD. Reward without dopamine. J Neurosci 2003; 23: 10827-31.
16. Cools R, D'Esposito M. Inverted-U-shaped dopamine actions on human working memory and cognitive control. Biol Psychiatry 2011; 69: e113-e125.
17. Cools R, Nakamura K, Daw ND. Serotonin and dopamine: unifying affective, activational, and decision functions. Neuropsychopharmacol-ogy 2011; 36: 98-113.
18. Cools R, Sheridan M, Jacobs E, D'Esposito M. Impulsive personality predicts dopamine-dependent changes in frontostriatal activity during component processes of working memory. J Neurosci 2007; 27: 5506-14.
19. Daw ND, Niv Y, Dayan P. Uncertainty-based competition between pre-frontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005; 8: 1704-11.
20. Day JJ, Carelli RM. The nucleus accumbens and Pavlovian reward learning. Neuroscientist 2007; 13: 148-59.
21. Dayan P, Niv Y, Seymour B, Daw ND. The misbehavior of value and the discipline of the will. Neural Netw 2006; 19: 1153-60.
22. de Wit S, Barker RA, Dickinson AD, Cools R. Habitual versus goal-directed action control in Parkinson disease. J Cogn Neurosci 2011; 23: 1218-29.
23. Denenberg VH, Kim DS, Palmiter RD. The role of dopamine in learning, memory, and performance of a water escape task. Behav Brain Res 2004; 148: 73-8.
24. Dickinson A, Smith J, Mirenowicz J. Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav Neurosci 2000; 114: 468-83.
25. Edwards M, Quinn N, Bhatia K. Parkinson's disease and other movement disorders. Oxford: Oxford University Press; 2008.
26. Fitzgerald TH, Seymour B, Bach DR, Dolan RJ. Differentiable neural substrates for learned and described value and risk. Curr Biol 2010; 20: 1823-9.
27. FitzGerald TH, Seymour B, Dolan RJ. The role of human orbitofrontal cortex in value comparison for incommensurable objects. J Neurosci 2009; 29: 8388-95.
28. Frank MJ, Moustafa AA, Haughey HM, Curran T, Hutchison KE. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci USA 2007a; 104: 16311-16.
29. Frank MJ, Samanta J, Moustafa AA, Sherman SJ. Hold your horses: impulsivity, deep brain stimulation, and medication in parkinsonism. Science 2007b; 318: 1309-12.
30. Frank MJ, Seeberger LC, O'Reilly RC. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004; 306: 1940-3.
31. Fuster JM. The prefrontal cortex-an update: time is of the essence. Neuron 2001; 30: 319-33.
32. Gibbs SE, D'Esposito M. Individual capacity differences predict working memory performance and prefrontal activity following dopamine stimulation. Cogn Affect Behav Neurosci 2005; 5: 212-221.
33. Gottfried JA, O'Doherty J, Dolan RJ. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 2003; 301: 1104-7.
34. Graef S, Biele G, Krugel LK, Marzinzik F, Wahl M, Wotka J, et al. Differential influence of levodopa on reward-based learning in Parkinson's disease. Front Hum Neurosci 2010; 4: 169.
35. Haber SN, Knutson B. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 2010; 35: 4-26.
36. Hnasko TS, Sotak BN, Palmiter RD. Morphine reward in dopamine-deficient mice. Nature 2005; 438: 854-857.
37. Ikemoto S, Panksepp J. The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Brain Res Rev 1999; 31: 6-41.
38. Jocham G, Klein TA, Ullsperger M. Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. J Neurosci 2011; 31: 1606-13.
39. Kable JW, Glimcher PW. The neurobiology of decision: consensus and controversy. Neuron 2009; 63: 733-45.
40. Klein TA, Neumann J, Reuter M, Hennig J, von Cramon DY, Ullsperger M. Genetically determined differences in learning from errors. Science 2007; 318: 1642-5.
41. Knowlton BJ, Mangels JA, Squire LR. A neostriatal habit learning system in humans. Science 1996; 273: 1399-402.
42. Koller WC, Melamed E. Parkinson's disease and related disorders: part 1. In: Aminoff MJ, Boller E, Swaab DE, editors. Handbook of clinical neurology. Vol. 83. Philadephia: Elsevier; 2007.
43. Lancaster JL, Rainey LH, Summerlin JL, Freitas CS, Fox PT, Evans AC, et al. Automated labeling of the human brain: a preliminary report on the development and evaluation of a forward-transform method. Hum Brain Mapp 1997; 5: 238-42.
44. Lex B, Hauber W. The role of nucleus accumbens dopamine in outcome encoding in instrumental and Pavlovian conditioning. Neurobiol Learn Mem 2010; 93: 283-90.
45. Luk CH, Wallis JD. Dynamic encoding of responses and outcomes by neurons in medial prefrontal cortex. J Neurosci 2009; 29: 7526-39.
46. Maia TV, Frank MJ. From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 2011; 14: 154-62.
47. Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH. An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. Neuroimage 2003; 19: 1233-9.
48. Matsumoto K, Suzuki W, Tanaka K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 2003; 301: 229-32.
49. Mazziotta JC, Toga AW, Evans A, Fox P, Lancaster J. A probabilistic atlas of the human brain: theory and rationale for its development. The International Consortium for Brain Mapping (ICBM). Neuroimage 1995; 2: 89-101.
50. Mazzoni P, Hristova A, Krakauer JW. Why don't we move faster? Parkinson's disease, movement vigor, and implicit motivation. J Neurosci 2007; 27: 7105-16.
51. McClure SM, Berns GS, Montague PR. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003; 38: 339-46.
52. McClure SM, Daw ND, Montague PR. A computational substrate for incentive salience. Trends Neurosci 2003; 26: 423-428.
53. Myers CE, Shohamy D, Gluck MA, Grossman S, Kluger A, Ferris S, et al. Dissociating hippocampal versus basal ganglia contributions to learning and transfer. J Cogn Neurosci 2003; 15: 185-93.
54. Naidich TP, Duvernoy HM, Delman BN, Sorensen AG, Kollias SS, Haacke EM. Duvernoy's Atlas of the Human Brain Stem and Cerebellum. NewYork: SpringerWien; 2009.
55. Niv Y. Cost, benefit, tonic, phasic: what do response rates tell us about dopamine and motivation? Ann NY Acad Sci 2007; 1104: 357-76.
56. Niv Y, Daw ND, Joel D, Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 2007; 191: 507-20.
57. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. Temporal difference models and reward-related learning in the human brain. Neuron 2003; 38: 329-37.
58. Padoa-Schioppa C, Assad JA. Neurons in the orbitofrontal cortex encode economic value. Nature 2006; 441: 223-26.
59. Palminteri S, Lebreton M, Worbe Y, Grabli D, Hartmann A, Pessiglione M. Pharmacological modulation of subliminal learning in Parkinson's and Tourette's syndromes. Proc Natl Acad Sci USA 2009; 106: 19179-84.
60. Palmiter RD. Dopamine signaling in the dorsal striatum is essential for motivated behaviors: lessons from dopamine-deficient mice. Ann NY Acad Sci 2008; 1129: 35-46.
61. Parkinson JA, Dalley JW, Cardinal RN, Bamford A, Fehnert B, Lachenal G, et al. Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function. Behav Brain Res 2002; 137: 149-63.
62. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006; 442: 1042-1045.
63. Plassmann H, O'Doherty JP, Rangel A. Appetitive and aversive goal values are encoded in the medial orbitofrontal cortex at the time of decision making. J Neurosci 2010; 30: 10799-808.
64. Robbins TW. Chemical neuromodulation of frontal-executive functions in humans and other animals. Exp Brain Res 2000; 133: 130-8.
65. Robinson S, Sandstrom SM, Denenberg VH, Palmiter RD. Distinguishing whether dopamine regulates liking, wanting, and/or learning about rewards. Behav Neurosci 2005; 119: 5-15.
66. Rowe JB, Hughes L, Ghosh BC, Eckstein D, Williams-Gray CH, Fallon S, et al. Parkinson's disease and dopaminergic therapy-differential effects on movement, reward and cognition. Brain 2008; 131: 2094-105.
67. Salamone JD, Correa M. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav Brain Res 2002; 137: 3-25.
68. Salamone JD, Correa M, Mingote S, Weber SM. Nucleus accumbens dopamine and the regulation of effort in food-seeking behavior: implications for studies of natural motivation, psychiatry, and drug abuse. J Pharmacol Exp Ther 2003; 305: 1-8.
69. Sawaguchi T. The effects of dopamine and its antagonists on directional delay-period activity of prefrontal neurons in monkeys during an oculomotor delayed-response task. Neurosci Res 2001; 41: 115-28.
70. Sawaguchi T, Goldman-Rakic PS. D1 dopamine receptors in prefrontal cortex: involvement in working memory. Science 1991; 251: 947-50.
71. Schonberg T, O'Doherty JP, Joel D, Inzelberg R, Segev Y, Daw ND. Selective impairment of prediction error signaling in human dorsolat-eral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. Neuroimage 2010; 49: 772-81.
72. Schultz W. Predictive reward signal of dopamine neurons. J Neurophysiol 1998; 80: 1-27.
73. Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science 1997; 275: 1593-1599.
74. Schwarz G. Estimating the dimension of a model. Ann Statis 1978; 6: 461-4.
75. Seymour B, McClure SM. Anchors, scales and the relative coding of value in the brain. Curr Opin Neurobiol 2008; 18: 173-8.
76. Shohamy D, Myers CE, Geghman KD, Sage J, Gluck MA. L-dopa impairs learning, but spares generalization, in Parkinson's disease. Neuropsy-chologia 2006; 44: 774-84.
77. Steeves TD, Miyasaki J, Zurowski M, Lang AE, Pellecchia G, Van Eimeren T, et al. Increased striatal dopamine release in Parkinsonian patients with pathological gambling: a [11C] raclopride PET study. Brain 2009; 132: 1376-85.
78. Stephan KE, Penny WD, Daunizeau J, Moran RJ, Friston KJ. Bayesian model selection for group studies. Neuroimage 2009; 46: 1004-17.
79. Stuss DT, Knight RT, editors. Principles of frontal lobe function. New York: Oxford University Press; 2002.
80. Sutton R, Barto AG. Reinforcement learning, an introduction. Cabridge, MA: MIT press; 1998.
81. Ungerstedt U. Striatal dopamine release after amphetamine or nerve degeneration revealed by rotational behaviour. Acta Physiol Scand Suppl 1971; 367: 49-68.
82. Voon V, Pessiglione M, Brezing C, Gallea C, Fernandez HH, Dolan RJ, et al. Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron 2010; 65: 135-42.
83. Wang M, Vijayraghavan S, Goldman-Rakic PS. Selective D2 receptor actions on the functional circuitry of working memory. Science 2004; 303: 853-6.
84. Watanabe M, Kodama T, Hikosaka K. Increase of extracellular dopamine in primate prefrontal cortex during a working memory task. J Neurophysiol 1997; 78: 2795-8.
85. Wise RA. Dopamine, learning and motivation. Nat Rev Neurosci 2004; 5: 483-494.
86. Wise RA, Rompre PP. Brain dopamine and reward. Annu Rev Psychol 1989; 40: 191-225.
87. Wunderlich K, Rangel A, O'Doherty JP. Neural computations underlying action-based decision making in the human brain. Proc Natl Acad Sci USA 2009; 106: 17199-204.
88. Wunderlich K, Rangel A, O'Doherty JP. Economic choices can be made using only stimulus values. Proc Natl Acad Sci USA 2010; 107: 15005-10.
89. Wyvell CL, Berridge KC. Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward: enhancement of reward "wanting" without enhanced "liking" or response reinforcement. J Neurosci 2000; 20: 8122-30.
90. Yin HH, Ostlund SB, Balleine BW. Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur J Neurosci 2008; 28: 1437-48.