Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making

Proceedings of the National Academy of Sciences of the United States of America

Volumn 112, Issue 5, 2015, Pages 1595-1600

  Deserno, Lorenz ^a,b,c   Huys, Quentin J M ^d,e   Boehme, Rebecca ^c   Buchert, Ralph ^c   Heinze, Hans Jochen ^a,b   Grace, Anthony A ^f   Dolan, Raymond J ^g,h   Heinz, Andreas ^c   Schlagenhauf, Florian ^a,c  

^a MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

^b OTTO VON GUERICKE UNIVERSITY   (Germany)

^c CHARITÉ UNIVERSITÄTSMEDIZIN BERLIN   (Germany)

^d UNIVERSITY OF ZURICH   (Switzerland)

^e UNIVERSITY HOSPITAL OF PSYCHIATRY   (Switzerland)

^f UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^g UNIVERSITY COLLEGE LONDON   (United Kingdom)

^h HUMBOLDT UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

6 FLUORODOPA F 18; DOPAMINE;

ADDICTION; ADULT; ARTICLE; BEHAVIOR; BRAIN REGION; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC TRANSMISSION; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; MODEL; NORMAL HUMAN; POSITRON EMISSION TOMOGRAPHY; PREFRONTAL CORTEX; PRESYNAPTIC MEMBRANE; PRIORITY JOURNAL; SCHIZOPHRENIA; METABOLISM; MIDDLE AGED; NUCLEAR MAGNETIC RESONANCE IMAGING; YOUNG ADULT;

ADULT; BEHAVIOR; CORPUS STRIATUM; DECISION MAKING; DOPAMINE; FEMALE; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; MIDDLE AGED; POSITRON-EMISSION TOMOGRAPHY; YOUNG ADULT;

EID: 84922309111     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1417219112     Document Type: Article

References (55)

1. Dolan RJ, Dayan P (2013) Goals and habits in the brain. Neuron 80(2):312-325.
2. Dickinson AD (1985) Action and habits: The development of behavioural autonomy. Philos Trans R Soc Lond B Biol Sci 308(1135):67-78.
3. Doll BB, Simon DA, Daw ND (2012) The ubiquity of model-based reinforcement learning. Curr Opin Neurobiol 22(6):1075-1081.
4. Balleine BW, Dickinson A (1998) Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology 37(4-5):407-419.
5. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12):1704-1711.
6. Gläscher J, Daw N, Dayan P, O'Doherty JP (2010) States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4):585-595.
7. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69(6):1204-1215.
8. Cools R (2011) Dopaminergic control of the striatum for high-level cognition. Curr Opin Neurobiol 21(3):402-407.
9. Hiroyuki N (2014) Multiplexing signals in reinforcement learning with internal models and dopamine. Curr Opin Neurobiol 25:123-129.
10. Schultz W (2013) Updating dopamine reward signals. Curr Opin Neurobiol 23(2): 229-238.
11. Seamans JK, Yang CR (2004) The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog Neurobiol 74(1):1-58.
12. Wunderlich K, Smittenaar P, Dolan RJ (2012) Dopamine enhances model-based over model-free choice behavior. Neuron 75(3):418-424.
13. Dayan P (2012) Twenty-five lessons from computational neuromodulation. Neuron 76(1):240-256.
14. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16(5):1936-1947.
15. SchultzW, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275(5306):1593-1599.
16. D'Ardenne K, McClure SM, Nystrom LE, Cohen JD (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 319(5867): 1264-1267.
17. Cools R, Gibbs SE, Miyakawa A, Jagust W, D'Esposito M (2008) Working memory capacity predicts dopamine synthesis capacity in the human striatum. J Neurosci 28(5): 1208-1212.
18. Vernaleken I, et al. (2007) "Prefrontal" cognitive performance of healthy subjects positively correlates with cerebral FDOPA influx: An exploratory [18F]-fluoro-L-DOPAPET investigation. Hum Brain Mapp 28(10):931-939.
19. Otto AR, Gershman SJ, Markman AB, Daw ND (2013) The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol Sci 24(5):751-761.
20. de Wit S, Barker RA, Dickinson AD, Cools R (2011) Habitual versus goal-directed action control in Parkinson disease. J Cogn Neurosci 23(5):1218-1229.
21. de Wit S, et al. (2012) Reliance on habits at the expense of goal-directed control following dopamine precursor depletion. Psychopharmacology (Berl) 219(2):621-631.
22. Kumakura Y, Cumming P (2009) PET studies of cerebral levodopa metabolism: A review of clinical findings and modeling approaches. Neuroscientist 15(6):635-650.
23. Schlagenhauf F, et al. (2013) Ventral striatal prediction error signaling is associated with dopamine synthesis capacity and fluid intelligence. Hum Brain Mapp 34(6): 1490-1499.
24. McDannald MA, Lucantonio F, Burke KA, Niv Y, Schoenbaum G (2011) Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 31(7):2700-2705.
25. Stephan KE, Penny WD, Daunizeau J, Moran RJ, Friston KJ (2009) Bayesian model selection for group studies. Neuroimage 46(4):1004-1017.
26. Braver TS, Cohen JD (1999) Dopamine, cognitive control, and schizophrenia: The gating model. Prog Brain Res 121:327-349.
27. Moran RJ, Symmonds M, Stephan KE, Friston KJ, Dolan RJ (2011) An in vivo assay of synaptic function mediating human cognition. Curr Biol 21(15):1320-1325.
28. Smittenaar P, FitzGerald TH, Romei V, Wright ND, Dolan RJ (2013) Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron 80(4):914-919.
29. Niv Y, Daw ND, Joel D, Dayan P (2007) Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berl) 191(3):507-520.
30. Beierholm U, et al. (2013) Dopamine modulates reward-related vigor. Neuropsychopharmacology 38(8):1495-1503.
31. Frank MJ, Seeberger LC, O'reilly RC (2004) By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science 306(5703):1940-1943.
32. Lee SW, Shimojo S, O'Doherty JP (2014) Neural computations underlying arbitration between model-based and model-free learning. Neuron 81(3):687-699.
33. Badre D, Hoffman J, Cooney JW, D'Esposito M (2009) Hierarchical cognitive control deficits following damage to the human frontal lobe. Nat Neurosci 12(4):515-522.
34. Koechlin E, Ody C, Kouneiher F (2003) The architecture of cognitive control in the human prefrontal cortex. Science 302(5648):1181-1185.
35. Steinberg EE, et al. (2013) A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16(7):966-973.
36. Goto Y, Grace AA (2005) Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat Neurosci 8(6):805-812.
37. Egerton A, Demjaha A, McGuire P, Mehta MA, Howes OD (2010) The test-retest reliability of 18 F-DOPA PET in assessing striatal and extrastriatal presynaptic dopaminergic function. Neuroimage 50(2):524-531.
38. Goto Y, Grace AA (2008) Dopamine modulation of hippocampal-prefrontal cortical interaction drives memory-guided behavior. Cereb Cortex 18(6):1407-1414.
39. Howes OD, et al. (2012) The nature of dopamine dysfunction in schizophrenia and what this means for treatment. Arch Gen Psychiatry 69(8):776-786.
40. Rakshi JS, et al. (1999) Frontal, midbrain and striatal dopaminergic function in early and advanced Parkinson's disease A 3D[18F]dopa-PET study. Brain 122(Pt 9):1637-1650.
41. Yin HH, Knowlton BJ, Balleine BW (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19(1):181-189.
42. Yin HH, Ostlund SB, Knowlton BJ, Balleine BW (2005) The role of the dorsomedial striatum in instrumental conditioning. Eur J Neurosci 22(2):513-523.
43. Knutson B, Gibbs SE (2007) Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology (Berl) 191(3):813-822.
44. Balleine BW, O 'Doherty JP (2010) Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35(1):48-69.
45. Wunderlich K, Dayan P, Dolan RJ (2012) Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 15(5):786-791.
46. Heinz AJ, Beck A, Meyer-Lindenberg A, Sterzer P, Heinz A (2011) Cognitive and neurobiological mechanisms of alcohol-related aggression. Nat Rev Neurosci 12(7): 400-413.
47. den Ouden HE, et al. (2013) Dissociable effects of dopamine and serotonin on reversal learning. Neuron 80(4):1090-1100.
48. Moran RJ, et al. (2013) Free energy, precision and learning: The role of cholinergic neuromodulation. J Neurosci 33(19):8227-8236.
49. Cools R, Nakamura K, Daw ND (2011) Serotonin and dopamine: Unifying affective, activational, and decision functions. Neuropsychopharmacology 36(1):98-113.
50. Goto Y, Grace AA (2008) Limbic and cortical information processing in the nucleus accumbens. Trends Neurosci 31(11):552-558.
51. Hietala J, et al. (1999) Depressive symptoms and presynaptic dopamine function in neuroleptic-naive schizophrenia. Schizophr Res 35(1):41-50.
52. Vernaleken I, et al. (2007) Asymmetry in dopamine D(2/3) receptors of caudate nucleus is lost with age. Neuroimage 34(3):870-878.
53. Martin-Soelch C, et al. (2011) Lateralization and gender differences in the dopaminergic response to unpredictable reward in the human ventral striatum. Eur J Neurosci 33(9):1706-1715.
54. Tomer R, Goldstein RZ, Wang GJ, Wong C, Volkow ND (2008) Incentive motivation is associated with striatal dopamine asymmetry. Biol Psychol 77(1):98-101.
55. Patlak CS, Blasberg RG (1985) Graphical evaluation of blood-to-brain transfer constants from multiple-time uptake data. Generalizations. J Cereb Blood Flow Metab 5(4):584-590.