Ventral striatum: A critical look at models of learning and evaluation

Current Opinion in Neurobiology

Volumn 21, Issue 3, 2011, Pages 387-392

  Van der Meer, Matthijs A A ^a   Redish, A David ^b  

^a UNIVERSITY OF WATERLOO   (Canada)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

BEHAVIOR; BRAIN FUNCTION; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC TRANSMISSION; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GAMMA RHYTHM; HIPPOCAMPUS; HIPPOCAMPUS THETA RHYTHM; HUMAN; LEARNING; NONHUMAN; POTENTIOMETRY; PRIORITY JOURNAL; REVIEW; REWARD; TEMPORAL DIFFERENCE REINFORCEMENT LEARNING; VENTRAL TEGMENTUM;

ANIMALS; COGNITION; CORPUS STRIATUM; HUMANS; LEARNING; MODELS, BIOLOGICAL; REINFORCEMENT (PSYCHOLOGY);

EID: 79960270857     PISSN: 09594388     EISSN: 18736882     Source Type: Journal    
DOI: 10.1016/j.conb.2011.02.011     Document Type: Review

References (75)

1. Mogenson G., Jones D., Yim C. From motivation to action: functional interface between the limbic system and the motor system. Prog Neurobiol 1980, 14:69-97.
2. Salamone J.D., Correa M., Farrar A.M., Nunes E.J., Pardo M. Dopamine, behavioral economics, and effort. Front Behav Neurosci 2009, 3:13.
3. Berridge K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl) 2007, 391-431.
4. Cardinal R.N., Parkinson J.A., Hall J., Everitt B.J. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 2002, 26:321-352.
5. Nicola S.M. The nucleus accumbens as part of a basal ganglia action selection circuit. Psychopharmacology 2007, 191:521-550.
6. Joel D., Niv Y., Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks: Official J Int Neural Network Soc 2002, 15:535-547.
7. O'Doherty J.P., Hampton A., Kim H. Model-based fMRI and its application to reward learning and decision making. Ann NY Acad Sci 2007, 1104:35-53.
8. Corrado G., Doya K. Understanding neural coding through the model-based analysis of decision making. J Neurosci 2007, 27:8178-8180.
9. Daw ND: reinforcement learning models of the dopamine system and their behavioral implications. Ph.D. thesis, Carnegie Mellon University, 2003.
10. Dayan P., Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol 2008, 18:185-196.
11. Maia T.V. Reinforcement learning, conditioning, and the brain: successes and challenges. Cognitive Affect Behav Neurosci 2009, 9:343-364.
12. Redish A.D., Jensen S., Johnson A., Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 2007, 114:784-805.
13. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput 2006, 18:283-328.
14. Zilli E.A., Hasselmo M.E. Modeling the role of working memory and episodic memory in behavioral tasks. Hippocampus 2008.
15. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
16. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia 1995, 249-270. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
17. Tobler P.N., Kobayashi S. Electrophysiological correlates of reward processing in dopamine neurons. Handbook of Reward and Decision Making 2009, 29-50. 29-50. J.C. Dreher, L. Tremblay (Eds.).
18. Bromberg-Martin E.S., Matsumoto M., Hikosaka O. Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons. Neuron 2010, 67:144-155.
19. Cheer J.F., Aragona B.J., Heien M.L.A.V., Seipel A.T., Carelli R.M., Wightman R.M. Coordinated accumbal dopamine release and neural activity drive goal-directed behavior. Neuron 2007, 54:237-244.
20. Day J.J., Roitman M.F., Wightman R.M., Carelli R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci 2007, 10:1020-1028.
21. Pagnoni G., Zink C.F., Montague P.R., Berns G.S. Activity in human ventral striatum locked to errors of reward prediction. Nat Neurosci 2002, 5:97-98.
22. O'Doherty J., Dayan P., Schultz J., Deichmann R., Friston K., Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452-454.
23. Hare T.A., O'Doherty J., Camerer C.F., Schultz W., Rangel A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 2008, 28:5623-5630.
24. D'Ardenne K., McClure S.M., Nystrom L.E., Cohen J.D. BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 2008, 319:1264-1267.
25. Schonberg T., O'Doherty J.P., Joel D., Inzelberg R., Segev Y., Daw N.D. Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. NeuroImage 2010, 49:772-781.
26. Roesch M.R., Singh T., Brown P.L., Mullins S.E., Schoenbaum G. Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J Neurosci 2009, 29:13365-13376.
27. Ito M., Doya K. Validation of decision-making models and analysis of decision variables in the rat basal ganglia. Journal of Neuroscience 2009, 29:9861-9874.
28. Khamassi M., Mulder A.B., Tabuchi E., Douchamps V., Wiener S.I. Anticipatory reward signals in ventral striatal neurons of behaving rats. Eur J Neurosci 2008, 28:1849-1866.
29. van der Meer M.A.A., Redish A.D. Theta phase precession in rat ventral striatum links place and reward information. J Neurosci 2011, 31:2843-2854.
30. Nakahara H., Itoh H., Kawagoe R., Takikawa Y., Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 2004, 41:269-280.
31. Knutson B., Taylor J., Kaufman M., Peterson R., Glover G. Distributed neural representation of expected value. J Neurosci 2005, 25:4806-4812.
32. Preuschoff K., Bossaerts P., Quartz S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 2006, 51:381-390.
33. McClure S.M., Ericson K.M., Laibson D.I., Loewenstein G., Cohen J.D. Time discounting for primary rewards. J Neurosci Official J Soc Neurosci 2007, 27:5796-5804.
34. Kim Y.B., Huh N., Lee H., Baeg E.H., Lee D., Jung M.W. Encoding of action history in the rat ventral striatum. J Neurophysiol 2007, 98:3548-3556.
35. Kim H., Sul J.H., Huh N., Lee D., Jung M.W. Role of striatum in updating values of chosen actions. J Neurosci 2009, 29:14701-14712.
36. Kimchi E.Y., Laubach M. Dynamic encoding of action selection by the medial striatum. J Neurosci 2009, 29:3148-3159.
37. Carelli R.M., Wondolowski J. Selective encoding of cocaine versus natural rewards by nucleus accumbens neurons is not related to chronic drug exposure. J Neurosci 2003, 23:11214-11223.
38. Setlow B., Schoenbaum G., Gallagher M. Neural encoding in ventral striatum during olfactory discrimination learning. Neuron 2003, 38:625-636.
39. Taha S.A., Fields H.L. Encoding of palatability and appetitive behaviors by distinct neuronal populations in the nucleus accumbens. J Neurosci 2005, 25:1193-1202.
40. Yun I.A., Wakabayashi K.T., Fields H.L., Nicola S.M. The ventral tegmental area is required for the behavioral and nucleus accumbens neuronal firing responses to incentive cues. J Neurosci 2004, 24:2923-2933.
41. Roesch M.R., Calu D.J., Esber G.R., Schoenbaum G. All that glitters dissociating attention and outcome-expectancy from prediction errors signals. J Neurophysiol 2010, 104:587-595.
42. Peters J., Büchel C. Neural representations of subjective reward value. Behav Brain Res 2010, 213:135-141.
43. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
44. van der Meer M.A.A., Johnson A., Schmitzer-Torbert N.C., Redish A.D. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 2010, 67:25-32.
45. Humphries M.D., Prescott T.J. The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog Neurobiol 2009, 90:385-417.
46. Tremblay L., Worbe Y., Hollerman J.R. The ventral striatum: a heterogenous structure involved in reward processing, motivation and decision-making. Handbook of Reward and Decision Making 2009, 51-78. J.C. Dreher, L. Tremblay (Eds.).
47. Jones J.L., Day J.J., Wheeler R.A., Carelli R.M. The basolateral amygdala differentially regulates conditioned neural responses within the nucleus accumbens core and shell. Neuroscience 2010, 169:1186-1198.
48. Krause M., German P.W., Taha S.A., Fields H.L. A pause in nucleus accumbens neuron firing is required to initiate and maintain feeding. J Neurosci 2010, 30:4746-4756.
49. Aragona B.J., Day J.J., Roitman M.F., Cleaveland N.A., Wightman R.M., Carelli R.M. Regional specificity in the real-time development of phasic dopamine transmission patterns during acquisition of a cue-cocaine association in rats. Eur J Neurosci 2009, 30:1889-1899.
50. Wightman R.M., Heien M.L.A.V., Wassum K.M., Sombers L.A., Aragona B.J., Khan A.S., Ariansen J.L., Cheer J.F., Phillips P.E.M., Carelli R.M. Dopamine release is heterogeneous within microenvironments of the rat nucleus accumbens. Eur J Neurosci 2007, 26:2046-2054.
51. Goto Y., Grace A.A. Limbic and cortical information processing in the nucleus accumbens. Trends Neurosci 2008, 31:552-558.
52. Voorn P., Vanderschuren L.J., Groenewegen H.J., Robbins T.W., Pennartz C.M. Putting a spin on the dorsal-ventral divide of the striatum. Trends Neurosci 2004, 27:468-474.
53. Pennartz C.M.A., Groenewegen H.J., Lopes da Silva F.H. The nucleus accumbens as a complex of functionally distinct neuronal ensembles: an integration of behavioural, electrophysiological, and anatomical data. Prog Neurobiol 1994, 42:719-761.
54. Ito R., Robbins T.W., Pennartz C.M., Everitt B.J. Functional interaction between the hippocampus and nucleus accumbens shell is necessary for the acquisition of appetitive spatial context conditioning. J Neurosci 2008, 28:6950-6959.
55. Lansink C.S., Goltstein P.M., Lankelma J.V., McNaughton B.L., Pennartz C.M. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol 2009, 7:e1000173.
56. Berke J.D. Fast oscillations in cortical-striatal networks switch frequency following rewarding events and stimulant drugs. Eur J Neurosci 2009, 30:848-859.
57. van der Meer M.A.A., Redish A.D. Low and high gamma oscillations in rat ventral striatum have distinct relationships to behavior, reward, and spiking activity on a learned spatial decision task. Front Integrative Neurosci 2009, 3:9.
58. Kalenscher T., Lansink C.S., Lankelma J.V., Pennartz C.M.A. Reward-associated gamma oscillations in ventral striatum are regionally differentiated and modulate local firing activity. J Neurophysiol 2010, 103:1658-1672.
59. van der Meer M.A.A., Kalenscher T., Lansink C.S., Pennartz C.M.A., Berke J.D., Redish A.D. Integrating early results on ventral striatal gamma oscillations in the rat. Front Neurosci 2010, 4.
60. Niv Y., Joel D., Dayan P. A normative perspective on motivation. Trends Cogn Sci 2006, 10:375-381.
61. Redish A.D., Jensen S., Johnson A. A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 2008, 31:415-487.
62. O'Keefe J., Nadel L. The Hippocampus as a Cognitive Map 1978, Clarendon Press.
63. Redish A.D. Beyond the Cognitive Map: From Place Cells to Episodic Memory 1999, MIT Press.
64. Balleine B.W. Incentive processes in instrumental conditioning. Handbook of Contemporary Learning Theories 2001, 307-366. Lawrence Erlbaum Associates. R.R. Mowrer, S.B. Klein, P.A. Philadelphia (Eds.).
65. de Borchgrave R., Rawlins J.N.P., Dickinson A., Balleine B.W. Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats. Exp Brain Res 2002, 144:50-68.
66. Corbit L.H., Muir J.L., Balleine B.W. The role of the nucleus accumbens in instrumental conditioning: evidence of a functional dissociation between accumbens core and shell. J Neurosci 2001, 21:3251-3260.
67. Singh T. Nucleus accumbens core and shell are necessary for reinforcer devaluation effects on Pavlovian conditioned responding. Front Integrative Neurosci 2010, 4:126.
68. Lex B., Hauber W. The role of nucleus accumbens dopamine in outcome encoding in instrumental and Pavlovian conditioning. Neurobiol Learning Memory 2010, 93:283-290.
69. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Networks 2006, 19:1153-1160.
70. Maia T.V. Two-factor theory, the actor-critic model, and conditioned avoidance. Learning Behav 2010, 38:50-67.
71. German P.W., Fields H.L. Rat nucleus accumbens neurons persistently encode locations associated with morphine reward. J Neurophysiol 2007, 97:2094-2106.
72. Nicola S.M. The flexible approach hypothesis: unification of effort and cue-responding hypotheses for the role of nucleus accumbens dopamine in the activation of reward-seeking behavior. J Neurosci 2010, 30:16585-16600.
73. van der Meer M.A.A., Redish A.D. Covert expectation-of-reward in rat ventral striatum at decision points. Front Integrative Neurosci 2009, 3:1-15.
74. Buckner R.L. The role of the hippocampus in prediction and imagination. Ann Rev Psychol 2010, 61:27-48.
75. van der Meer M.A.A., Redish A.D. Expectancies in decision making, reinforcement learning, and ventral striatum. Front Neurosci 2010, 4:29-37.