Pavlovian valuation systems in learning and decision making

Current Opinion in Neurobiology

Volumn 22, Issue 6, 2012, Pages 1054-1061

  Clark, Jeremy J ^a   Hollon, Nick G ^a,b   Phillips, Paul E M ^a,b  

^a UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

ANIMAL; BRAIN; DECISION MAKING; FEEDBACK SYSTEM; HUMAN; INSTRUMENTAL CONDITIONING; LEARNING; PHYSIOLOGY; REVIEW; SENSORY NERVE;

AFFERENT PATHWAYS; ANIMALS; BRAIN; CONDITIONING, OPERANT; DECISION MAKING; DOPAMINE; FEEDBACK, PHYSIOLOGICAL; HUMANS; LEARNING;

EID: 84878176981     PISSN: 09594388     EISSN: 18736882     Source Type: Journal    
DOI: 10.1016/j.conb.2012.06.004     Document Type: Review

References (83)

1. Tolman E.C. There is more than one kind of learning. Psychol Rev 1949, 56:144-155.
2. Rangel A., Camerer C., Montague P.R. A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci 2008, 9:545-556.
3. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
4. Toates F. The interaction of cognitive and stimulus-response processes in the control of behaviour. Neurosci Biobehav Rev 1998, 22:59-83.
5. Kahneman D. Maps of bounded rationality: psychology for behavioral economics. Am Econ Rev 2003, 95:1449-1475.
6. Redish A.D., Jensen S., Johnson A. A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 2008, 31:415-437. discussion 437-487.
7. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
8. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and non-reinforcement. Classical Conditioning II: Current Research and Theory 1972, 64-99. Appleton-Century-Crofts, New York. A.H. Black, W.F. Prokasy (Eds.).
9. Berridge K.C. Motivation concepts in behavioral neuroscience. Physiol Behav 2004, 81:179-209.
10. McClure S.M., Daw N.D., Montague P.R. A computational substrate for incentive salience. Trends Neurosci 2003, 26:423-428.
11. Balleine B.W., Daw N.D., O'Doherty J.P. Multiple forms of value learning and the function of dopamine. Neuroeconomics: Decision Making and the Brain 2009, 367-387. Elsevier, New York. P.W. Glimcher, C.F. Camerer, E. Fehr, R.A. Poldrack (Eds.).
12. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Netw 2006, 19:1153-1160.
13. Konorski J. Integrative Activity of the Brain 1967, University of Chicago Press, Chicago.
14. Rescorla R.A. Learning about qualitatively different outcomes during a blocking procedure. Learn Behav 1999, 27:140-151.
15. Kruse J.M., Overmier J.B., Konz W.A., Rokke E. Pavlovian conditioned stimulus effects upon instrumental choice behavior are reinforcer specific. Learn Motiv 1983, 14:165-181.
16. Ostlund S.B., Maidment N.T. Dopamine receptor blockade attenuates the general incentive motivational effects of noncontingently delivered rewards and reward-paired cues without affecting their ability to bias action selection. Neuropsychopharmacology 2012, 37:508-519.
17. Boakes R.A. Performance on learning to associate a stimulus with positive reinforcement. Operant-Pavlovian Interactions 1977, 67-97. Lawrence Erlbaum Associates, Hillsdale, NJ. H. Davis, H.M.B. Hurwitz (Eds.).
18. Wilcove W.G., Miller J.C. CS-USC presentations and a lever: human autoshaping. J Exp Psychol 1974, 103:868-877.
19. Wise R.A. Dopamine, learning and motivation. Nat Rev Neurosci 2004, 5:483-494.
20. Zweifel L., Parker J., Lobb C., Rainwater A., Wall V., Fadok J., Darvas M., Kim M., Mizumori S., Paladini C., et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc Natl Acad Sci USA 2009, 106:7281-7288.
21. Tsai H.C., Zhang F., Adamantidis A., Stuber G.D., Bonci A., de Lecea L., Deisseroth K. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 2009, 324:1080-1084.
22. Ljungberg T., Apicella P., Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 1992, 67:145-163.
23. Dommett E., Coizet V., Blaha C.D., Martindale J., Lefebvre V., Walton N., Mayhew J.E., Overton P.G., Redgrave P. How visual stimuli activate dopaminergic neurons at short latency. Science 2005, 307:1476-1479.
24. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 1996, 16:1936-1947.
25. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 2001, 412:43-48.
26. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
27. Pan W.X., Schmidt R., Wickens J.R., Hyland B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J Neurosci 2005, 25:6235-6242.
28. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482:85-88.
29. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
30. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
31. Pessiglione M., Seymour B., Flandin G., Dolan R.J., Frith C.D. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006, 442:1042-1045.
32. D'Ardenne K., McClure S.M., Nystrom L.E., Cohen J.D. BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 2008, 319:1264-1267.
33. Day J.J., Roitman M.F., Wightman R.M., Carelli R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci 2007, 10:1020-1028.
34. Flagel S.B., Clark J.J., Robinson T.E., Mayo L., Czuj A., Willuhn I., Akers C.A., Clinton S.M., Phillips P.E., Akil H. A selective role for dopamine in stimulus-reward learning. Nature 2011, 469:53-57.
35. Parkinson J.A., Dalley J.W., Cardinal R.N., Bamford A., Fehnert B., Lachenal G., Rudarakanchana N., Halkerston K.M., Robbins T.W., Everitt B.J. Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function. Behav Brain Res 2002, 137:149-163.
36. Robinson T.E., Flagel S.B. Dissociating the predictive and incentive motivational properties of reward-related cues through the study of individual differences. Biol Psychiatry 2009, 65:869-873.
37. Schoenbaum G., Roesch M.R., Stalnaker T.A., Takahashi Y.K. A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat Rev Neurosci 2009, 10:885-892.
38. Rolls E.T., Baylis L.L. Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex. J Neurosci 1994, 14:5437-5452.
39. Critchley H.D., Rolls E.T. Hunger and satiety modify the responses of olfactory and visual neurons in the primate orbitofrontal cortex. J Neurophysiol 1996, 75:1673-1686.
40. Schoenbaum G., Chiba A.A., Gallagher M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat Neurosci 1998, 1:155-159.
41. Tremblay L., Schultz W. Relative reward preference in primate orbitofrontal cortex. Nature 1999, 398:704-708.
42. Padoa-Schioppa C., Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 2006, 441:223-226.
43. Roesch M.R., Taylor A.R., Schoenbaum G. Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 2006, 51:509-520.
44. Kennerley S.W., Wallis J.D. Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables. Eur J Neurosci 2009, 29:2061-2073.
45. Kepecs A., Uchida N., Zariwala H.A., Mainen Z.F. Neural correlates, computation and behavioural impact of decision confidence. Nature 2008, 455:227-231.
46. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 2006, 26:8360-8367.
47. Hare T.A., O'Doherty J., Camerer C.F., Schultz W., Rangel A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 2008, 28:5623-5630.
48. Izquierdo A., Suda R.K., Murray E.A. Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J Neurosci 2004, 24:7540-7548.
49. Gallagher M., McMahan R.W., Schoenbaum G. Orbitofrontal cortex and representation of incentive value in associative learning. J Neurosci 1999, 19:6610-6614.
50. Kamin L.J. Predictability, surprise, attention and conditioning. Punishment and Aversive Behavior 1969, 279-296. Appleton-Century-Crofts, New York. B.A. Campbell, R.M. Church (Eds.).
51. McDannald M.A., Lucantonio F., Burke K.A., Niv Y., Schoenbaum G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 2011, 31:2700-2705.
52. Burke K.A., Franz T.M., Miller D.N., Schoenbaum G. The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 2008, 454:340-344.
53. Ostlund S.B., Balleine B.W. Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental conditioning. J Neurosci 2007, 27:4819-4825.
54. Ostlund S.B., Balleine B.W. The contribution of orbitofrontal cortex to action selection. Ann N Y Acad Sci 2007, 1121:174-192.
55. Schoenbaum G., Takahashi Y., Liu T.L., McDannald M.A. Does the orbitofrontal cortex signal value?. Ann N Y Acad Sci 2011, 1239:87-99.
56. Cardinal R.N., Parkinson J.A., Hall J., Everitt B.J. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 2002, 26:321-352.
57. Hatfield T., Han J.S., Conley M., Gallagher M., Holland P. Neurotoxic lesions of basolateral, but not central, amygdala interfere with Pavlovian second-order conditioning and reinforcer devaluation effects. J Neurosci 1996, 16:5256-5265.
58. Corbit L.H., Balleine B.W. Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of Pavlovian-instrumental transfer. J Neurosci 2005, 25:962-970.
59. Johnson A.W., Gallagher M., Holland P.C. The basolateral amygdala is critical to the expression of Pavlovian and instrumental outcome-specific reinforcer devaluation effects. J Neurosci 2009, 29:696-704.
60. Parkinson J.A., Robbins T.W., Everitt B.J. Dissociable roles of the central and basolateral amygdala in appetitive emotional learning. Eur J Neurosci 2000, 12:405-413.
61. Chang S.E., Wheeler D.S., Holland P.C. Effects of lesions of the amygdala central nucleus on autoshaped lever pressing. Brain Res 2012, 1450:49-56.
62. Ito R., Everitt B.J., Robbins T.W. The hippocampus and appetitive Pavlovian conditioning: effects of excitotoxic hippocampal lesions on conditioned locomotor activity and autoshaping. Hippocampus 2005, 15:713-721.
63. Takahashi Y.K., Roesch M.R., Stalnaker T.A., Haney R.Z., Calu D.J., Taylor A.R., Burke K.A., Schoenbaum G. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 2009, 62:269-280.
64. Lodge D.J. The medial prefrontal and orbitofrontal cortices differentially regulate dopamine system function. Neuropsychopharmacology 2011, 36:1227-1236.
65. Takahashi Y.K., Roesch M.R., Wilson R.C., Toreson K., O'Donnell P., Niv Y., Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 2011, 14:1590-1597.
66. van Zessen R., Phillips J.L., Budygin E.A., Stuber G.D. Activation of VTA GABA neurons disrupts reward consumption. Neuron 2012, 73:1184-1194.
67. Tsuchida A., Doll B.B., Fellows L.K. Beyond reversal: a critical role for human orbitofrontal cortex in flexible learning from probabilistic feedback. J Neurosci 2010, 30:16868-16875.
68. Walton M.E., Behrens T.E., Buckley M.J., Rudebeck P.H., Rushworth M.F. Separable learning systems in the macaque brain and the role of orbitofrontal cortex in contingent learning. Neuron 2010, 65:927-939.
69. Bromberg-Martin E.S., Matsumoto M., Hong S., Hikosaka O. A pallidus-habenula-dopamine pathway signals inferred stimulus values. J Neurophysiol 2010, 104:1068-1076.
70. Redish A.D., Jensen S., Johnson A., Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 2007, 114:784-805.
71. Gershman S.J., Blei D.M., Niv Y. Context, learning, and extinction. Psychol Rev 2010, 117:197-209.
72. Mogenson G.J., Jones D.L., Yim C.Y. From motivation to action: functional interface between the limbic system and the motor system. Prog Neurobiol 1980, 14:69-97.
73. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
74. Parkinson J.A., Willoughby P.J., Robbins T.W., Everitt B.J. Disconnection of the anterior cingulate cortex and nucleus accumbens core impairs Pavlovian approach behavior: further evidence for limbic cortical-ventral striatopallidal systems. Behav Neurosci 2000, 114:42-63.
75. Singh T., McDannald M.A., Haney R.Z., Cerri D.H., Schoenbaum G. Nucleus accumbens core and shell are necessary for reinforcer devaluation effects on Pavlovian conditioned responding. Front Integr Neurosci 2010, 4:126.
76. Corbit L.H., Balleine B.W. The general and outcome-specific forms of Pavlovian-instrumental transfer are differentially mediated by the nucleus accumbens core and shell. J Neurosci 2011, 31:11786-11794.
77. Shaham Y., Shalev U., Lu L., De Wit H., Stewart J. The reinstatement model of drug relapse: history, methodology and major findings. Psychopharmacology (Berl) 2003, 168:3-20.
78. Di Chiara G., Imperato A. Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats. Proc Natl Acad Sci USA 1988, 85:5274-5278.
79. Kalivas P.W., Volkow N.D. The neural basis of addiction: a pathology of motivation and choice. Am J Psychiatry 2005, 162:1403-1413.
80. Lucantonio F., Stalnaker T.A., Shaham Y., Niv Y., Schoenbaum G. The impact of orbitofrontal dysfunction on cocaine addiction. Nat Neurosci 2012, 15:358-366.
81. Volkow N.D., Fowler J.S., Wang G.J., Hitzemann R., Logan J., Schlyer D.J., Dewey S.L., Wolf A.P. Decreased dopamine D2 receptor availability is associated with reduced frontal metabolism in cocaine abusers. Synapse 1993, 14:169-177.
82. Bickel W.K., Marsch L.A. Toward a behavioral economic understanding of drug dependence: delay discounting processes. Addiction 2001, 96:73-86.
83. Bowen S., Chawla N., Collins S.E., Witkiewitz K., Hsu S., Grow J., Clifasefi S., Garner M., Douglass A., Larimer M.E., et al. Mindfulness-based relapse prevention for substance use disorders: a pilot efficacy trial. Subst Abus 2009, 30:295-305.