Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation

Cognitive, Affective and Behavioral Neuroscience

Volumn 14, Issue 2, 2014, Pages 473-492

  Dayan, Peter ^a   Berridge, Kent C ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Basal ganglia; Decision making; Dopamine; Pavlovian, classical conditioning; Reward, Motivation

Indexed keywords

BRAIN; COMPUTER SIMULATION; CONDITIONED REFLEX; DECISION MAKING; HUMAN; INSTRUMENTAL CONDITIONING; PHYSIOLOGY; PSYCHOLOGICAL MODEL; REWARD;

BRAIN; COMPUTER SIMULATION; CONDITIONING, CLASSICAL; CONDITIONING, OPERANT; DECISION MAKING; HUMANS; MODELS, PSYCHOLOGICAL; REWARD;

EID: 84904384936     PISSN: 15307026     EISSN: None     Source Type: Journal    
DOI: 10.3758/s13415-014-0277-8     Document Type: Review

References (184)

1. Anson, J. E., Bender, L., & Melvin, K. B. (1969). Sources of reinforcement in the establishment of self-punitive behavior. Journal of Comparative and Physiological Psychology, 67, 376-380.
2. Balleine, B. W. (1994). Asymmetrical interactions between thirst and hunger in Pavlovian-instrumental transfer. Quarterly Journal of Experimental Psychology, 47B, 211-231.
3. Balleine, B. W. (2005). Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits. Physiology and Behavior, 86, 717-730.
4. Balleine, B. W., & Dickinson, A. (1991). Instrumental performance following reinforcer devaluation depends upon incentive learning. Quarterly Journal of Experimental Psychology, 43, 279-296.
5. Balleine, B. W., Garner, C., Gonzalez, F., & Dickinson, A. (1995). Motivational control of heterogeneous instrumental chains. Journal of Experimental Psychology: Animal Behavior Processes, 21, 203.
6. Balleine, B. W., & O'Doherty, J. P. (2010). Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48-69.
7. Barron, H. C., Dolan, R. J., & Behrens, T. E. (2013). Online evaluation of novel choices by simultaneous representation of multiple memories. Nature Neuroscience, 16, 1492-1498. doi: 10.1038/nn.3515
8. Barto, A. G. (1995). Adaptive critics and the basal ganglia. In J. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215-232). Cambridge: MIT Press.
9. Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 834-846.
10. Berridge, K. C. (2007). The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology, 191, 391-431.
11. Berridge, K. C. (2012). From prediction error to incentive salience: Mesolimbic computation of reward motivation. European Journal of Neuroscience, 35, 1124-1143. doi: 10.1111/j.1460-9568.2012.07990.x
12. Berridge, K. C., & Schulkin, J. (1989). Palatability shift of a salt-associated incentive during sodium depletion. Quarterly Journal of Experimental Psychology, 41B, 121-138.
13. Berthoud, H. R., & Morrison, C. (2008). The brain, appetite, and obesity. Annual Review of Psychology, 59, 55-92.
14. Bindra, D. (1978). How adaptive behavior is produced: A perceptual-motivational alternative to response reinforcements. Behavioral and Brain Sciences, 1, 41-52. doi: 10.1017/S0140525X00059380. disc. 52-91.
15. Boileau, I., Dagher, A., Leyton, M., Gunn, R. N., Baker, G. B., Diksic, M., & Benkelfat, C. (2006). Modeling sensitization to stimulants in humans: An [11C]raclopride/positron emission tomography study in healthy men. Archives of General Psychiatry, 63, 1386-1395. doi: 10.1001/archpsyc.63.12.1386
16. Boorman, E. D., Behrens, T. E., Woolrich, M. W., & Rushworth, M. F. (2009). How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron, 62, 733-743.
17. Bouton, M. E., & Moody, E. W. (2004). Memory processes in classical conditioning. Neuroscience & Biobehavioral Reviews, 28, 663-674.
18. Bray, S., Rangel, A., Shimojo, S., Balleine, B. W., & O'Doherty, J. P. (2008). The neural mechanisms underlying the influence of pavlovian cues on human decision making. Journal of Neuroscience, 28, 5861-5866.
19. Breland, K., & Breland, M. (1961). The misbehavior of organisms. American Psychologist, 16, 681-684.
20. Brischoux, F., Chakraborty, S., Brierley, D. I., & Ungless, M. A. (2009). Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proceedings of the National Academy of Sciences, 106, 4894-4899.
21. Bushong, B., King, L. M., Camerer, C. F., & Rangel, A. (2010). Pavlovian processes in consumer choice: The physical presence of a good increases willingness-to-pay. American Economic Review, 100, 1556-1571.
22. Camille, N., Tsuchida, A., & Fellows, L. K. (2011). Double dissociation of stimulus-value and action-value learning in humans with orbitofrontal or anterior cingulate cortex damage. Journal of Neuroscience, 31, 15048-15052.
23. Campbell, M., Hoane, A. J., Jr., & Hsu, F.-H. (2002). Deep Blue. Artificial Intelligence, 134, 57-83.
24. Colwill, R. M., & Rescorla, R. A. (1988). Associations between the discriminative stimulus and the reinforcer in instrumental learning. Journal of Experimental Psychology: Animal Behavior Processes, 14, 155-164.
25. Corbit, L. H., & Balleine, B. W. (2003). Instrumental and Pavlovian incentive processes have dissociable effects on components of a heterogeneous instrumental chain. Journal of Experimental Psychology: Animal Behavior Processes, 29, 99-106.
26. Corbit, L. H., & Balleine, B. W. (2005). Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of Pavlovian-instrumental transfer. Journal of Neuroscience, 25, 962-970.
27. Corbit, L. H., Janak, P. H., & Balleine, B. W. (2007). General and outcome-specific forms of Pavlovian-instrumental transfer: the effect of shifts in motivational state and inactivation of the ventral tegmental area. European Journal of Neuroscience, 26, 3141-3149.
28. Darvas, M., & Palmiter, R. D. (2010). Restricting dopaminergic signaling to either dorsolateral or medial striatum facilitates cognition. Journal of Neuroscience, 30, 1158-1165.
29. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215. doi: 10.1016/j.neuron.2011.02.027
30. Daw, N. D., Kakade, S., & Dayan, P. (2002). Opponent interactions between serotonin and dopamine. Neural Networks, 15, 603-616.
31. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704-1711.
32. Dayan, P. (1993). Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5, 613-624.
33. Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience, 8, 429-453. doi: 10.3758/CABN.8.4.429
34. Dayan, P., & Huys, Q. J. M. (2008). Serotonin, inhibition, and negative mood. PLoS Computational Biology, 4, e4. doi: 10.1371/journal.pcbi. 0040004
35. Dayan, P., Niv, Y., Seymour, B., & Daw, N. D. (2006). The misbehavior of value and the discipline of the will. Neural Networks, 19, 1153-1160.
36. Dayan, P. (2012). How to set the switches on this thing. Current Opinion in Neurobiology, 22(6):1068-1074. doi: 10.1016/j.conb.2012.05.011
37. de la Fuente-Fernández, R., Phillips, A. G., Zamburlini, M., Sossi, V., Calne, D. B., Ruth, T. J., & Stoessl, A. J. (2002). Dopamine release in human ventral striatum and expectation of reward. Behavioural Brain Research, 136, 359-363.
38. de Wit, S., & Dickinson, A. (2009). Associative theories of goal-directed behaviour: A case for animal-human translational models. Psychological Research, 73, 463-476.
39. Dezfouli, A., & Balleine, B. W. (2012). Habits, action sequences and reinforcement learning. European Journal of Neuroscience, 35, 1036-1051.
40. Dezfouli, A., & Balleine, B. W. (2013). Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized. PLoS Computional Biology, 9, e1003364. doi: 10.1371/journal.pcbi.1003364
41. Dickinson, A. (1986). Re-examination of the role of the instrumental contingency in the sodium-appetite irrelevant incentive effect. Quarterly Journal of Experimental Psychology, 38B, 161-172.
42. Dickinson, A. (2012). Associative learning and animal cognition. Philosophical Transactions of the Royal Society B, 367, 2733-2742.
43. Dickinson, A., & Balleine, B. W. (2002). The role of learning in motivation. In C. R. Gallistel (Ed.), Stevens' handbook of experimental psychology (Vol. 3, pp. 497-533). New York: Wiley.
44. Dickinson, A., & Balleine, B. W. (2010). Hedonics: The cognitive-motivational interface. In M. L. Kringelbach & K. C. Berridge (Eds.), Pleasures of the brain (pp. 74-84). Oxford: Oxford University Press.
45. Dickinson, A., & Dawson, G. R. (1987). Pavlovian processes in the motivational control of instrumental performance. Quarterly Journal of Experimental Psychology, 39, 201-213.
46. Dickinson, A., & Dearing, M. F. (1979). Appetitive-aversive interactions and inhibitory processes. In A. Dickinson & R. A. Boakes (Eds.), Mechanisms of learning and motivation: A memorial to Jerzy Konorski (pp. 203-231). Hillsdale: Erlbaum.
47. Dickinson, A., Smith, J., & Mirenowicz, J. (2000). Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behavioral Neuroscience, 114, 468.
48. DiFeliceantonio, A. G., & Berridge, K. C. (2012). Which cue to "want"? Opioid stimulation of central amygdala makes goal-trackers show stronger goal-tracking, just as sign-trackers show stronger sign-tracking. Behavioural Brain Research, 230, 399-408. doi: 10.1016/j.bbr.2012.02.032
49. DiFeliceantonio, A. G., Mabrouk, O. S., Kennedy, R. T., & Berridge, K. C. (2012). Enkephalin surges in dorsal neostriatum as a signal to eat. Current Biology, 22, 1918-1924.
50. Doll, B. B., Jacobs, W. J., Sanfey, A. G., & Frank, M. J. (2009). Instructional control of reinforcement learning: A behavioral and neurocomputational investigation. Brain Research, 1299, 74-94.
51. Doll, B. B., Simon, D. A., & Daw, N. D. (2012). The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology, 22, 1075-1081.
52. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Networks, 12, 961-974.
53. Dwyer, D. M., Mackintosh, N. J., & Boakes, R. A. (1998). Simultaneous activation of the representations of absent cues results in the formation of an excitatory association between them. Journal of Experimental Psychology: Animal Behavior Processes, 24, 163-171. doi: 10.1037/0097-7403.24.2.163
54. Estes, W. K. (1943). Discriminative conditioning: I. A discriminative property of conditioned anticipation. Journal of Experimental Psychology, 32, 150-155.
55. Estes, W. K., & Skinner, B. F. (1941). Some quantitative properties of anxiety. Journal of Experimental Psychology, 29, 390-400.
56. Faure, A., Reynolds, S. M., Richard, J. M., & Berridge, K. C. (2008). Mesolimbic dopamine in desire and dread: enabling motivation to be generated by localized glutamate disruptions in nucleus accumbens. Journal of Neuroscience, 28, 7184-7192.
57. Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010). Evidence for model-based action planning in a sequential finger movement task. Journal of Motor Behavior, 42, 371-379.
58. Flagel, S. B., Clark, J. J., Robinson, T. E., Mayo, L., Czuj, A., Willuhn, I., & Akil, H. (2011). A selective role for dopamine in stimulus-reward learning. Nature, 469, 53-57. doi: 10.1038/nature09588
59. Floresco, S. B., West, A. R., Ash, B., Moore, H., & Grace, A. A. (2003). Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nature Neuroscience, 6, 968-973.
60. Foster, D. J., & Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680-683.
61. Foster, D. J., & Wilson, M. A. (2007). Hippocampal theta sequences. Hippocampus, 17, 1093-1099.
62. Fowler, H., & Miller, N. E. (1963). Facilitation and Inhibition of runway performance by hind- and forepaw shock of various intensities. Journal of Comparative and Physiological Psychology, 56, 801-805.
63. Fudim, O. K. (1978). Sensory preconditioning of flavors with a formalin-produced sodium need. Journal of Experimental Psychology: Animal Behavior Processes, 4, 276-285.
64. Ganesan, R., & Pearce, J. M. (1988). Effect of changing the unconditioned stimulus on appetitive blocking. Journal of Experimental Psychology: Animal Behavior Processes, 14, 280-291.
65. Gao, Q., & Horvath, T. L. (2008). Neuronal control of energy homeostasis. FEBS Letters, 582, 132-141.
66. Gershman, S. J., Markman, A. B., & Otto, A. R. (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General, 143, 182-194. doi: 10.1037/a0030844
67. Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66, 585-595. doi: 10.1016/j.neuron.2010.04.016
68. Goto, Y., & Grace, A. A. (2005). Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nature Neuroscience, 8, 805-812.
69. Hall, J., Parkinson, J. A., Connor, T. M., Dickinson, A., & Everitt, B. J. (2001). Involvement of the central nucleus of the amygdala and nucleus accumbens core in mediating Pavlovian influences on instrumental behaviour. European Journal of Neuroscience, 13, 1984-1992.
70. Hassabis, D., Kumaran, D., Vann, S. D., & Maguire, E. A. (2007). Patients with hippocampal amnesia cannot imagine new experiences. Proceedings of the National Academy of Sciences, 104, 1726-1731.
71. Herrnstein, R. J. (1990). Levels of stimulus control: A functional approach. Cognition, 37, 133-166.
72. Hikosaka, O., Nakahara, H., Rand, M. K., Sakai, K., Lu, X., Nakamura, K., & Doya, K. (1999). Parallel neural networks for learning sequential procedures. Trends in Neurosciences, 22, 464-471.
73. Hinton, G. E., & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society B, 352, 1177-1190.
74. Holland, P. C. (1977). Conditioned stimulus as a determinant of the form of the Pavlovian conditioned response. Journal of Experimental Psychology: Animal Behavior Processes, 3, 77-104. doi: 10.1037/0097-7403.3.1.77
75. Holland, P. C. (1990). Event representation in Pavlovian conditioning: Image and action. Cognition, 37, 105-131.
76. Holland, P. C. (2004). Relations between Pavlovian-instrumental transfer and reinforcer devaluation. Journal of Experimental Psychology: Animal Behavior Processes, 30, 104-117. doi: 10.1037/0097-7403.30.2.104
77. Holland, P. C., & Gallagher, M. (2003). Double dissociation of the effects of lesions of basolateral and central amygdala on conditioned stimulus-potentiated feeding and Pavlovian-instrumental transfer. European Journal of Neuroscience, 17, 1680-1694.
78. Holland, P. C., Lasseter, H., & Agarwal, I. (2008). Amount of training and cue-evoked taste-reactivity responding in reinforcer devaluation. Journal of Experimental Psychology: Animal Behavior Processes, 34, 119-132.
79. Holland, P. C., & Rescorla, R. A. (1975). The effect of two ways of devaluing the unconditioned stimulus after first-and second-order appetitive conditioning. Journal of Experimental Psychology: Animal Behavior Processes, 1, 355-363. doi: 10.1037/0097-7403.1.4.355
80. Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012). Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Computational Biology, 8, e1002410. doi: 10.1371/journal.pcbi.1002410
81. Hyman, S. E., Malenka, R. C., & Nestler, E. J. (2006). Neural mechanisms of addiction: The role of reward-related learning and memory. Annual Review of Neuroscience, 29, 565-598.
82. Jenkins, H. M., & Moore, B. R. (1973). The form of the auto-shaped response with food or water reinforcers. Journal of the Experimental Analysis of Behavior, 20, 163-181.
83. Johnson, A., & Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience, 27, 12176-12189.
84. Jones, J. L., Esber, G. R., McDannald, M. A., Gruber, A. J., Hernandez, A., Mirenzi, A., & Schoenbaum, G. (2012). Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science, 338, 953-956. doi: 10.1126/science.1227489
85. Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7, e1002055. doi: 10.1371/journal.pcbi.1002055
86. Killcross, S., & Coutureau, E. (2003). Coordination of actions and habits in the medial prefrontal cortex of rats. Cerebral Cortex, 13, 400-408.
87. Killcross, S., Robbins, T. W., & Everitt, B. J. (1997). Different types of fear-conditioned behaviour mediated by separate nuclei within amygdala. Nature, 388, 377-380.
88. Kishida, K. T., Sandberg, S. G., Lohrenz, T., Comair, Y. G., Sáez, I., Phillips, P. E. M., & Montague, P. R. (2011). Sub-second dopamine detection in human striatum. PLoS ONE, 6, e23291. doi: 10.1371/journal.pone. 0023291
89. Koob, G. F., & Volkow, N. D. (2010). Neurocircuitry of addiction. Neuropsychopharmacology, 35, 217-238.
90. Korotkova, T. M., Brown, R. E., Sergeeva, O. A., Ponomarenko, A. A., & Haas, H. L. (2006). Effects of arousal- and feeding-related neuropeptides on dopaminergic and GABAergic neurons in the ventral tegmental area of the rat. European Journal of Neuroscience, 23, 2677-2685.
91. Krause, E. G., & Sakai, R. R. (2007). Richter and sodium appetite: From adrenalectomy to molecular biology. Appetite, 49, 353-367.
92. Krieckhaus, E. E., & Wolf, G. (1968). Acquisition of sodium by rats: Interaction of innate mechanisms and latent learning. Journal of Comparative and Physiological Psychology, 65, 197-201.
93. Lammel, S., Lim, B. K., & Malenka, R. C. (2014). Reward and aversion in a heterogeneous midbrain dopamine system. Neuropharmacology, 76(Part B), 351-359.
94. Lammel, S., Lim, B. K., Ran, C., Huang, K. W., Betley, M. J., Tye, K. M., & Malenka, R. C. (2012). Input-specific control of reward and aversion in the ventral tegmental area. Nature, 491, 212-217. doi: 10.1038/nature11527
95. Leyton, M., & Vezina, P. (2012). On cue: Striatal ups and downs in addictions. Biological Psychiatry, 72, e21-e22.
96. Li, J., & Daw, N. D. (2011). Signals in human striatum are appropriate for policy update rather than value prediction. Journal of Neuroscience, 31, 5504-5511.
97. Lovibond, P. F. (1981). Appetitive Pavlovian-instrumental interactions: Effects of inter-stimulus interval and baseline reinforcement conditions. Quarterly Journal of Experimental Psychology, 33B, 257-269.
98. Lovibond, P. F. (1983). Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus. Journal of Experimental Psychology: Animal Behavior Processes, 9, 225-247. doi: 10.1037/0097-7403.9.3.225
99. Mackintosh, N. J. (1983). Conditioning and associative learning. Oxford: Oxford University Press.
100. Mahler, S. V., & Berridge, K. C. (2012). What and when to "want"? Amygdala-based focusing of incentive salience upon sugar and sex. Psychopharmacology, 221, 407-426.
101. Marr, D. (1982). Vision: A computational investigation into the human representation and processing of visual information. San Francisco: Freeman.
102. Matsumoto, M., & Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature, 459, 837-841. doi: 10.1038/nature08028
103. McAllister, D. E., McAllister, W. R., Hampton, S. R., & Scoles, M. T. (1980). Escape-from-fear performance as affected by handling method and an additional CS-shock treatment. Animal Learning & Behavior, 8, 417-423.
104. McClure, S. M., Daw, N. D., & Montague, P. R. (2003). A computational substrate for incentive salience. Trends in Neurosciences, 26, 423-428.
105. McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. (2011). Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. Journal of Neuroscience, 31, 2700-2705. doi: 10.1523/JNEUROSCI.5499-10.2011
106. McDannald, M. A., Takahashi, Y. K., Lopatina, N., Pietras, B. W., Jones, J. L., & Schoenbaum, G. (2012). Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. European Journal of Neuroscience, 35, 991-996. doi: 10.1111/j.1460-9568.2011.07982.x
107. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16, 1936-1947.
108. Moore, R. Y., & Bloom, F. E. (1978). Central catecholamine neuron systems: Anatomy and physiology of the dopamine systems. Annual Review of Neuroscience, 1, 129-169.
109. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 9, 1057-1063.
110. Morse, W. H., Mead, R. N., & Kelleher, R. T. (1967). Modulation of elicited behavior by a fixed-interval schedule of electric shock presentation. Science, 157, 215-217.
111. Murschall, A., & Hauber, W. (2006). Inactivation of the ventral tegmental area abolished the general excitatory influence of Pavlovian cues on instrumental performance. Learning and Memory, 13, 123-126.
112. Nieh, E. H., Kim, S. Y., Namburi, P., & Tye, K. M. (2013). Optogenetic dissection of neural circuits underlying emotional valence and motivated behaviors. Brain Research, 1511, 73-92.
113. Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191, 507-520. doi: 10.1007/s00213-006-0502-4
114. O'Doherty, J. P. (2011). Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Annals of the New York Academy of Sciences, 1239, 118-129.
115. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454. doi: 10.1126/science.1094285
116. Pavlov, I. P. (1927). Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex (G. V. Anrep, Trans.). London: Oxford University Press.
117. Pecina, S., & Berridge, K. C. (2013). Dopamine or opioid stimulation of nucleus accumbens similarly amplify cue-triggered "wanting" for reward: Entire core and medial shell mapped as substrates for PIT enhancement. European Journal of Neuroscience, 37, 1529-1540.
118. Pezzulo, G., Rigoli, F., & Chersi, F. (2013). The mixed instrumental controller: Using value of Information to combine habitual choice and mental simulation. Frontiers in Psychology, 4, 92.
119. Pfeiffer, B. E., & Foster, D. J. (2013). Hippocampal place-cell sequences depict future paths to remembered goals. Nature, 497, 74-79.
120. Phillips, P. E., Stuber, G. D., Heien, M. L., Wightman, R. M., & Carelli, R. M. (2003). Subsecond dopamine release promotes cocaine seeking. Nature, 422, 614-618.
121. Prevost, C., Liljeholm, M., Tyszka, J. M., & O'Doherty, J. P. (2012). Neural correlates of specific and general Pavlovian-to-instrumental transfer within human amygdalar subregions: A high-resolution fMRI study. Journal of Neuroscience, 32, 8383-8390.
122. Puterman, M. L. (2009). Markov decision processes: Discrete stochastic dynamic programming (Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics (Vol. 414). New York: Wiley.
123. Rescorla, R. A. (1973). Effect of US habituation following conditioning. Journal of Comparative and Physiological Psychology, 82, 137-143.
124. Rescorla, R. A. (1974). Effect of inflation of the unconditioned stimulus value following conditioning. Journal of Comparative and Physiological Psychology, 86, 101-106. doi: 10.1037/h0035964
125. Rescorla, R. A. (1988). Pavlovian conditioning: It's not what you think it is. American Psychologist, 43, 151-160. doi: 10.1037/0003-066X.43.3.151
126. Rescorla, R. A., & Freberg, L. (1978). Extinction of within-compound flavor associations. Learning and Motivation, 9, 411-427.
127. Rescorla, R. A., & Solomon, R. L. (1967). Two-process learning theory: Relationships between Pavlovian conditioning and instrumental learning. Psychological Review, 74, 151-182. doi: 10.1037/h0024475
128. Richard, J. M., & Berridge, K. C. (2011). Nucleus accumbens dopamine/glutamate interaction switches modes to generate desire versus dread: D1 alone for appetitive eating but D1 and D2 together for fear. Journal of Neuroscience, 31, 12866-12879.
129. Rizley, R. C., & Rescorla, R. A. (1972). Associations in second-order conditioning and sensory preconditioning. Journal of Comparative and Physiological Psychology, 81, 1-11.
130. Robinson, M. J. F., & Berridge, K. C. (2013). Instant transformation of learned repulsion into motivational "wanting. Current Biology, 23, 282-289. doi: 10.1016/j.cub.2013.01.016
131. Robinson, T. E., & Berridge, K. C. (1993). The neural basis of drug craving: An incentive-sensitization theory of addiction. Brain Research Reviews, 18, 247-291.
132. Robinson, T. E., & Berridge, K. C. (2008). The incentive sensitization theory of addiction: Some current issues. Philosophical Transactions of the Royal Society B, 363, 3137-3146.
133. Robinson, T. E., & Kolb, B. (2004). Structural plasticity associated with exposure to drugs of abuse. Neuropharmacology, 47, 33-46.
134. Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10, 1615-1624. doi: 10.1038/nn2013
135. Roitman, M. F., Stuber, G. D., Phillips, P. E., Wightman, R. M., & Carelli, R. M. (2004). Dopamine operates as a subsecond modulator of food seeking. Journal of Neuroscience, 24, 1265-1271.
136. Rossi, M. A., Sukharnikova, T., Hayrapetyan, V. Y., Yang, L., & Yin, H. H. (2013). Operant self-stimulation of dopamine neurons in the substantia nigra. PLoS ONE, 8, e65799. doi: 10.1371/journal.pone.0065799
137. Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems [Technical Report]. Cambridge: Cambridge University Engineering Department.
138. Salamone, J. D., & Correa, M. (2002). Motivational views of reinforcement: Implications for understanding the behavioral functions of nucleus accumbens dopamine. Behavioural Brain Research, 137, 3-25.
139. Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science, 310, 1337-1340.
140. Saunders, B. T., & Robinson, T. E. (2012). The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses. European Journal of Neuroscience, 36, 2521-2532.
141. Schacter, D. L., Addis, D. R., & Buckner, R. L. (2008). Episodic simulation of future events: Concepts, data, and applications. Annals of the New York Academy of Sciences, 1124, 39-60.
142. Schacter, D. L., Addis, D. R., Hassabis, D., Martin, V. C., Spreng, R. N., & Szpunar, K. K. (2012). The future of memory: Remembering, imagining, and the brain. Neuron, 76, 677-694.
143. Schulkin, J., Arnell, P., & Stellar, E. (1985). Running to the taste of salt in mineralocorticoid-treated rats. Hormones and Behavior, 19, 413-425.
144. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27.
145. Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annual Review of Psychology, 57, 87-115.
146. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
147. Simon, D. A., & Daw, N. D. (2011). Neural correlates of forward planning in a spatial decision task in humans. Journal of Neuroscience, 31, 5526-5539.
148. Smith, K. S., Berridge, K. C., & Aldridge, J. W. (2011). Disentangling pleasure from incentive salience and learning signals in brain reward circuitry. Proceedings of the National Academy of Sciences, 108, E255-E264.
149. Smith, K. S., Virkud, A., Deisseroth, K., & Graybiel, A. M. (2012). Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proceedings of the National Academy of Sciences, 109, 18932-18937.
150. Spreng, R. N., Mar, R. A., & Kim, A. S. N. (2009). The common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: A quantitative meta-analysis. Journal of Cognitive Neuroscience, 21, 489-510.
151. Stouffer, E. M., & White, N. M. (2005). A latent cue preference based on sodium depletion in rats. Learning and Memory, 12, 549-552.
152. Suri, R. E., & Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91, 871-890.
153. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
154. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.
155. Swanson, L. W. (1982). The projections of the ventral tegmental area and adjacent regions: A combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Research Bulletin, 9, 321-353.
156. Talmi, D., Seymour, B., Dayan, P., & Dolan, R. J. (2008). Human Pavlovian-instrumental transfer. Journal of Neuroscience, 28, 360-368.
157. Thistlethwaite, D. (1952). Conditions of irrelevant-incentive learning. Journal of Comparative and Physiological Psychology, 45, 517-525. doi: 10.1037/h0055075
158. Thomas, M. J., Kalivas, P. W., & Shaham, Y. (2008). Neuroplasticity in the mesolimbic dopamine system and cocaine addiction. British Journal of Pharmacology, 154, 327-342.
159. Timberlake, W., & Grant, D. L. (1975). Auto-shaping in rats to the presentation of another rat predicting food. Science, 190, 690-692.
160. Tindell, A. J., Berridge, K. C., Zhang, J., Peciña, S., & Aldridge, J. W. (2005). Ventral pallidal neurons code incentive motivation: Amplification by mesolimbic sensitization and amphetamine. European Journal of Neuroscience, 22, 2617-2634.
161. Tindell, A. J., Smith, K. S., Berridge, K. C., & Aldridge, J. W. (2009). Dynamic computation of incentive salience: "Wanting" what was never "liked. Journal of Neuroscience, 29, 12220-12228.
162. Toates, F. (1986). Motivational systems. Cambridge: Cambridge University Press.
163. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55, 189-208. doi: 10.1037/h0061626
164. Tolman, E. C. (1949). The nature and functioning of wants. Psychological Review, 56, 357-369. doi: 10.1037/h0063540
165. Tolman, E. C. (1955). Performance vectors and the unconscious. Acta Psychologica, 11, 31-40.
166. Tomie, A. (1996). Locating reward cue at response manipulandum (CAM) induces symptoms of drug abuse. Neuroscience & Biobehavioral Reviews, 20, 503-535.
167. van der Meer, M. A. A., Johnson, A., Schmitzer-Torbert, N. C., & Redish, A. D. (2010). Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron, 67, 25-32.
168. Vezina, P., & Leyton, M. (2009). Conditioned cues and the expression of stimulant sensitization in animals and humans. Neuropharmacology, 56(Suppl. 1), 160-168.
169. Volkow, N. D., Wang, G.-J., Fowler, J. S., & Tomasi, D. (2012). Addiction circuitry in the human brain. Annual Review of Pharmacology and Toxicology, 52, 321-336.
170. Wanat, M. J., Willuhn, I., Clark, J. J., & Phillips, P. E. (2009). Phasic dopamine release in appetitive behaviors and drug addiction. Current Drug Abuse Reviews, 2, 195-213.
171. Wassum, K. M., Ostlund, S. B., Balleine, B. W., & Maidment, N. T. (2011). Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling. Learning and Memory, 18, 475-483.
172. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Doctoral dissertation, University of Cambridge.
173. Wilson, C. L., Sherman, J. E., & Holman, E. W. (1981). Aversion to the reinforcer differentially affects conditioned reinforcement and instrumental responding. Journal of Experimental Psychology: Animal Behavior Processes, 7, 165-174.
174. Wirsig, C. R., & Grill, H. J. (1982). Contribution of the rat's neocortex to ingestive control: I. Latent learning for the taste of sodium chloride. Journal of Comparative and Physiological Psychology, 96, 615-627. doi: 10.1037/h0077911
175. Wise, R. A. (2009). Roles for nigrostriatal - not just mesocorticolimbic
176. Witten, I. B., Steinberg, E. E., Lee, S. Y., Davidson, T. J., Zalocusky, K. A., Brodsky, M., & Deisseroth, K. (2011). Recombinase-driver rat lines: Tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron, 72, 721-733. doi: 10.1016/j.neuron.2011.10.028
177. Wolf, M. E., & Ferrario, C. R. (2010). AMPA receptor plasticity in the nucleus accumbens after repeated exposure to cocaine. Neuroscience & Biobehavioral Reviews, 35, 185-211.
178. Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience, 15, 786-791. doi: 10.1038/nn.3068
179. Wyvell, C. L., & Berridge, K. C. (2000). Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward: enhancement of reward "wanting" without enhanced "liking" or response reinforcement. Journal of Neuroscience, 20, 8122-8130.
180. Yager, L. M., & Robinson, T. E. (2013). A classically conditioned cocaine cue acquires greater control over motivated behavior in rats prone to attribute incentive salience to a food cue. Psychopharmacology, 226, 217-228.
181. Zaghloul, K. A., Blanco, J. A., Weidemann, C. T., McGill, K., Jaggi, J. L., Baltuch, G. H., & Kahana, M. J. (2009). Human substantia nigra neurons encode unexpected financial rewards. Science, 323, 1496-1499. doi: 10.1126/science.1167342
182. Zener, K., & McCurdy, H. G. (1939). Analysis of motivational factors in conditioned behavior: I. The differential effect of changes in hunger upon conditioned, unconditioned, and spontaneous salivary secretion. Journal of Psychology, 8, 321-350.
183. Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., & Aldridge, J. W. (2009). A neural computational model of incentive salience. PLoS Computational Biology, 5, e1000437. doi: 10.1371/journal.pcbi.1000437
184. Zigman, J. M., Jones, J. E., Lee, C. E., Saper, C. B., & Elmquist, J. K. (2006). Expression of ghrelin receptor mRNA in the rat and the mouse brain. Journal of Comparative Neurology, 494, 528-548.