Dual reward prediction components yield Pavlovian sign- and goal-tracking

PLoS ONE

Volumn 9, Issue 10, 2014, Pages

  Kaveri, Sivaramakrishnan ^a,b   Nakahara, Hiroyuki ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

ANIMAL EXPERIMENT; ARTICLE; BEHAVIOR; BIOLOGICAL MODEL; CONDITIONING; DUAL REWARD PREDICTION; GOAL TRACKING RESPONSE; LEARNING MODEL; MATHEMATICAL COMPUTING; NONHUMAN; PAVLOVIAN CONDITIONED APPROACH TASK; PREDICTION; PSYCHOLOGIC TEST; RAT; REINFORCEMENT; REWARD; SECONDARY REINFORCEMENT TASK; SIGN TRACKING RESPONSE; STIMULUS RESPONSE; ANIMAL; ANIMAL BEHAVIOR; ASSOCIATION; COMPUTER SIMULATION; INSTRUMENTAL CONDITIONING; LEARNING; MOTIVATION; PHYSIOLOGY; THEORETICAL MODEL;

ANIMALS; BEHAVIOR, ANIMAL; COMPUTER SIMULATION; CONDITIONING, OPERANT; CUES; GOALS; LEARNING; MODELS, THEORETICAL; RATS; REWARD;

EID: 84907903556     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0108142     Document Type: Article

References (50)

1. Dayan P (2009) Dopamine, reinforcement learning, and addiction. Pharmacopsychiatry 42 Suppl 1: S56-65.
2. Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8: 429-453.
3. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38: 329-337.
4. Ratcliff R, Frank MJ (2012) Reinforcement-based decision making in corticostriatal circuits: mutual constraints by neurocomputational and diffusion models. Neural Comput 24: 1186-1229.
5. Maia TV, Frank MJ (2011) From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 14: 154-162.
6. Doll BB, Hutchison KE, Frank MJ (2011) Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J Neurosci 31: 6188-6198.
7. Frank MJ, Doll BB, Oas-Terpstra J, Moreno F (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci 12: 1062-1068.
8. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16: 1936-1947.
9. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
10. Sutton RS, Barto AG (1990) Time-Derivative Models of Pavlovian Reinforcement. Learning and Computational Neuroscience: Foundations of Adaptive Networks: MIT Press. pp. 497-537.
11. Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36: 285-298.
12. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, et al. (2011) A selective role for dopamine in stimulus-reward learning. Nature 469: 53-57.
13. Flagel SB, Robinson TE, Clark JJ, Clinton SM, Watson SJ, et al. (2010) An animal model of genetic vulnerability to behavioral disinhibition and responsiveness to reward-related cues: implications for addiction. Neuropsychopharmacology 35: 388-400.
14. Flagel SB, Akil H, Robinson TE (2009) Individual differences in the attribution of incentive salience to reward-related cues: Implications for addiction. Neuropharmacology 56 Suppl 1: 139-148.
15. Meyer PJ, Lovic V, Saunders BT, Yager LM, Flagel SB, et al. (2012) Quantifying individual variation in the propensity to attribute incentive salience to reward cues. PloS one 7: e38987.
16. Tomie A, Aguado AS, Pohorecky LA, Benjamin D (2000) Individual differences in pavlovian autoshaping of lever pressing in rats predict stress-induced corticosterone release and mesolimbic levels of monoamines. Pharmacol Biochem Behav 65: 509-517.
17. Boakes RA (1977) Performance on learning to associate a stimulus with positive reinforcement. In: Davis H, Hurwitz, H., editor. Operant-Pavlovian Interactions. Hillsdale, NJ: Erlbaum. pp. 67-97.
18. Nakahara H, Hikosaka O (2012) Learning to represent reward structure: A key to adapting to complex environments. Neuroscience Research 74: 177-183.
19. Nakahara H (2014) Multiplexing signals in reinforcement learning with internal models and dopamine. Current Opinion in Neurobiology 25: 123-129.
20. Konorski J (1967) Integrative activity of the brain; an interdisciplinary approach. Chicago: University of Chicago Press. xii, 531 p.
21. Holland PC (1985) Element pretraining influences the content of appetitive serial compound conditioning in rats. J Exp Psychol Anim Behav Process 11: 367-387.
22. Holland PC (1990) Event representation in Pavlovian conditioning: image and action. Cognition 37: 105-131.
23. Pickens CL, Holland PC (2004) Conditioning and cognition. Neurosci Biobehav Rev 28: 651-661.
24. Balleine BW, Dickinson A (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37: 407-419.
25. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, et al. (2011) A selective role for dopamine in stimulus-reward learning. Nature 469: 53-57.
26. Rescorla RA (1988) Pavlovian Conditioning It's Not What You Think It Is. American Psychologist 43: 151-160.
27. Clark JJ, Hollon NG, Phillips PE (2012) Pavlovian valuation systems in learning and decision making. Curr Opin Neurobiol 22: 1054-1061.
28. Saunders BT, Robinson TE (2012) The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses. Eur J Neurosci 36: 2521-2532.
29. Lesaint F, Sigaud O, Flagel SB, Robinson TE, Khamassi M (2014) Modelling Individual Differences in the Form of Pavlovian Conditioned Approach Responses: A Dual Learning Systems Approach with Factored Representations. Plos Computational Biology 10.
30. Khamassi M, Mulder AB, Tabuchi E, Douchamps V, Wiener SI (2008) Anticipatory reward signals in ventral striatal neurons of behaving rats. Eur J Neurosci 28: 1849-1866.
31. Samejima K, Doya K (2007) Multiple representations of belief states and action values in corticobasal ganglia loops. Ann N Y Acad Sci 1104: 213-228.
32. Corbit LH, Muir JL, Balleine BW (2001) The role of the nucleus accumbens in instrumental conditioning: Evidence of a functional dissociation between accumbens core and shell. J Neurosci 21: 3251-3260.
33. Corbit LH, Balleine BW (2011) The general and outcome-specific forms of Pavlovian-instrumental transfer are differentially mediated by the nucleus accumbens core and shell. J Neurosci 31: 11786-11794.
34. Rolls ET, Critchley HD, Browning AS, Hernadi I, Lenard L (1999) Responses to the sensory properties of fat of neurons in the primate orbitofrontal cortex. J Neurosci 19: 1532-1540.
35. Rolls ET (2000) The orbitofrontal cortex and reward. Cereb Cortex 10: 284-294.
36. McDannald MA, Takahashi YK, Lopatina N, Pietras BW, Jones JL, et al. (2012) Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur J Neurosci 35: 991-996.
37. Takahashi YK, Roesch MR, Wilson RC, Toreson K, O'Donnell P, et al. (2011) Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 14: 1590-1597.
38. Rolls ET (2000) Memory systems in the brain. Annu Rev Psychol 51: 599-630.
39. Yaxley S, Rolls ET, Sienkiewicz ZJ (1988) The responsiveness of neurons in the insular gustatory cortex of the macaque monkey is independent of hunger. Physiol Behav 42: 223-229.
40. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8: 1704-1711.
41. Behrens TE, Woolrich MW, Walton ME, Rushworth MF (2007) Learning the value of information in an uncertain world. Nat Neurosci 10: 1214-1221.
42. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69: 1204-1215.
43. Saunders BT, Robinson TE (2012) The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses. European Journal of Neuroscience.
44. Anselme P, Robinson MJ, Berridge KC (2012) Reward uncertainty enhances incentive salience attribution as sign-tracking. Behavioural Brain Research.
45. Littman ML, Sutton RS, Singh S (2002) Predictive Representations of State. In: Dietterich TG, Becker S, Ghahramani Z, editors; Vancouver, British Columbia, Canada. MIT Press. pp. 1555-1561.
46. Singh SP, James MR, Rudary MR (2004) Predictive State Representations: A New Theory of Modeling Dynamical Systems. In: Chickering DM, Halpern JY, editors; Banff, Canada. AUAI Press. pp. 512-518.
47. Dayan P (1993) Improving generalization for temporal difference learning: the successor representation. Neural Computation 5: 613-624.
48. Gershman SJ, Moore CD, Todd MT, Norman KA, Sederberg PB (2012) The successor representation and temporal context. Neural Comput 24: 1553-1568.
49. Bromberg-Martin ES, Matsumoto M, Nakahara H, Hikosaka O (2010) Multiple timescales of memory in lateral habenula and dopamine neurons. Neuron 67: 499-510.
50. Bromberg-Martin ES, Matsumoto M, Hikosaka O (2010) Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons. Neuron 67: 144-155.