Modelling Individual Differences in the Form of Pavlovian Conditioned Approach Responses: A Dual Learning Systems Approach with Factored Representations

PLoS Computational Biology

Volumn 10, Issue 2, 2014, Pages

  Lesaint, Florian ^a,b   Sigaud, Olivier ^a,b   Flagel, Shelly B ^c,d,e   Robinson, Terry E ^e   Khamassi, Mehdi ^a,b  

^a UNIVERSITÉ PIERRE ET MARIE CURIE   (France)

^b CNRS   (France)

^c UNIVERSITY OF MICHIGAN HEALTH SYSTEM   (United States)

^d UNIVERSITY OF MICHIGAN   (United States)

^e UNIVERSITY OF MICHIGAN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMINES; NEUROPHYSIOLOGY; PHYSIOLOGICAL MODELS; RATS; REINFORCEMENT LEARNING;

CLASSICAL MODELING; DOPAMINE; FOOD DELIVERY; INDIVIDUAL DIFFERENCES; LEARN+; MODEL FREE; PHYSIOLOGICAL OBSERVATION; REINFORCEMENT LEARNINGS; REWARD-PREDICTION ERROR; SYSTEMS APPROACH;

FORECASTING;

DOPAMINE; DOPAMINE RECEPTOR BLOCKING AGENT; FLUPENTIXOL;

ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; BRAIN REGION; CONDITIONED REFLEX; DOPAMINERGIC ACTIVITY; EXPERIMENTAL MODEL; LEARNING; MOTIVATION; NEUROSCIENCE; NONHUMAN; PREDICTION; RAT; REINFORCEMENT; REWARD; STIMULUS RESPONSE; ALGORITHM; ANIMAL; BIOLOGICAL MODEL; BIOLOGY; BRAIN; CONDITIONING; DRUG EFFECTS; PHYSIOLOGY; PSYCHOLOGICAL MODEL; SYSTEMS BIOLOGY;

ALGORITHMS; ANIMALS; BEHAVIOR, ANIMAL; BRAIN; COMPUTATIONAL BIOLOGY; CONDITIONING (PSYCHOLOGY); DOPAMINE; DOPAMINE ANTAGONISTS; FLUPENTHIXOL; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; RATS; REINFORCEMENT (PSYCHOLOGY); SYSTEMS BIOLOGY;

EID: 84895773656     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1003466     Document Type: Article

References (87)

1. Sutton RS, Barto AG (1998) Reinforcement learning: An introduction. The MIT Press.
2. Sutton RS, Barto AG (1987) A temporal-difference model of classical conditioning. In: Proceedings of the ninth annual conference of the cognitive science society. Seattle, WA, pp. 355-378.
3. Barto AG (1995) Adaptive critics and the basal ganglia. In: Houk JC, Davis JL, Beiser DG, editors, Models of information processing in the basal ganglia, The MIT Press. pp. 215-232.
4. Clark JJ, Hollon NG, Phillips PEM, (2012) Pavlovian valuation systems in learning and decision making. Curr Opin Neurobiol 22: 1054-1061.
5. Simon DA, Daw ND (2012) Dual-system learning models and drugs of abuse. In: Computational Neuroscience of Drug Addiction, Springer. pp. 145-161.
6. Cardinal RN, Parkinson JA, Hall J, Everitt BJ, (2002) Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 26: 321-352.
7. Yin HH, Ostlund SB, Knowlton BJ, Balleine BW, (2005) The role of the dorsomedial striatum in instrumental conditioning. Eur J neurosci 22: 513-523.
8. Solway A, Botvinick MM, (2012) Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol Rev 119: 120-154.
9. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ, (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69: 1204-1215.
10. Graybiel AM, (2008) Habits, rituals, and the evaluative brain. Annu Rev Neurosci 31: 359-387.
11. Yin HH, Knowlton BJ, Balleine BW, (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J neurosci 19: 181-189.
12. Schultz W, (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1-27.
13. Fiorillo CD, Tobler PN, Schultz W, (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898-1902.
14. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, et al. (2011) A selective role for dopamine in stimulus-reward learning. Nature 469: 53-57.
15. Danna CL, Elmer GI, (2010) Disruption of conditioned reward association by typical and atypical antipsychotics. Pharmacol Biochem Behav 96: 40-47.
16. Dayan P, Niv Y, Seymour B, Daw ND, (2006) The misbehavior of value and the discipline of the will. Neural Netw 19: 1153-1160.
17. Daw ND, Niv Y, Dayan P, (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-1711.
18. Keramati M, Dezfouli A, Piray P, (2011) Speed/Accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol 7: e1002055.
19. Gläscher J, Daw ND, Dayan P, O'Doherty JP, (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66: 585-595.
20. Flagel SB, Watson SJ, Robinson TE, Akil H, (2007) Individual differences in the propensity to approach signals vs goals promote different adaptations in the dopamine system of rats. Psychopharmacology 191: 599-607.
21. Flagel SB, Akil H, Robinson TE, (2009) Individual differences in the attribution of incentive salience to reward-related cues: Implications for addiction. Neuropharmacology 56: 139-148.
22. Robinson TE, Flagel SB, (2009) Dissociating the predictive and incentive motivational properties of reward-related cues through the study of individual differences. Biol psychiatry 65: 869-873.
23. Mahler SV, Berridge KC, (2009) Which cue to "want?" Central amygdala opioid activation enhances and focuses incentive salience on a prepotent reward cue. J Neurosci 29: 6500-13.
24. DiFeliceantonio AG, Berridge KC, (2012) Which cue to 'want'? Opioid stimulation of central amygdala makes goal-trackers show stronger goal-tracking, just as sign-trackers show stronger sign-tracking. Behav Brain Res 230: 399-408.
25. Saunders BT, Robinson TE, (2012) The role of dopamine in the accumbens core in the expression of pavlovian-conditioned responses. Eur J neurosci 36: 2521-2532.
26. Meyer PJ, Lovic V, Saunders BT, Yager LM, Flagel SB, et al. (2012) Quantifying individual variation in the propensity to attribute incentive salience to reward cues. PLoS ONE 7: e38987.
27. Berridge KC, (2007) The debate over dopamines role in reward: the case for incentive salience. Psychopharmacology 191: 391-431.
28. Lovic V, Saunders BT, Yager LM, Robinson TE, (2011) Rats prone to attribute incentive salience to reward cues are also prone to impulsive action. Behav Brain Res 223: 255-261.
29. Williams BA, (1994) Conditioned reinforcement: Experimental and theoretical issues. Behav Anal 17: 261-285.
30. Skinner BF (1938) The behavior of organisms: An experimental analysis. Appleton-Century-Crofts New York, 82-82 pp.
31. Lomanowska AM, Lovic V, Rankine MJ, Mooney SJ, Robinson TE, et al. (2011) Inadequate early social experience increases the incentive salience of reward-related cues in adulthood. Behav Brain Res 220: 91-99.
32. Humphries MD, Khamassi M, Gurney K, (2012) Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Front Neurosci 6: 9.
33. Khamassi M, Humphries MD, (2012) Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies. Front Behav Neurosci 6.
34. Huys QJM, Eshel N, O'Nions E, Sheridan L, Dayan P, et al. (2012) Bonsai trees in your head: How the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput Biol 8: e1002410.
35. Doya K, Samejima K, Katagiri Ki, Kawato M, (2002) Multiple model-based reinforcement learning. Neural Comput 14: 1347-1369.
36. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z, (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychol Rev 114: 784-805.
37. Takahashi YK, Roesch MR, Stalnaker TA, Haney RZ, Calu DJ, et al. (2009) The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62: 269-280.
38. McDannald MA, Lucantonio F, Burke KA, Niv Y, Schoenbaum G, (2011) Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 31: 2700-2705.
39. McDannald MA, Takahashi YK, Lopatina N, Pietras BW, Jones JL, et al. (2012) Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur J neurosci 35: 991-996.
40. Cleland GG, Davey GCL, (1983) Autoshaping in the rat: The effects of localizable visual and auditory signals for food. J Exp Anal Behav 40: 47-56.
41. Meyer PJ, Aldridge JW, Robinson TE (2010) Auditory and visual cues are differentially attributed with incentive salience but similarly affected by amphetamine, 2010 neuroscience meeting planner. In: Society for Neuroscience Annual Meeting (SfN10).
42. Schmajuk NA, Lam YW, Gray JA, (1996) Latent inhibition: A neural network approach. J Exp Psychol Anim Behav Process 22: 321-349.
43. Balkenius C, (1999) Dynamics of a classical conditioning model. Auton Robots 7: 41-56.
44. Stout SC, Miller RR, (2007) Sometimes-competing retrieval (SOCR): A formalization of the comparator hypothesis. Psychol Rev 114: 759-783.
45. Courville AC, Daw ND, Touretzky DS, (2006) Bayesian theories of conditioning in a changing world. Trends Cogn Sci 10: 294-300.
46. Gershman SJ, Niv Y, (2012) Exploring a latent cause theory of classical conditioning. Anim Learn Behav 40: 255-268.
47. Kamin LJ (1967) Predictability, surprise, attention, and conditioning. In: Campbell BA, Church RMa, editors, Punishment and aversive behavior, New York: Appleton-Century-Crofts. pp. 279-296.
48. Lattal KM, Nakajima S, (1998) Overexpectation in appetitive pavlovian and instrumental conditioning. Anim Learn Behav 26: 351-360.
49. Bellman R (1957) Dynamic programming. Princeton University Press.
50. Khamassi M, Martinet LE, Guillot A (2006) Combining self-organizing maps with mixtures of experts: application to an actor-critic model of reinforcement learning in the basal ganglia. In: From Animals to Animats 9, Springer. pp. 394-405.
51. Elfwing S, Uchibe E, Doya K, (2013) Scaled free-energy based reinforcement learning for robust and efficient learning in high-dimensional state spaces. Front Neurorobot 7: 3.
52. Boutilier C, Dearden R, Goldszmidt M, (2000) Stochastic dynamic programming with factored representations. Artif Intell 121: 49-107.
53. Degris T, Sigaud O, Wuillemin PH (2006) Learning the structure of factored markov decision processes in reinforcement learning problems. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp. 257-264.
54. Vigorito CM, Barto AG (2008) Autonomous hierarchical skill acquisition in factored mdps. In: Yale Workshop on Adaptive and Learning Systems, New Haven, Connecticut. volume 63, p. 109.
55. Guitart-Masip M, Huys QJM, Fuentemilla L, Dayan P, Duzel E, et al. (2012) Go and no-go learning in reward and punishment: interactions between affect and effect. Neuroimage 62: 154-166.
56. Huys QJM, Cools R, Gölzer M, Friedel E, Heinz A, et al. (2011) Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding. PLoS Comput Biol 7: e1002028.
57. Yin HH, Ostlund SB, Balleine BW, (2008) Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur J neurosci 28: 1437-1448.
58. Corbit LH, Balleine BW, (2005) Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of pavlovian-instrumental transfer. J Neurosci 25: 962-970.
59. Balsam PD, Payne D, (1979) Intertrial interval and unconditioned stimulus durations in autoshaping. Anim Learn Behav 7: 477-482.
60. Gibbon J, Balsam P (1981) Spreading association in time, Academic Press. pp. 219-253.
61. Gallistel CR, Gibbon J, (2000) Time, rate, and conditioning. Psychol Rev 107: 289-344.
62. Tomie A, Festa ED, Sparta DR, Pohorecky LA, (2003) Lever conditioned stimulus-directed autoshaping induced by saccharin-ethanol unconditioned stimulus solution: effects of ethanol concentration and trial spacing. Alcohol 30: 35-44.
63. Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H, (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9: 1057-1063.
64. Roesch MR, Calu DJ, Schoenbaum G, (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10: 1615-1624.
65. Bellot J, Sigaud O, Khamassi M (2012) Which temporal difference learning algorithm best reproduces dopamine activity in a multi-choice task? In: From Animals to Animats 12, Springer. pp. 289-298.
66. Tomie A, Lincks M, Nadarajah SD, Pohorecky LA, Yu L, (2012) Pairings of lever and food induce pavlovian conditioned approach of sign-tracking and goal-tracking in c57bl/6 mice. Behav Brain Res 226: 571-578.
67. Kobayashi S, Schultz W, (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28: 7837-7846.
68. Daw ND, Courville AC, Touretzky DS, (2006) Representation and timing in theories of the dopamine system. Neural Comput 18: 1637-1677.
69. Fiorillo CD, Newsome WT, Schultz W, (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11: 966-973.
70. Gurney KN, Humphries MD, Wood R, Prescott TJ, Redgrave P, (2004) Testing computational hypotheses of brain systems function: a case study with the basal ganglia. Network 15: 263-290.
71. Robinson MJF, Berridge KC, (2013) Instant transformation of learned repulsion into motivational "wanting". Current Biology 23: 282-289.
72. Panlilio LV, Thorndike EB, Schindler CW, (2007) Blocking of conditioning to a cocaine-paired stimulus: testing the hypothesis that cocaine perpetually produces a signal of larger-than-expected reward. Pharmacol Biochem Behav 86: 774-777.
73. Redish AD, (2004) Addiction as a computational process gone awry. Science 306: 1944-1947.
74. Daw ND, Niv Y, Dayan P (2006) Actions, policies, values and the basal ganglia. In: Bezard E, editor, Recent Breakthroughs in Basal Ganglia Research, Nova Science Publishers, Inc Hauppauge, NY. pp. 91-106.
75. Yin HH, Knowlton BJ, (2006) The role of the basal ganglia in habit formation. Nat Rev Neurosci 7: 464-476.
76. Thorn CA, Atallah H, Howe M, Graybiel AM, (2010) Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 66: 781-795.
77. Bornstein AM, Daw ND, (2011) Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Curr Opin Neurobiol 21: 374-380.
78. van der Meer M, Kurth-Nelson Z, Redish AD, (2012) Information processing in decision-making systems. Neuroscientist 18: 342-359.
79. Flagel SB, Cameron CM, Pickup KN, Watson SJ, Akil H, et al. (2011) A food predictive cue must be attributed with incentive salience for it to induce c-fos mRNA expression in cortico-striatalthalamic brain regions. Neuroscience 196: 80-96.
80. Mink JW, (1996) The basal ganglia: focused selection and inhibition of competing motor programs. Prog Neurobiol 50: 381-425.
81. Redgrave P, Prescott TJ, Gurney K, (1999) The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89: 1009-1023.
82. Gurney K, Prescott TJ, Redgrave P, (2001) A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol Cybern 84: 401-410.
83. Baird III LC (1993) Advantage updating. Technical report, DTIC Document.
84. Dayan P, Balleine BW, (2002) Reward, motivation, and reinforcement learning. Neuron 36: 285-298.
85. Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE, (1991) Adaptive mixtures of local experts. Neural Comput 3: 79-87.
86. Deb K, Pratap A, Agarwal S, Meyarivan T, (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evol Comput 6: 182-197.
87. Mouret JB, Doncieux S (2010) SFERESv2: Evolvin' in the Multi-Core World. In: WCCI 2010 IEEE World Congress on Computational Intelligence, Congress on Evolutionary Computation (CEC). pp. 4079-4086.