A reinforcement learning approach to instrumental contingency degradation in rats

Journal of Physiology Paris

Volumn 105, Issue 1-3, 2011, Pages 36-44

  Dutech, Alain ^a   Coutureau, Etienne ^b,c   Marchand, Alain R ^b,c  

^a LORIA   (France)

^b UNIVERSITÉ DE BORDEAUX   (France)

^c UMR 5287   (France)

Author keywords

Contingency degradation; Instrumental; Model free learning; Prefrontal cortex; Rats; SARSA; Simulation

Indexed keywords

ALGORITHM; ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ANIMAL MODEL; ARTICLE; BRAIN DAMAGE; CONCEPTUAL FRAMEWORK; CONTINGENCY DEGRADATION; CONTROLLED STUDY; DECISION MAKING; FOOD INTAKE; INSTRUMENTAL CONDITIONING; MATHEMATICAL MODEL; MEDIAL PREFRONTAL CORTEX; MOTIVATION; NONHUMAN; PREFRONTAL CORTEX; PROBABILITY; RAT; REINFORCEMENT; TIME PERCEPTION;

ANIMALS; BEHAVIOR, ANIMAL; CONDITIONING, OPERANT; EXTINCTION, PSYCHOLOGICAL; MODELS, NEUROLOGICAL; NEURONS; RATS; REINFORCEMENT (PSYCHOLOGY); TOUCH; TOUCH PERCEPTION;

EID: 82555190923     PISSN: 09284257     EISSN: 17697115     Source Type: Journal    
DOI: 10.1016/j.jphysparis.2011.07.017     Document Type: Article

References (30)

1. Azzi R., Fix D.S.R., Keller F.S., Rocha e Silva M.I. Exteroceptive control of response under delayed reinforcement. J. Exp. Anal. Behav. 1964, 7:159-162.
2. Balleine B.W., Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37:407-419.
3. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010, 35:48-69.
4. Bradtke, S.J., Duff, M.O., 1995. Reinforcement learning methods for continuous-time Markov decision problems. In: Proc. Adv. Neural Inform. Process. Systems (NIPS).
5. Calvert A.L., Green L., Myerson J. Delay discounting of qualitatively different reinforcers in rats. J. Exp. Anal. Behav. 2010, 93:171-184.
6. Corbit L.H., Balleine B.W. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 2003, 146:145-157.
7. Coutureau E., Marchand A.R., Di Scala G. Goal-directed responding is sensitive to lesions to the prelimbic cortex or basolateral nucleus of the amygdala but not to their disconnection. Behav. Neurosci. 2009, 123:443-448.
8. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
9. Daw N.D., Courville A.C., Touretzky D.S. Representation and timing in theories of the dopamine system. Neural Comput. 2006, 18:1637-1677.
10. Doll B.B., Jacobs W.J., Sanfey A.G., Frank M.J. Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res. 2009, 1299:74-94.
11. Fiorillo C.D., Tobler P.N., Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 2003, 299:1898-1902.
12. Frank M.J., Claus E.D. Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychol. Rev. 2006, 113:300-326.
13. Fuster J.M. The prefrontal cortex-an update: time is of the essence. Neuron 2001, 30:319-333.
14. Gisquet-Verrier P., Delatour B. The role of the rat prelimbic/infralimbic cortex in working memory: not involved in the short-term maintenance but in monitoring and processing functions. Neuroscience 2006, 141:585-596.
15. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
16. Hammond L.J. The effect of contingency upon the appetitive conditioning of free-operant behavior. J. Exp. Anal. Behav. 1980, 34:297-304.
17. Joel D., Niv Y., Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks 2002, 15:535-547.
18. Killcross S., Coutureau E. Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 2003, 13:400-408.
19. Kim H., Sul J.H., Huh N., Lee D., Jung M.W. Role of striatum in updating values of chosen actions. J. Neurosci. 2009, 29:14701-14712.
20. Kobayashi S., Schultz W. Influence of reward delays on responses of dopamine neurons. J. Neurosci. 2008, 28:7837-7846.
21. Maia T.V. Reinforcement learning, conditioning, and the brain: successes and challenges. Cogn. Affect. Behav. Neurosci. 2009, 9:343-364.
22. Matsumoto K., Tanaka K. The role of the medial prefrontal cortex in achieving goals. Curr. Opin. Neurobiol. 2004, 14:178-185.
23. Naneix F., Marchand A.R., Di Scala G., Pape J.R., Coutureau E. A role for medial prefrontal dopaminergic innervation in instrumental conditioning. J. Neurosci. 2009, 29:6599-6606.
24. Niv Y., Daw N.D., Dayan P. Choice values. Nat. Neurosci. 2006, 9:987-988.
25. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 2007, 191:507-520.
26. Ostlund S.B., Winterbauer N.E., Balleine B.W. Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. J. Neurosci. 2009, 29:8280-8287.
27. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA, London, UK.
28. Tanaka S.C., Balleine B.W., O'Doherty J.P. Calculating consequences: brain systems that encode the causal effects of actions. J. Neurosci. 2008, 28:6750-6755.
29. Yin H.H., Knowlton B.J., Balleine B.W. Inactivation of dorsolateral striatum enhances sensitivity to changes in the action-outcome contingency in instrumental conditioning. Behav. Brain Res. 2006, 166:189-196.
30. Yoon T., Okada J., Jung M.W., Kim J.J. Prefrontal cortex and hippocampus subserve different components of working memory in rats. Learn. Memory 2008, 15:97-105.