Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters

Progress in Brain Research

Volumn 202, Issue , 2013, Pages 441-464

  Khamassi, Mehdi ^a,b,c,d   Enel, Pierre ^a,b   Dominey, Peter Ford ^a,b   Procyk, Emmanuel ^a,b  

^a STEM CELL AND BRAIN RESEARCH INSTITUTE   (France)

^b UNIVERSITÉ DE LYON   (France)

^c UNIVERSITÉ PIERRE ET MARIE CURIE   (France)

^d CNRS   (France)

Author keywords

Computational modeling; Decision making; Medial prefrontal cortex; Metalearning; Neurorobotics; Reinforcement learning

Indexed keywords

ADAPTIVE BEHAVIOR; ANTERIOR CINGULATE; ARTICLE; BEHAVIORAL SCIENCE; DECISION MAKING; ELECTROENCEPHALOGRAM; ERROR; HAPLORHINI; HUMAN; HUMAN COMPUTER INTERACTION; NEUROPHYSIOLOGY; NONHUMAN; PREDICTION; PREFRONTAL CORTEX; PRIORITY JOURNAL; PROBABILITY; REINFORCEMENT; REWARD; ROBOTICS; SIMULATION; STOCHASTIC MODEL; TASK PERFORMANCE; VENTROMEDIAL PREFRONTAL CORTEX;

EID: 84872378856     PISSN: 00796123     EISSN: 18757855     Source Type: Book Series    
DOI: 10.1016/B978-0-444-62604-2.00022-8     Document Type: Chapter

References (67)

1. Alexander W.H., Brown J.W. Medial prefrontal cortex as an action-outcome predictor. Nat. Neurosci. 2011, 14:1338-1344.
2. Alexander G.E., Crutcher M.D., DeLong M.R. Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor, " prefrontal" and " limbic" functions. Prog. Brain Res. 1990, 85:119-146.
3. Amiez C., Joseph J.P., Procyk E. Primate anterior cingulate cortex and adaptation of behaviour. From Monkey Brain to Human Brain 2005, MIT Press, Cambridge, MA. S. Dehaene, J.R. Duhamel, M.D. Hauser, G. Rizzolatti (Eds.).
4. Amiez C., Joseph J.P., Procyk E. Anterior cingulate error-related activity is modulated by predicted reward. Eur. J. Neurosci. 2005, 21:3447-3452.
5. Amiez C., Joseph J.P., Procyk E. Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex 2006, 16:1040-1055.
6. Auer P., Cesa-Bianchi N., Fischer P. Finite-time analysis of the multiarmed bandit. Mach. Learn. 2002, 47:235-256.
7. Badre D., Wagner A.D. Selection, integration, and conflict monitoring; assessing the nature and generality of prefrontal cognitive control mechanisms. Neuron 2004, 41:473-487.
8. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
9. Behrens T.E., Woolrich M.W., Walton M.E., Rushworth M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
10. Botvinick M.M., Braver T.S., Barch D.M., Carter C.S., Cohen J.D. Conflict monitoring and cognitive control. Psychol. Rev. 2001, 108:624-652.
11. Brown J.W., Braver T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 2005, 307:1118-1121.
12. Cesa-Bianchi N., Gabor L., Stoltz G. Regret minimization under partial monitoring. Math. Oper. Res. 2006, 31:562-580.
13. Chavarriaga R., Strösslin T., Sheynikhovich D., Gerstner W. A computational model of parallel navigation systems in rodents. Neuroinformatics 2005, 3:223-242.
14. Cohen J.D., McClure S.M., Yu A.J. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362:933-942.
15. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
16. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879.
17. Dehaene S., Kerszberg M., Changeux J.P. A neuronal model of a global workspace in effortful cognitive tasks. Proc. Natl. Acad. Sci. USA 1998, 95:14529-14534.
18. Doya K. Metalearning and neuromodulation. Neural Netw. 2002, 15:495-506.
19. Doya K. Modulators of decision making. Nat. Neurosci. 2008, 11:410-416.
20. Durstewitz D., Seamans J.K. The dual-state theory of prefrontal cortex dopamine function with relevance to catechol-o-methyltransferase genotypes and schizophrenia. Biol. Psychiatry 2008, 64:739-749.
21. Fluxe K., Hokfelt T., Johansson O., Jonsson G., Lidbrink P., Ljungdahl A. The origin of the dopamine nerve terminals in limbic and frontal cortex. Evidence for mesocortico dopamine neurons. Brain Res. 1974, 82:349-355.
22. Frank M.J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 2005, 17(1):51-72.
23. Frank M.J., Doll B.B., Oas-Terpstra J., Moreno F. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci. 2009, 12:1062-1068.
24. Garivier A., Moulines E. On upper-confidence bound policies for non-stationary bandit problems 2008, Arxiv, preprint arXiv:0805.3415.
25. Hartland C., Gelly S., Baskiotis N., Teytaud O., Sebag M. Multi-armed bandit, dynamic environments and meta-bandits 2006.
26. Holroyd C.B., Coles M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 2002, 109:679-709.
27. Houk J.C., Adams J., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia 1995, 249-270. MIT Press, Cambridge, MA.
28. Humphries M.D., Prescott T.J. The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog. Neurobiol. 2010, 90:385-417.
29. Ishii S., Yoshida W., Yoshimoto J. Control of exploitation-exploration meta-parameter in reinforcement learning. Neural Netw. 2002, 15:665-687.
30. Johnston K., Levin H.M., Koval M.J., Everling S. Top-down control-signal dynamics in anterior cingulate and prefrontal cortex neurons following task switching. Neuron 2007, 53:453-462.
31. Kennerley S.W., Walton M.E., Behrens T.E., Buckley M.J., Rushworth M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 2006, 9:940-947.
32. Khamassi M., Lachèze L., Girard B., Berthoz A., Guillot A. Actor-critic models of reinforcement learning in the basal ganglia: from natural to artificial rats. Adapt. Behav. 2005, 13(2):131-148.
33. Khamassi M., Mulder A.B., Tabuchi E., Douchamps V., Wiener S.I. Anticipatory reward signals in ventral striatal neurons of behaving rats. Eur. J. Neurosci. 2008, 28:1849-1866.
34. Khamassi M., Lallée S., Enel P., Procyk E., Dominey P.F. Robot cognitive control with a neurophysiologically inspired reinforcement learning model. Front. Neurorobot. 2011, 5:1.
35. Kolling N., Behrens T.E., Mars R.B., Rushworth M.F. Neural mechanisms of foraging. Science 2012, 336(6077):95-98.
36. Kouneiher F., Charron S., Koechlin E. Motivation and cognitive control in the human prefrontal cortex. Nat. Neurosci. 2009, 12:939-945.
37. Krichmar J.L. The neuromodulatory system-a framework for survival and adaptive behavior in a challenging world. Adapt. Behav. 2008, 16:385-399.
38. Luksys G., Gerstner W., Sandi C. Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning. Nat. Neurosci. 2009, 12:1180-1186.
39. MacDonald A.W., Cohen J.D., Stenger V.A., Carter C.S. Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. Science 2000, 288:1835-1838.
40. Mars R.B., Sallet J., Rushwort M.F.S., Yeung N. Neural Basis of Motivational and Cognitive Control 2011, MIT Press, Cambridge, MA.
41. Matsumoto M., Matsumoto K., Abe H., Tanaka K. Medial prefrontal cell activity signaling prediction errors of action values. Nat. Neurosci. 2007, 10:647-656.
42. Miller E.K., Cohen J.D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 2001, 24:167-202.
43. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
44. Paus T. Primate anterior cingulate cortex: where motor control, drive and cognition interface. Nat. Rev. Neurosci. 2001, 2:417-424.
45. Procyk E., Goldman-Rakic P.S. Modulation of dorsolateral prefrontal delay activity during self-organized behavior. J. Neurosci. 2006, 26:11313-11323.
46. Procyk E., Joseph J.P. Characterization of serial order encoding in the monkey anterior cingulate sulcus. Eur. J. Neurosci. 2001, 14:1041-1046.
47. Procyk E., Tanaka Y.L., Joseph J.P. Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 2000, 3:502-508.
48. Quilodran R., Rothé M., Procyk E. Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 2008, 57(2):314-325.
49. Reynolds J.N., Hyland B.I., Wickens J.R. A cellular mechanism of reward-related learning. Nature 2001, 413:67-70.
50. Rothé M., Quilodran R., Sallet J., Procyk E. Coordination of high gamma activity in anterior cingulate and lateral prefrontal cortical areas during adaptation. J. Neurosci. 2011, 31:11110-11117.
51. Rudebeck P.H., Behrens T.E., Kennerley S.W., Baxter M.G., Buckley M.J., Walton M.E., Rushworth M.F. Frontal cortex subregions play distinct roles in choices between actions and stimuli. J. Neurosci. 2008, 28:13775-13785.
52. Rushworth M.F., Behrens T.E., Rudebeck P.H., Walton M.E. Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour. Trends Cogn Sci 2007, 11(4):168-176.
53. Rushworth M.F., Behrens T.E. Choice, uncertainty and value in prefrontal and cingulate cortex. Nat. Neurosci. 2008, 11:389-397.
54. Sallet J., Quilodran R., Rothé M., Vezoli J., Joseph J.P., Procyk E. Expectations, gains, and losses in the anterior cingulate cortex. Cogn. Affect. Behav. Neurosci. 2007, 7:327-336.
55. Samejima K., Ueda Y., Doya K., Kimura M. Representation of action-specific reward values in the striatum. Science 2005, 310:1337-1340.
56. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
57. Schweighofer N., Doya K. Meta-learning in reinforcement learning. Neural Netw. 2003, 16:5-9.
58. Schweighofer N., Tanaka S.C., Doya K. Serotonin and the evaluation of future rewards: theory, experiments, and possible neural mechanisms. Ann. N. Y. Acad. Sci. 2007, 1104:289-300.
59. Seo H., Lee D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 2007, 27:8366-8377.
60. Seo H., Lee D. Cortical mechanisms for reinforcement learning in competitive games. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2008, 363:3845-3857.
61. Seo H., Lee D. Behavioral and neural changes after gains and losses of conditioned reinforcers. J. Neurosci. 2009, 29(11):3627-3641.
62. Shima K., Tanji J. Role for cingulate motor area cells in voluntary movement selection based on reward. Science 1998, 282:1335-1338.
63. Silton R.L., Heller W., Towers D.N., Engels A.S., Spielberg J.M., Edgar J.C., Sass S.M., Stewart J.L., Sutton B.P., Banich M.T., Miller G.A. The time course of activity in dorsolateral prefrontal cortex and anterior cingulate cortex during top-down attentional control. Neuroimage 2010, 50:1292-1302.
64. Sul J.H., Kim H., Huh N., Lee D., Jung M.W. Distinct roles of rodent orbitofrontal and medial prefrontal cortex in decision making. Neuron 2010, 66:449-460.
65. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
66. Tanaka S.C., Doya K., Okada G., Ueda K., Okamoto Y., Yamawaki S. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 2004, 7:887-893.
67. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46(4):681-692.