A Neural Signature of Hierarchical Reinforcement Learning

Neuron

Volumn 71, Issue 2, 2011, Pages 370-379

  Ribas Fernandes, José J F ^a,b   Solway, Alec ^a   Diuk, Carlos ^a   McGuire, Joseph T ^c   Barto, Andrew G ^d   Niv, Yael ^a   Botvinick, Matthew M ^a  

^a PRINCETON UNIVERSITY   (United States)

^b CHAMPALIMAUD FOUNDATION   (Portugal)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

^d Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; BEHAVIORAL SCIENCE; BRAIN NERVE CELL; CELL ACTIVITY; CONCEPTUAL FRAMEWORK; CONTROLLED STUDY; DOMINANCE BEHAVIOR; ELECTROENCEPHALOGRAM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; LEARNING; LEARNING ALGORITHM; MACHINE LEARNING; MALE; MATHEMATICAL COMPUTING; NERVE CELL; NEUROIMAGING; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; TASK PERFORMANCE;

ADOLESCENT; ADULT; BRAIN; BRAIN MAPPING; BRAIN WAVES; COMPUTER SIMULATION; ELECTROENCEPHALOGRAPHY; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LINEAR MODELS; MAGNETIC RESONANCE IMAGING; MALE; MODELS, BIOLOGICAL; OXYGEN; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY); YOUNG ADULT;

EID: 79960637995     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2011.05.042     Document Type: Article

References (60)

1. Badre D. Cognitive control, hierarchy, and the rostro-caudal organization of the frontal lobes. Trends Cogn. Sci. (Regul. Ed.) 2008, 12:193-200.
2. Badre D., Frank M.J. Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. Cereb. Cortex 2011, in press. Published online June 21, 2011.
3. Badre D., Hoffman J., Cooney J.W., D'Esposito M. Hierarchical cognitive control deficits following damage to the human frontal lobe. Nat. Neurosci. 2009, 12:515-522.
4. Baker T.E., Holroyd C.B. Dissociated roles of the anterior cingulate cortex in reward and conflict processing as revealed by the feedback error-related negativity and N200. Biol. Psychol. 2011, 87:25-34.
5. Barto A.G. Adaptive critics and the basal ganglia. Models of Information Processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge, MA. J.C. Houk, J. Davis, D. Beiser (Eds.).
6. Barto A.G., Mahadevan S. Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst. 2003, 13:341-379.
7. Botvinick M.M. Hierarchical models of behavior and prefrontal function. Trends Cogn. Sci. (Regul. Ed.) 2008, 12:201-208.
8. Botvinick M., Nystrom L.E., Fissell K., Carter C.S., Cohen J.D. Conflict monitoring versus selection-for-action in anterior cingulate cortex. Nature 1999, 402:179-181.
9. Botvinick M.M., Niv Y., Barto A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 2009, 113:262-280.
10. Brainard D.H. The psychophysics toolbox. Spat. Vis. 1997, 10:433-436.
11. Breiter H.C., Aharon I., Kahneman D., Dale A., Shizgal P. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 2001, 30:619-639.
12. Christoff K., Keramatian K., Gordon A.M., Smith R., Mädler B. Prefrontal organization of cognitive control according to levels of abstraction. Brain Res. 2009, 1286:94-105.
13. Christoph G.R., Leonzio R.J., Wilcox K.S. Stimulation of the lateral habenula inhibits dopamine-containing neurons in the substantia nigra and ventral tegmental area of the rat. J. Neurosci. 1986, 6:613-619.
14. Cooper R., Shallice T. Contention scheduling and the control of routine activities. Cogn. Neuropsychol. 2000, 17:297-338.
15. Cooper J.C., Knutson B. Valence and salience contribute to nucleus accumbens activation. Neuroimage 2008, 39:538-547.
16. Corbetta M., Patel G., Shulman G.L. The reorienting system of the human brain: from environment to theory of mind. Neuron 2008, 58:306-324.
17. Cox R.W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 1996, 29:162-173.
18. Daw N.D., Frank M.J. Reinforcement learning and higher level cognition: introduction to special issue. Cognition 2009, 113:259-261.
19. Dayan P., Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr. Opin. Neurobiol. 2008, 18:185-196.
20. Dietterich T.G. The MAXQ method for hierarchical reinforcement learning. Proceedings of the Fifteenth International Conference on Machine Learning 1998, 118-126. Morgan Kaufman, San Francisco. J.W. Shavlik (Ed.).
21. Grafman J. The human prefrontal cortex has evolved to represent components of structured event complexes. Handbook of Neuropsychology 2002, Elsevier, Amsterdam. J. Grafman (Ed.).
22. Grent-'t-Jong T., Woldorff M.G. Timing and sequence of brain activity in top-down control of visual-spatial attention. PLoS Biol. 2007, 5:e12.
23. Hare T.A., O'Doherty J., Camerer C.F., Schultz W., Rangel A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci. 2008, 28:5623-5630.
24. Haruno M., Kawato M. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw. 2006, 19:1242-1254.
25. Hestvik A., Maxfield N., Schwartz R.G., Shafer V. Brain responses to filled gaps. Brain Lang. 2007, 100:301-316.
26. Holroyd C.B., Coles M.G.H. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 2002, 109:679-709.
27. Holroyd C.B., Nieuwenhuis S., Yeung N., Cohen J.D. Errors in reward prediction are reflected in the event-related brain potential. Neuroreport 2003, 14:2481-2484.
28. Holroyd C.B., Nieuwenhuis S., Yeung N., Nystrom L., Mars R.B., Coles M.G.H., Cohen J.D. Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nat. Neurosci. 2004, 7:497-498.
29. Jocham G., Ullsperger M. Neuropharmacology of performance monitoring. Neurosci. Biobehav. Rev. 2009, 33:48-60.
30. Knutson B., Taylor J., Kaufman M., Peterson R., Glover G. Distributed neural representation of expected value. J. Neurosci. 2005, 25:4806-4812.
31. Kouneiher F., Charron S., Koechlin E. Motivation and cognitive control in the human prefrontal cortex. Nat. Neurosci. 2009, 12:939-945.
32. Krigolson O.E., Holroyd C.B. Evidence for hierarchical error processing in the human brain. Neuroscience 2006, 137:13-17.
33. Lashley K.S. The problem of serial order in behavior. Cerebral Mechanisms in Behavior: The Hixon Symposium 1951, 112-136. Wiley, New York. L.A. Jeffress (Ed.).
34. Matsumoto M., Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 2007, 447:1111-1115.
35. Miltner W.H.R., Braun C.H., Coles M.G.H. Event-related brain potentials following incorrect feedback in a time-estimation task: evidence for a " generic" neural system for error detection. J. Cogn. Neurosci. 1997, 9:788-798.
36. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16:1936-1947.
37. Nicola S.M., Surmeier J., Malenka R.C. Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu. Rev. Neurosci. 2000, 23:185-215.
38. Nieuwenhuis S., Slagter H.A., von Geusau N.J.A., Heslenfeld D.J., Holroyd C.B. Knowing good from bad: differential activation of human cortical areas by positive and negative outcomes. Eur. J. Neurosci. 2005, 21:3161-3168.
39. Niv Y. Reinforcement learning in the brain. J. Math. Psychol. 2009, 53:139-154.
40. O'Doherty J.P., Hampton A., Kim H. Model-based fMRI and its application to reward learning and decision making. Ann. N Y Acad. Sci. 2007, 1104:35-53.
41. O'Doherty J.P., Dayan P., Friston K.J., Critchley H.D., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
42. O'Doherty J.P., Buchanan T.W., Seymour B., Dolan R.J. Predictive neural coding of reward preference involves dissociable responses in human ventral midbrain and ventral striatum. Neuron 2006, 49:157-166.
43. Parr R., Russell S. Reinforcement learning with hierarchies of machines. Adv. Neural Inf. Process Sys. 1998, 10:1043-1049.
44. Phan K.L., Wager T.D., Taylor S.F., Liberzon I. Functional neuroimaging studies of human emotions. CNS Spectr. 2004, 9:258-266.
45. Poldrack R.A. Can cognitive processes be inferred from neuroimaging data?. Trends Cogn. Sci. (Regul. Ed.) 2006, 10:59-63.
46. Reynolds J.R., O'Reilly R.C. Developing PFC representations using reinforcement learning. Cognition 2009, 113:281-292.
47. Salas R., Baldwin P., de Biasi M., Montague P.R. BOLD responses to negative reward prediction errors in human habenula. Front. Hum. Neurosci. 2010, 4:36.
48. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
49. Seymour B., O'Doherty J.P., Dayan P., Koltzenburg M., Jones A.K., Dolan R.J., Friston K.J., Frackowiak R.S. Temporal difference models describe higher-order learning in humans. Nature 2004, 429:664-667.
50. Seymour B., Daw N., Dayan P., Singer T., Dolan R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 2007, 27:4826-4831.
51. Singh S., Barto A.G., Chentanez N. Intrinsically motivated reinforcement learning. Advances in Neural Information Processing Systems 17: Proceedings of the 2004 Conference 2005, 1281-1288. MIT Press, Cambridge, MA. L.K. Saul, Y. Weiss, L. Bottou (Eds.).
52. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
53. Sutton R.S., Precup D., Singh S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 1999, 112:181-211.
54. Talairach J., Tournoux P. Co-Planar Stereotaxic Atlas of the Human Brain 1988, Thieme Medical Publishers, Inc., New York.
55. Ullsperger M., von Cramon D.Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 2003, 23:4308-4314.
56. van Veen V., Holroyd C.B., Cohen J.D., Stenger V.A., Carter C.S. Errors without conflict: implications for performance monitoring theories of anterior cingulate cortex. Brain Cogn. 2004, 56:267-276.
57. Yacubian J., Gläscher J., Schroeder K., Sommer T., Braus D.F., Büchel C. Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. J. Neurosci. 2006, 26:9530-9537.
58. Yamaguchi S., Tsuchiya H., Kobayashi S. Electrophysiologic correlates of visuo-spatial attention shift. Electroencephalogr. Clin. Neurophysiol. 1995, 94:450-461.
59. Yeung N., Botvinick M.M., Cohen J.D. The neural basis of error detection: conflict monitoring and the error-related negativity. Psychol. Rev. 2004, 111:931-959.
60. Yeung N., Holroyd C.B., Cohen J.D. ERP correlates of feedback and reward processing in the presence and absence of response choice. Cereb. Cortex 2005, 15:535-544.