Two Anatomically and Computationally Distinct Learning Signals Predict Changes to Stimulus-Outcome Associations in Hippocampus

Neuron

Volumn 89, Issue 6, 2016, Pages 1343-1354

  Boorman, Erie D ^a,b,c   Rajendran, Vani G ^a   O'Reilly, Jill X ^a   Behrens, Tim E ^a,b  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c MAX PLANCK UCL CENTRE FOR COMPUTATIONAL PSYCHIATRY AND AGEING RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; DECISION MAKING; FEEDBACK SYSTEM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HIPPOCAMPUS; HUMAN; HUMAN EXPERIMENT; LATERAL ORBITOFRONTAL CORTEX; LEARNING; MALE; MEDIAL TEMPORAL LOBE; MESENCEPHALON; NORMAL HUMAN; PRIORITY JOURNAL; TASK PERFORMANCE; BIOLOGICAL MODEL; BLOOD; BRAIN MAPPING; COMPUTER SIMULATION; HEMISPHERIC DOMINANCE; NERVE TRACT; PHOTOSTIMULATION; PHYSIOLOGY; PREDICTIVE VALUE; PREFRONTAL CORTEX; STATISTICAL MODEL; VASCULARIZATION; YOUNG ADULT;

OXYGEN;

ADULT; BRAIN MAPPING; CHOICE BEHAVIOR; COMPUTER SIMULATION; FEEDBACK; FEMALE; FUNCTIONAL LATERALITY; HEALTHY VOLUNTEERS; HIPPOCAMPUS; HUMANS; LEARNING; LINEAR MODELS; MALE; MESENCEPHALON; MODELS, BIOLOGICAL; NEURAL PATHWAYS; OXYGEN; PHOTIC STIMULATION; PREDICTIVE VALUE OF TESTS; PREFRONTAL CORTEX; YOUNG ADULT;

EID: 84960810439     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2016.02.014     Document Type: Article

References (65)

1. Barron H.C., Dolan R.J., Behrens T.E. Online evaluation of novel choices by simultaneous representation of multiple memories. Nat. Neurosci. 2013, 16:1492-1498.
2. Behrens T.E., Woolrich M.W., Walton M.E., Rushworth M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
3. Behrens T.E., Hunt L.T., Rushworth M.F. The computation of social behavior. Science 2009, 324:1160-1164.
4. Bird C.M., Burgess N. The hippocampus and memory: insights from spatial processing. Nat. Rev. Neurosci. 2008, 9:182-194.
5. Boorman E.D., Behrens T.E., Woolrich M.W., Rushworth M.F. How green is the grass on the other side Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 2009, 62:733-743.
6. Boorman E.D., O'Doherty J.P., Adolphs R., Rangel A. The behavioral and neural mechanisms underlying the tracking of expertise. Neuron 2013, 80:1558-1571.
7. Boorman E.D., Rushworth M.F., Behrens T.E. Ventromedial prefrontal and anterior cingulate cortex adopt choice and default reference frames during sequential multi-alternative choice. J. Neurosci. 2013, 33:2242-2253.
8. Buckley M.J., Mansouri F.A., Hoda H., Mahboubi M., Browning P.G., Kwok S.C., Phillips A., Tanaka K. Dissociable components of rule-guided behavior depend on distinct medial and prefrontal regions. Science 2009, 325:52-58.
9. Carmichael S.T., Price J.L. Limbic connections of the orbital and medial prefrontal cortex in macaque monkeys. J. Comp. Neurol. 1995, 363:615-641.
10. Chib V.S., Rangel A., Shimojo S., O'Doherty J.P. Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. J. Neurosci. 2009, 29:12315-12320.
11. Clark A.M., Bouret S., Young A.M., Murray E.A., Richmond B.J. Interaction between orbital prefrontal and rhinal cortex is required for normal estimates of expected value. J. Neurosci. 2013, 33:1833-1845.
12. Collins A.G., Frank M.J. Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol. Rev. 2014, 121:337-366.
13. Cooper J.C., Kreps T.A., Wiebe T., Pirkl T., Knutson B. When giving is good: ventromedial prefrontal cortex activation for others' intentions. Neuron 2010, 67:511-521.
14. D'Ardenne K., McClure S.M., Nystrom L.E., Cohen J.D. BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 2008, 319:1264-1267.
15. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
16. den Ouden H.E., Friston K.J., Daw N.D., McIntosh A.R., Stephan K.E. A dual role for prediction error in associative learning. Cereb. Cortex 2009, 19:1175-1185.
17. Gasbarri A., Packard M.G., Campana E., Pacitti C. Anterograde and retrograde tracing of projections from the ventral tegmental area to the hippocampal formation in the rat. Brain Res. Bull. 1994, 33:445-452.
18. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
19. Gremel C.M., Costa R.M. Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions. Nat. Commun. 2013, 4:2264.
20. Grill-Spector K., Henson R., Martin A. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 2006, 10:14-23.
21. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 2006, 26:8360-8367.
22. Hampton A.N., Adolphs R., Tyszka M.J., O'Doherty J.P. Contributions of the amygdala to reward expectancy and choice signals in human prefrontal cortex. Neuron 2007, 55:545-555.
23. Hare T.A., O'Doherty J., Camerer C.F., Schultz W., Rangel A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci. 2008, 28:5623-5630.
24. Howard J.D., Gottfried J.A., Tobler P.N., Kahnt T. Identity-specific coding of future rewards in the human orbitofrontal cortex. Proc. Natl. Acad. Sci. USA 2015, 112:5195-5200.
25. Iglesias S., Mathys C., Brodersen K.H., Kasper L., Piccirelli M., den Ouden H.E., Stephan K.E. Hierarchical prediction errors in midbrain and basal forebrain during sensory learning. Neuron 2013, 80:519-530.
26. Itti L., Baldi P. Bayesian surprise attracts human attention. Vision Res. 2009, 49:1295-1306.
27. Jocham G., Klein T.A., Ullsperger M. Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. J. Neurosci. 2011, 31:1606-1613.
28. Jones J.L., Esber G.R., McDannald M.A., Gruber A.J., Hernandez A., Mirenzi A., Schoenbaum G. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 2012, 338:953-956.
29. Kennerley S.W., Walton M.E., Behrens T.E., Buckley M.J., Rushworth M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 2006, 9:940-947.
30. Klein-Flügge M.C., Hunt L.T., Bach D.R., Dolan R.J., Behrens T.E. Dissociable reward and timing signals in human midbrain and ventral striatum. Neuron 2011, 72:654-664.
31. Klein-Flügge M.C., Barron H.C., Brodersen K.H., Dolan R.J., Behrens T.E. Segregated encoding of reward-identity and stimulus-reward associations in human orbitofrontal cortex. J. Neurosci. 2013, 33:3202-3211.
32. Kondo H., Saleem K.S., Price J.L. Differential connections of the perirhinal and parahippocampal cortex with the orbital and medial prefrontal networks in macaque monkeys. J. Comp. Neurol. 2005, 493:479-509.
33. Kravitz A.V., Tye L.D., Kreitzer A.C. Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 2012, 15:816-818.
34. Kumaran D., Summerfield J.J., Hassabis D., Maguire E.A. Tracking the emergence of conceptual knowledge during human decision making. Neuron 2009, 63:889-901.
35. Lim S.L., O'Doherty J.P., Rangel A. The decision value computations in the vmPFC and striatum use a relative value code that is guided by visual attention. J. Neurosci. 2011, 31:13214-13223.
36. McDannald M.A., Lucantonio F., Burke K.A., Niv Y., Schoenbaum G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 2011, 31:2700-2705.
37. Meyer T., Olson C.R. Statistical learning of visual transitions in monkey inferotemporal cortex. Proc. Natl. Acad. Sci. USA 2011, 108:19401-19406.
38. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16:1936-1947.
39. Morrison S.E., Saez A., Lau B., Salzman C.D. Different time courses for learning-related changes in amygdala and orbitofrontal cortex. Neuron 2011, 71:1127-1140.
40. Noonan M.P., Mars R.B., Rushworth M.F. Distinct roles of three frontal cortical areas in reward-guided behavior. J. Neurosci. 2011, 31:14399-14412.
41. Noonan M.P., Kolling N., Walton M.E., Rushworth M.F. Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement. Eur. J. Neurosci. 2012, 35:997-1010.
42. O'Reilly J.X., Schüffelgen U., Cuell S.F., Behrens T.E., Mars R.B., Rushworth M.F. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proc. Natl. Acad. Sci. USA 2013, 110:E3660-E3669.
43. Payzan-LeNestour E., Bossaerts P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 2011, 7:e1001048.
44. Peters J., Büchel C. Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 2010, 66:138-148.
45. Rudebeck P.H., Murray E.A. The orbitofrontal oracle: cortical mechanisms for the prediction and evaluation of specific behavioral outcomes. Neuron 2014, 84:1143-1156.
46. Rudebeck P.H., Mitz A.R., Chacko R.V., Murray E.A. Effects of amygdala lesions on reward-value coding in orbital and medial prefrontal cortex. Neuron 2013, 80:1519-1531.
47. Rudebeck P.H., Saunders R.C., Prescott A.T., Chau L.S., Murray E.A. Prefrontal mechanisms of behavioral flexibility, emotion regulation and value updating. Nat. Neurosci. 2013, 16:1140-1145.
48. Rushworth M.F., Noonan M.P., Boorman E.D., Walton M.E., Behrens T.E. Frontal cortex and reward-guided learning and decision-making. Neuron 2011, 70:1054-1069.
49. Schwartenbeck P., FitzGerald T.H., Dolan R. Neural signals encoding shifts in beliefs. Neuroimage 2016, 125:578-586.
50. Shohamy D., Wagner A.D. Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron 2008, 60:378-389.
51. Stalnaker T.A., Franz T.M., Singh T., Schoenbaum G. Basolateral amygdala lesions abolish orbitofrontal-dependent reversal impairments. Neuron 2007, 54:51-58.
52. Stalnaker T.A., Cooch N.K., McDannald M.A., Liu T.L., Wied H., Schoenbaum G. Orbitofrontal neurons infer the value and identity of predicted outcomes. Nat. Commun. 2014, 5:3926.
53. Stalnaker T.A., Cooch N.K., Schoenbaum G. What the orbitofrontal cortex does not do. Nat. Neurosci. 2015, 18:620-627.
54. Stefanacci L., Amaral D.G. Some observations on cortical inputs to the macaque monkey amygdala: an anterograde tracing study. J. Comp. Neurol. 2002, 451:301-323.
55. Summerfield C., Egner T. Expectation (and attention) in visual cognition. Trends Cogn. Sci. 2009, 13:403-409.
56. Summerfield C., Trittschuh E.H., Monti J.M., Mesulam M.M., Egner T. Neural repetition suppression reflects fulfilled perceptual expectations. Nat. Neurosci. 2008, 11:1004-1006.
57. Takahashi Y.K., Roesch M.R., Wilson R.C., Toreson K., O'Donnell P., Niv Y., Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci. 2011, 14:1590-1597.
58. Walton M.E., Behrens T.E., Buckley M.J., Rudebeck P.H., Rushworth M.F. Separable learning systems in the macaque brain and the role of orbitofrontal cortex in contingent learning. Neuron 2010, 65:927-939.
59. Wiggs C.L., Martin A. Properties and mechanisms of perceptual priming. Curr. Opin. Neurobiol. 1998, 8:227-233.
60. Wikenheiser A.M., Redish A.D. Hippocampal theta sequences reflect current goals. Nat. Neurosci. 2015, 18:289-294.
61. Wilson R.C., Takahashi Y.K., Schoenbaum G., Niv Y. Orbitofrontal cortex as a cognitive map of task space. Neuron 2014, 81:267-279.
62. Wittmann B.C., Schott B.H., Guderian S., Frey J.U., Heinze H.J., Düzel E. Reward-related FMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent long-term memory formation. Neuron 2005, 45:459-467.
63. Woolrich M. Robust group analysis using outlier inference. Neuroimage 2008, 41:286-301.
64. Woolrich M.W., Ripley B.D., Brady M., Smith S.M. Temporal autocorrelation in univariate linear modeling of FMRI data. Neuroimage 2001, 14:1370-1386.
65. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 15:786-791.