Neural mechanisms of credit assignment in a multicue environment

Journal of Neuroscience

Volumn 36, Issue 4, 2016, Pages 1096-1112

  Akaishi, Rei ^a,b   Kolling, Nils ^a   Brown, Joshua W ^c   Rushworth, Matthew ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b TOKYO METROPOLITAN INSTITUTE OF MEDICAL SCIENCE   (Japan)

^c INDIANA UNIVERSITY   (United States)

Author keywords

Decision making; Learning; Medial prefrontal cortex; Orbitofrontal cortex

Indexed keywords

ADULT; ARTICLE; ASSOCIATION; DECISION MAKING; FEMALE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; MATHEMATICAL MODEL; MEDIAL FRONTAL CORTEX; NONHUMAN; NORMAL HUMAN; ORBITAL CORTEX; PRIORITY JOURNAL; BIOLOGICAL MODEL; BLOOD; ENVIRONMENT; FRONTAL LOBE; IMAGE PROCESSING; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; VASCULARIZATION; YOUNG ADULT;

OXYGEN;

ADULT; CUES; DECISION MAKING; ENVIRONMENT; FEMALE; FRONTAL LOBE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; MODELS, BIOLOGICAL; OXYGEN; YOUNG ADULT;

EID: 84956766765     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.3159-15.2016     Document Type: Article

References (64)

1. Abe H, Lee D (2011) Distributed coding of actual and hypothetical outcomes in the orbital and dorsolateral prefrontal cortex. Neuron 70: 731-741. CrossRef Medline
2. Beckmann CF, Jenkinson M, Smith SM (2003) General multilevel linear modeling for group analysis in FMRI. Neuroimage 20:1052-1063.CrossRef Medline
3. Beckmann M, Johansen-Berg H, Rushworth MF (2009) Connectivity-based parcellation of human cingulate cortex and its relation to functional specialization. J Neurosci 29:1175-1190. CrossRef Medline
4. Behrens TE, Woolrich MW, Walton ME, Rushworth MF (2007) Learning the value of information in an uncertain world. Nat Neurosci 10: 1214-1221. CrossRef Medline
5. Boorman ED, Behrens TE, Woolrich MW, Rushworth MF (2009) How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62:733-743. CrossRef Medline
6. Chau BK, Kolling N, Hunt LT, Walton ME, Rushworth MF (2014) A neural mechanism underlying failure of optimal choice with multiple alternatives. Nat Neurosci 17:463-470. CrossRef Medline
7. Chau BK, Sallet J, Papageorgiou GK, Noonan MP, Bell AH, Walton ME, Rushworth MF (2015) Contrasting roles for orbitofrontal cortex and amygdala in credit assignment and learning: a functional magnetic resonance study in macaques. Neuron 87:1106-1118. Medline
8. Cisek P, Kalaska JF (2010) Neural mechanisms for interacting with a world full of action choices. Annu Rev Neurosci 33:269-298. CrossRef Medline
9. Clarke HF, Walker SC, Dalley JW, Robbins TW, Roberts AC (2007) Cognitive inflexibility after prefrontal serotonin depletion is behaviorally and neurochemically specific. Cereb Cortex 17:18-27. CrossRef Medline
10. Damoiseaux JS, Rombouts SA, Barkhof F, Scheltens P, Stam CJ, Smith SM, Beckmann CF (2006) Consistent resting-state networks across healthy subjects. Proc Natl Acad Sci U S A 103:13848-13853. CrossRef Medline
11. Deichmann R, Gottfried JA, Hutton C, Turner R (2003) Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage 19:430-441.CrossRef Medline
12. Dias R, Robbins TW, Roberts AC (1996) Dissociation in prefrontal cortex of affective and attentional shifts. Nature 380:69-72. CrossRef Medline
13. Dias R, Robbins TW, Roberts AC (1997) Dissociable forms of inhibitory control within prefrontal cortex with an analog of the Wisconsin Card Sort Test: restriction to novel situations and independence from “online” processing. J Neurosci 17:9285-9297. Medline
14. Donoso M, Collins AG, Koechlin E (2014) Foundations of human reasoning in the prefrontal cortex. Science 344:1481-1486. CrossRef Medline
15. Fellows LK (2006) Deciding how to decide: ventromedial frontal lobe damage affects information acquisition in multi-attribute decision making. Brain 129:944-952. CrossRef Medline
16. Gibson JJ (1979) The ecological approach to visual perception. Boston: Houghton, Mifflin and Company.
17. Gould IC, Nobre AC, Wyart V, Rushworth MF (2012) Effects of decision variables and intraparietal stimulation on sensorimotor oscillatory activity in the human brain. J Neurosci 32:13805-13818. CrossRef Medline
18. Hikosaka O, Kim HF, Yasuda M, Yamamoto S (2014) Basal ganglia circuits for reward value-guided behavior. Annu Rev Neurosci 37:289-306.CrossRef Medline
19. Howard JD, Gottfried JA, Tobler PN, Kahnt T (2015) Identity-specific coding of future rewards in the human orbitofrontal cortex. Proc Natl Acad Sci U S A 112:5195-5200. CrossRef Medline
20. Hunt LT, Dolan RJ, Behrens TE (2014) Hierarchical competitions subserving multi-attribute choice. Nat Neurosci 17:1613-1622. CrossRef Medline
21. Jenkinson M, Smith S (2001) A global optimisation method for robust affine registration of brain images. Med Image Anal 5:143-156. CrossRef Medline
22. Jenkinson M, Bannister P, Brady M, Smith S (2002) Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17:825-841. CrossRef Medline
23. Johansen-Berg H, Gutman DA, Behrens TE, Matthews PM, Rushworth MF, Katz E, Lozano AM, Mayberg HS (2008) Anatomical connectivity of the subgenual cingulate region targeted with deep brain stimulation for treatment-resistant depression. Cereb Cortex 18:1374-1383. CrossRef Medline
24. Kira S, Yang T, Shadlen MN (2015) A neural implementation of Wald’s sequential probability ratio test. Neuron 85:861-873. CrossRef Medline
25. Klein-Flügge MC, Barron HC, Brodersen KH, Dolan RJ, Behrens TE (2013) Segregated encoding of reward-identity and stimulus-reward associations in human orbitofrontal cortex. J Neurosci 33:3202-3211. CrossRef Medline
26. Knowlton BJ, Mangels JA, Squire LR (1996) A neostriatal habit learning system in humans. Science 273:1399-1402. CrossRef Medline
27. Konishi S, Nakajima K, Uchida I, Kameyama M, Nakahara K, Sekihara K, Miyashita Y (1998) Transient activation of inferior prefrontal cortex during cognitive set shifting. Nat Neurosci 1:80-84. CrossRef Medline
28. Kringelbach ML, Rolls ET (2004) The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology. Prog Neurobiol 72:341-372. CrossRef Medline
29. Lebreton M, Abitbol R, Daunizeau J, Pessiglione M (2015) Automatic integration of confidence in the brain valuation signal. Nat Neurosci 18: 1159-1167. CrossRef Medline
30. Lee D, Seo H, JungMW (2012) Neural basis of reinforcement learning and decision making. Annu Rev Neurosci 35:287-308. CrossRef Medline
31. Lim SL, O’Doherty JP, Rangel A (2011) The decision value computations in the vmPFC and striatum use a relative value code that is guided by visual attention. J Neurosci 31:13214-13223. CrossRef Medline
32. Mackintosh NJ (1975) A theory of attention: variations in the associability of stimuli with reinforcement. Psychol Rev 82:276. CrossRef
33. Mackintosh NJ (1976) Overshadowing and stimulus intensity. Anim Learn Behav 4:186-192. CrossRef Medline
34. McDannald MA, Lucantonio F, Burke KA, Niv Y, Schoenbaum G (2011) Ventral striatum and orbitofrontal cortex are both required for modelbased, but not model-free, reinforcement learning. J Neurosci 31: 2700-2705. CrossRef Medline
35. McDannald MA, Esber GR, Wegener MA, Wied HM, Liu TL, Stalnaker TA, Jones JL, Trageser J, Schoenbaum G (2014) Orbitofrontal neurons acquire responses to “valueless” Pavlovian cues during unblocking. eLife3:e02653. CrossRef Medline
36. McNamee D, Rangel A, O’Doherty JP (2013) Category-dependent and category-independent goal-value codes in human ventromedial prefrontal cortex. Nat Neurosci 16:479-485. CrossRef Medline
37. Monosov IE, Hikosaka O (2012) Regionally distinct processing of rewards and punishments by the primate ventromedial prefrontal cortex. J Neurosci 32:10318-10330. CrossRef Medline
38. Morrison SE, Salzman CD (2009) The convergence of information about rewarding and aversive stimuli in single neurons. J Neurosci 29: 11471-11483. CrossRef Medline
39. Nassar MR, Wilson RC, Heasly B, Gold JI (2010) An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J Neurosci 30:12366-12378. CrossRef Medline
40. Neubert FX, Mars RB, Sallet J, Rushworth MF (2015) Connectivity reveals relationship of brain areas for reward-guided learning and decision making in human and monkey frontal cortex. Proc Natl Acad Sci U S A 112: E2695-E2704. CrossRef Medline
41. Nickerson RS (1998) Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol 2:175. CrossRef
42. Niv Y, Daniel R, Geana A, Gershman SJ, Leong YC, Radulescu A, Wilson RC (2015) Reinforcement learning in multidimensional environments relies on attention mechanisms. J Neurosci 35:8145-8157. CrossRef Medline
43. Noonan MP, Walton ME, Behrens TE, Sallet J, Buckley MJ, Rushworth MF (2010) Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex. Proc Natl Acad Sci U S A 107: 20547-20552. CrossRef Medline
44. Noonan MP, Mars RB, Rushworth MF (2011) Distinct roles of three frontal cortical areas in reward-guided behavior. J Neurosci 31:14399-14412.CrossRef Medline
45. Noonan MP, Kolling N, Walton ME, Rushworth MF (2012) Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement. Eur J Neurosci 35:997-1010. CrossRef Medline
46. O’Doherty J, Kringelbach ML, Rolls ET, Hornak J, Andrews C (2001) Abstract reward and punishment representations in the human orbitofrontal cortex. Nat Neurosci 4:95-102. CrossRef Medline
47. O’Doherty J, Critchley H, Deichmann R, Dolan RJ (2003) Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices. J Neurosci 23:7931-7939. Medline
48. Otten LJ, Henson RN, Rugg MD (2001) Depth of processing effects on neural correlates of memory encoding Relationship between findings from across-and within-task comparisons. Brain 124:399-412. CrossRef Medline
49. Rich EL, Wallis JD (2014) Functional organization of the orbitofrontal cortex. J Cogn Neurosci 26:1347-1362. CrossRef Medline
50. Rudebeck PH, Saunders RC, Prescott AT, Chau LS, Murray EA (2013) Prefrontal mechanisms of behavioral flexibility, emotion regulation and value updating. Nat Neurosci 16:1140-1145. CrossRef Medline
51. Rushworth MF, Noonan MP, Boorman ED, Walton ME, Behrens TE (2011) Frontal cortex and reward-guided learning and decision-making. Neuron 70:1054-1069. CrossRef Medline
52. Sharot T, Riccardi AM, Raio CM, Phelps EA (2007) Neural mechanisms mediating optimism bias. Nature 450:102-105. CrossRef Medline
53. Smith SM (2002) Fast robust automated brain extraction. HumBrain Mapp 17:143-155. CrossRef Medline
54. Smith SM, Jenkinson M, Woolrich MW, Beckmann CF, Behrens TE, Johansen- Berg H, Bannister PR, De Luca M, Drobnjak I, Flitney DE, Niazy RK, Saunders J, Vickers J, Zhang Y, De Stefano N, Brady JM, Matthews PM (2004) Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23:S208-S219. CrossRef Medline
55. Strait CE, Blanchard TC, Hayden BY (2014) Reward value comparison via mutual inhibition in ventromedial prefrontal cortex. Neuron 82:1357-1366. CrossRef Medline
56. Sutherland NJ, Mackintosh NS (1971) Mechanisms of animal discrimination learning. New York: Academic.
57. Takahashi YK, Roesch MR, Wilson RC, Toreson K, O’Donnell P, Niv Y, Schoenbaum G (2011) Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 14: 1590-1597. CrossRef Medline
58. Vaidya AR, Fellows LK (2015) Ventromedial frontal cortex is critical for guiding attention to reward-predictive visual features in humans. J Neurosci 35:12813-12823 CrossRef Medline
59. Walton ME, Behrens TE, Buckley MJ, Rudebeck PH, Rushworth MF (2010) Separable learning systems in themacaquebrainandthe role of orbitofrontal cortex in contingent learning. Neuron 65:927-939. CrossRef Medline
60. Wilson RC, Niv Y (2012) Inferring relevance in a changing world. Front Hum Neurosci 5:189. CrossRef Medline
61. Wilson RC, Nassar MR, Gold JI (2010) Bayesian online learning of the hazard rate in change-point problems. Neural Comput 22:2452-2476.CrossRef Medline
62. Woolrich MW, Behrens TE, Beckmann CF, Jenkinson M, Smith SM (2004) Multilevel linear modelling for FMRI group analysis using Bayesian inference. Neuroimage 21:1732-1747. CrossRef Medline
63. Worsley KJ, Evans AC, Marrett S, Neelin P (1992) A three-dimensional statistical analysis for CBF activation studies in human brain. J Cereb Blood Flow Metab 12:900-918. CrossRef Medline
64. Yang T, Shadlen MN (2007) Probabilistic reasoning by neurons. Nature 447:1075-1080. CrossRef Medline