Dissociable functions of reward inference in the lateral prefrontal cortex and the striatum

Frontiers in Psychology

Volumn 6, Issue , 2015, Pages

  Tanaka, Shingo ^a   Pan, Xiaochuan ^a,b   Oguchi, Mineki ^a   Taylor, Jessica E ^a   Sakagami, Masamichi ^a  

^a TAMAGAWA UNIVERSITY   (Japan)

^b EAST CHINA UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

lateral prefrontal cortex; model based learning; model free learning; reward inference; striatum

Indexed keywords

EID: 84993981800     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2015.00995     Document Type: Review

References (87)

1. Amemori K. Sawaguchi T. (2006). Rule-dependent shifting of sensorimotor representation in the primate prefrontal cortex. Eur. J. Neurosci. 23, 1895–1909. 10.1111/j.1460-9568.2006.04702.x16623846
2. Antzoulatos E. G. Miller E. K. (2011). Differences between neural activity in prefrontal cortex and striatum during learning of novel abstract categories. Neuron 71, 243–249. 10.1016/j.neuron.2011.05.04021791284
3. Antzoulatos E. G. Miller E. K. (2014). Increases in functional connectivity between prefrontal cortex and striatum during category learning. Neuron 83, 216–225. 10.1016/j.neuron.2014.05.00524930701
4. Asaad W. F. Eskandard E. N. (2011). Encoding of both positive and negative reward prediction errors by neurons of the primate lateral prefrontal cortex and caudate nucleus. J. Neurosci. 31, 17772–17787. 10.1523/JNEUROSCI.3793-11.201122159094
5. Balleine B. W. Dickinson A. (1998). Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–419. 10.1016/S0028-3908(98)00033-19704982
6. Balleine B. W. O’Doherty J. P. (2010). Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35, 48–69. 10.1038/npp.2009.13119776734
7. Behrens T. E. Woolrich M. W. Walton M. E. Rushworth M. F. (2007). Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221. 10.1038/nn195417676057
8. Behrens T. E. J. Hunt L. T. Woolrich M. W. Rushworth M. F. S. (2008). Associative learning of social value. Nature 456, 245–249. 10.1038/nature0753819005555
9. Beierholm U. R. Anen C. Quartz S. Bossaerts P. (2011). Separate encoding of model-based and model-free valuations in the human brain. Neuroimage 58, 955–962. 10.1016/j.neuroimage.2011.06.07121757014
10. Blaisdell A. P. Sawa K. Leising K. J. Waldmann M. R. (2006). Causal reasoning in rats. Science 311, 1020–1022. 10.1126/science.112187216484500
11. Bromberg-Martin E. S. Matsumoto M. Nakahara H. Hikosaka O. (2010). Multiple timescales of memory in lateral habenula and dopamine neurons. Neuron 67, 499–510. 10.1016/j.neuron.2010.06.03120696385
12. Bunge S. A. Kahnm I. Wallis J. D. Miller E. K. Wagner A. D. (2003). Neural circuits subserving the retrieval and maintenance of abstract rules. J. Neurophysiol. 90, 3419–3428. 10.1152/jn.00910.200212867532
13. Cromer J. A. Roy J. E. Miller E. K. (2010). Representation of multiple, independent categories in the primate prefrontal cortex. Neuron 66, 796–807. 10.1016/j.neuron.2010.05.00520547135
14. Daw N. D. Gershman S. J. Seymour B. Dayan P. Dolan R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215. 10.1016/j.neuron.2011.02.02721435563
15. Daw N. D. Niv Y. Dayan P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711. 10.1038/nn156016286932
16. Deserno L. Huys Q. J. Boehme R. Buchert R. Heinze H. J. Grace A. A. et al. (2015). Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc. Natl. Acad. Sci. U.S.A. 112, 1595–1600. 10.1073/pnas.141721911225605941
17. Diekhof E. K. Kaps L. Falkai P. Gruber O. (2012). The role of the human ventral striatum and the medial orbitofrontal cortex in the representation of reward magnitude—an activation likelihood estimation meta-analysis of neuroimaging studies of passive reward expectancy and outcome processing. Neuropsychologia 50, 1252–1266. 10.1016/j.neuropsychologia.2012.02.00722366111
18. Doll B. B. Simon D. A. Daw N. D. (2012). The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 22, 1075–1081. 10.1016/j.conb.2012.08.00322959354
19. Donoso M. Collins A. G. Koechlin E. (2014). Human cognition. Foundations of human reasoning in the prefrontal cortex. Science 344, 1481–1486. 10.1126/science.125225424876345
20. Doya K. (2008). Modulators of decision making. Nat. Neurosci. 11, 410–416. 10.1038/nn207718368048
21. Duncan J. (2001). An adaptive coding model of neural function in prefrontal cortex. Nat. Rev. Neurosci. 2, 820–829. 10.1038/3509757511715058
22. Enomoto K. Matsumoto N. Nakai S. Satoh T. Sato T. K. Ueda Y. et al. (2011). Dopamine neurons learn to encode the long-term value of multiple future rewards. Proc. Natl. Acad. Sci. U.S.A. 108, 15462–15467. 10.1073/pnas.101445710821896766
23. Freedman D. J. Riesenhuber M. Poggio T. Miller E. K. (2001). Categorical representation of visual stimuli in the primate prefrontal cortex. Science 291, 312–316. 10.1126/science.291.5502.31211209083
24. Fu W. T. Anderson J. R. (2008). Solving the credit assignment problem: explicit and implicit learning of action sequences with probabilistic outcomes. Psychol. Res. 72, 321–330. 10.1007/s00426-007-0113-717447083
25. Garrison J. Erdeniz B. Done J. (2013). Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci. Biobehav. Rev. 37, 1297–1310. 10.1016/j.neubiorev.2013.03.02323567522
26. Glascher J. Daw N. Dayan P. O’Doherty J. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595. 10.1016/j.neuron.2010.04.01620510862
27. Haber S. N. Knutson B. (2010). The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35, 4–26. 10.1038/npp.2009.12919812543
28. Han S. Huettel S. A. Dobbins I. G. (2009). Rule-dependent prefrontal cortex activity across episodic and perceptual decisions: an fMRI investigation of the criterial classification account. J. Cogn. Neurosci. 21, 922–937. 10.1162/jocn.2009.2106018578596
29. Hare T. A. O’Doherty J. Camerer C. F. Schultz W. Rangel A. (2008). Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci. 28, 5623–5630. 10.1523/JNEUROSCI.1309-08.200818509023
30. Hayden B. Y. Nair A. C. McCoy A. N. Platt M. L. (2008). Posterior cingulate cortex mediates outcome-contingent allocation of behavior. Neuron 60, 19–25. 10.1016/j.neuron.2008.09.01218940585
31. Hikosaka O. Bromberg-Martin E. Hong S. Matsumoto M. (2008). New insights on the subcortical representation of reward. Curr. Opin. Neurobiol. 18, 203–208. 10.1016/j.conb.2008.07.00218674617
32. Hikosaka O. Isoda M. (2010). Switching from automatic to controlled behavior: cortico-basal ganglia mechanisms. Trends Cogn. Sci. 14, 154–161. 10.1016/j.tics.2010.01.00620181509
33. Histed M. H. Pasupathy A. Miller E. K. (2009). Learning substrates in the primate prefrontal cortex and striatum: sustained activity related to successful actions. Neuron 63, 244–253. 10.1016/j.neuron.2009.06.01919640482
34. Hollerman J. R. Tremblay L. Schultz W. (1998). Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 80, 947–963.9705481
35. Hoshi E. Shima K. Tanji J. (2000). Neuronal activity in the primate prefrontal cortex in the process of motor selection based on two behavioral rules. J. Neurophysiol. 83, 2355–2373.10758139
36. Joel D. Niv Y. Ruppin E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15, 535–547. 10.1016/S0893-6080(02)00047-312371510
37. Johnson-Laird P. N. Byrne R. M. Schaeken W. (1992). Propositional reasoning by model. Psychol. Rev. 99, 418–439. 10.1037/0033-295X.99.3.4181365811
38. Kahnt T. Heinzle J. Park S. Q. Haynes J. D. (2011). Decoding the formation of reward predictions across learning. J. Neurosci. 31, 14624–14630. 10.1523/JNEUROSCI.3412-11.201121994378
39. Killcross S. Coutureau E. (2003). Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 13, 400–408. 10.1093/cercor/13.4.40012631569
40. Knutson B. Cooper J. C. (2005). Functional magnetic resonance imaging of reward prediction. Curr. Opin. Neurol. 18, 411–417. 10.1097/01.wco.0000173463.24758.f616003117
41. Kobayashi S. Kawagoe R. Takikawa Y. Koizumi M. Sakagami M. Hikosaka O. (2007). Functional differences between macaque prefrontal cortex and caudate nucleus during eye movements with and without reward. Exp. Brain Res. 176, 341–355. 10.1007/s00221-006-0622-416902776
42. Lee S. W. Shimojo S. O’Doherty J. P. (2014). Neural computations underlying arbitration between model-based and model-free learning. Neuron 81, 687–699. 10.1016/j.neuron.2013.11.02824507199
43. Levy D. J. Glimcher P. W. (2012). The root of all value: a neural common currency for choice. Curr. Opin. Neurobiol. 22, 1027–1038. 10.1016/j.conb.2012.06.00122766486
44. Maia T. V. (2009). Reinforcement learning, conditioning, and the brain: successes and challenges. Cogn. Affect. Behav. Neurosci. 9, 343–364. 10.3758/CABN.9.4.34319897789
45. McClure S. M. Laibson D. I. Loewenstein G. Cohen J. D. (2004). Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507. 10.1126/science.110090715486304
46. McGonigle B. O. Chalmers M. (1977). Are monkeys logical? Nature 267, 694–696. 10.1038/267694a0406574
47. McNamee D. Liljeholm M. Zika O. O’Doherty J. P. (2015). Characterizing the associative content of brain structures involved in habitual and goal-directed actions in humans: a multivariate FMRI study. J. Neurosci. 35, 3764–3771. 10.1523/JNEUROSCI.4677-14.201525740507
48. Meyers E. M. Freedman D. J. Kreiman G. Miller E. K. Poggio T. (2008). Dynamic population coding of category information in inferior temporal and prefrontal cortex. J. Neurophysiol. 100, 1407–1419. 10.1152/jn.90248.200818562555
49. Miller E. K. (2000). The prefrontal cortex and cognitive control. Nat. Rev. Neurosci. 1, 59–65. 10.1038/3503622811252769
50. Murphy R. A. Mondragón E. Murphy V. A. (2008). Rule learning by rats. Science 319, 1849–1851. 10.1126/science.115156418369151
51. Nakahara H. Itoh H. Kawagoe R. Takikawa Y. Hikosaka O. (2004). Dopamine neurons can represent context-dependent prediction error. Neuron 22, 269–280. 10.1016/S0896-6273(03)00869-914741107
52. O’Doherty J. Dayan P. Schultz J. Deichmann R. Friston K. Dolan R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454. 10.1126/science.109428515087550
53. O’Doherty J. P. Dayan P. Friston K. Critchley H. Dolan R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron 38, 329–337. 10.1016/S0896-6273(03)00169-712718865
54. Padoa-Schioppa C. Assad J. A. (2006). Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226. 10.1038/nature0467616633341
55. Pasupathy A. Miller E. K. (2005). Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 433, 873–876. 10.1038/nature0328715729344
56. Pan X. Fan H. Sawa K. Tsuda I. Tsukada M. Sakagami M. (2014). Reward inference by primate prefrontal and striatal neurons. J. Neurosci. 34, 1380–1396. 10.1523/JNEUROSCI.2263-13.201424453328
57. Pan X. Sakagami M. (2012). Category representation and generalization in the prefrontal cortex. Eur. J. Neurosci. 35, 1083–1091. 10.1111/j.1460-9568.2011.07981.x22487038
58. Pan X. Sawa K. Tsuda I. Tsukada M. Sakagami M. (2008). Reward prediction based on stimulus categorization in primate lateral prefrontal cortex. Nat. Neurosci. 11, 703–712. 10.1038/nn.212818500338
59. Paton J. J. Belova M. A. Morrison S. E. Salzman C. D. (2006). The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439, 865–870. 10.1038/nature0449016482160
60. Rangel A. Camerer C. Montague R. (2008). A framework for studying the neurobiology ofvalue-based decision-making. Nat. Rev. Neurosci. 9, 545–556. 10.1038/nrn235718545266
61. Roesch M. R. Olson C. R. (2003). Impact of expected reward on neuronal activity in prefrontal cortex, frontal and supplementary eye fields and premotor cortex. J. Neurophysiol. 90, 1766–1789. 10.1152/jn.00019.200312801905
62. Rushworth M. Noonan M. Boorman E. Walton M. Behrens T. (2011). Frontal cortex and reward-guided learning and decision-making. Neuron 70, 1054–1069. 10.1016/j.neuron.2011.05.01421689594
63. Rygula R. Walker S. C. Clarke H. F. Robbins T. W. Roberts A. C. (2010). Differential contributions of the primate ventrolateral prefrontal and orbitofrontal cortex to serial reversal learning. J. Neurosci. 30, 14552–14559. 10.1523/JNEUROSCI.2631-10.201020980613
64. Sakagami M. Niki H. (1994). Encoding of behavioral significance of visual stimuli by primate prefrontal neurons: relation to relevant task conditions. Exp. Brain Res. 97, 423–436. 10.1007/BF002415368187854
65. Sakagami M. Pan X. (2007). Functional role of the ventrolateral prefrontal cortex in decision making. Curr. Opin. Neurobiol. 17, 228–233. 10.1016/j.conb.2007.02.00817350248
66. Sakagami M. Tsutsui K. (1999). The hierarchical organization of decision making in the primate prefrontal cortex. Neurosci. Res. 34, 79–89. 10.1016/S0168-0102(99)00038-310498334
67. Sakagami M. Watanabe M. (2007). Integration of cognitive and motivational information in the primate lateral prefrontal cortex. Ann. N. Y. Acad. Sci. 1104, 89–107. 10.1196/annals.1390.01017416923
68. Samejima K. Ueda Y. Doya K. Kimura M. (2005). Representation of action-specific reward values in the striatum. Science 310, 1337–1340. 10.1126/science.111527016311337
69. Schultz W. (2000). Multiple reward signals in the brain. Nat. Rev. Neurosci. 1, 199–207. 10.1038/3504456311257908
70. Schultz W. (2006). Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115. 10.1146/annurev.psych.56.091103.07022916318590
71. Schultz W. Dayan P. Montague P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593–1599. 10.1126/science.275.5306.15939054347
72. Seger C. A. Miller E. K. (2010). Category learning in the brain. Annu. Rev. Neurosci. 33, 203–219. 10.1146/annurev.neuro.051508.13554620572771
73. Shima K. Isoda M. Mushiake H. Tanji J. (2007). Categorization of behavioural sequences in the prefrontal cortex. Nature 45, 315–318. 10.1038/nature0547017183266
74. Sutton R. S. Barto A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
75. Tanaka S. C. Samejima K. Okada G. Ueda K. Okamoto Y. Yamawaki S. et al. (2006). Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw. 19, 1233–1241. 10.1016/j.neunet.2006.05.03916979871
76. Tolman E. C. (1948). Cognitive maps in rats and men. Psychol. Rev. 55, 189–208. 10.1037/h006162618870876
77. Tricomi E. Balleine B. W. O’Doherty J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 29, 2225–2232. 10.1111/j.1460-9568.2009.06796.x19490086
78. Vallentin D. Bongard S. Nieder A. (2012). Numerical rule coding in the prefrontal, premotor, and posterior parietal cortices of macaques. J. Neurosci. 32, 6621–6630. 10.1523/JNEUROSCI.5071-11.201222573684
79. Wallis J. D. Anderson K. C. Miller E. K. (2001). Single neurons in prefrontal cortex encode abstract rules. Nature 411, 953–956. 10.1038/3508208111418860
80. Wallis J. D. Miller E. K. (2003). Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur. J. Neurosci. 18, 2069–2081. 10.1046/j.1460-9568.2003.02922.x14622240
81. Walsh M. M. Anderson J. R. (2014). Navigating complex decision spaces: problems and paradigms in sequential choice. Psychol. Bull. 140, 466–486. 10.1037/a003345523834192
82. Watanabe M. (1996). Reward expectancy in primate prefrontal neurons. Nature 382, 629–632. 10.1038/382629a08757133
83. White I. M. Wise S. P. (1999). Rule-dependent neuronal activity in the prefrontal cortex. Exp. Brain Res. 126, 315–335. 10.1007/s00221005074010382618
84. Wunderlich K. Dayan P. Dolan R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 11, 786–791. 10.1038/nn.306822406551
85. Yamada H. Matsumoto N. Kimura M. (2004). Tonically active neurons in the primate caudate nucleus and putamen differentially encode instructed motivational outcomes of action. J. Neurosci. 24, 3500–3510. 10.1523/JNEUROSCI.0068-04.200415071097
86. Yoshida W. Seymour B. Friston K. J. Dolan R. J. (2010). Neural mechanisms of belief inference during cooperative games. J. Neurosci. 30, 10744–10751. 10.1523/JNEUROSCI.5895-09.201020702705
87. Zentall T. R. Galizio M. Critchfield T. S. (2002). Categorization, concept learning and behavior analysis: an introduction. J. Exp. Anal. Behav. 78, 237–248. 10.1901/jeab.2002.78-23712507002