Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Cognitive, Affective and Behavioral Neuroscience

Volumn 15, Issue 2, 2015, Pages 435-459

  Chase, Henry W ^a   Kumar, Poornima ^b,c   Eickhoff, Simon B ^d,e   Dombrovski, Alexandre Y ^a  

^a UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^b MCLEAN HOSPITAL   (United States)

^c HARVARD MEDICAL SCHOOL   (United States)

^d FORSCHUNGSZENTRUM JÜLICH   (Germany)

^e HEINRICH HEINE UNIVERSITY   (Germany)

Author keywords

Expected value; Meta analysis; Prediction error; Reinforcement learning

Indexed keywords

ANIMAL; BRAIN; BRAIN MAPPING; HUMAN; META ANALYSIS; PHYSIOLOGY; PSYCHOLOGICAL MODEL; REINFORCEMENT; STATISTICAL MODEL; STATISTICS;

ANIMALS; BRAIN; BRAIN MAPPING; HUMANS; LIKELIHOOD FUNCTIONS; MODELS, PSYCHOLOGICAL; REINFORCEMENT (PSYCHOLOGY); STATISTICS AS TOPIC;

EID: 84939939233     PISSN: 15307026     EISSN: None     Source Type: Journal    
DOI: 10.3758/s13415-015-0338-7     Document Type: Article

References (147)

1. Arsenault, J. T., Nelissen, K., Jarraya, B., & Vanduffel, W. (2013). Dopaminergic reward signals selectively decrease fMRI activity in primate visual cortex. Neuron, 77, 1174–1186. doi:10.1016/j.neuron.2013.01.008
2. Balleine, B. W., & Dickinson, A. (1998). Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology, 37, 407–419.
3. Bartra, O., McGuire, J. T., & Kable, J. W. (2013). The valuation system: A coordinate-based meta-analysis of BOLD fMRI experiments examining neural correlates of subjective value. NeuroImage, 76, 412–427. doi:10.1016/j.neuroimage.2013.02.063
4. Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10, 1214–1221. doi:10.1038/nn1954
5. Bellebaum, C., Jokisch, D., Gizewski, E. R., Forsting, M., & Daum, I. (2012). The neural coding of expected and unexpected monetary performance outcomes: Dissociations between active and observational learning. Behavioural Brain Research, 227, 241–251. doi:10.1016/j.bbr.2011.10.042
6. Bennett, C. M., & Miller, M. B. (2010). How reliable are the results from functional magnetic resonance imaging? Annals of the New York Academy of Sciences, 1191, 133–155. doi:10.1111/j.1749-6632.2010.05446.x
7. Bernacer, J., Corlett, P. R., Ramachandra, P., McFarlane, B., Turner, D. C., Clark, L., & Murray, G. K. (2013). Methamphetamine-induced disruption of frontostriatal reward learning signals: Relation to psychotic symptoms. American Journal of Psychiatry, 170, 1326–1334. doi:10.1176/appi.ajp.2013.12070978
8. Bouret, S., & Richmond, B. J. (2010). Ventromedial and orbital prefrontal neurons differentially encode internally and externally driven motivational values in monkeys. Journal of Neuroscience, 30, 8591–8601. doi:10.1523/JNEUROSCI. 0049-10.2010
9. Bray, S., & O’Doherty, J. (2007). Neural coding of reward-prediction error signals during classical conditioning with attractive faces. Journal of Neurophysiology, 97, 3036–3045. doi:10.1152/jn.01211.2006
10. Brovelli, A., Laksiri, N., Nazarian, B., Meunier, M., & Boussaoud, D. (2008). Understanding the neural computations of arbitrary visuomotor learning through fMRI and associative learning theory. Cerebral Cortex, 18, 1485–1495. doi:10.1093/cercor/bhm198
11. Bush, R. R., & Mosteller, F. (1951). A model for stimulus generalization and discrimination. Psychological Review, 58, 413–423. doi:10.1037/H0054576
12. Bush, R. R., & Mosteller, F. (1953). A stochastic model with applications to learning. Annals of Mathematical Statistics, 24, 559–585. doi:10.1214/aoms/1177728914
13. Carmichael, S. T., & Price, J. L. (1996). Connectional networks within the orbital and medial prefrontal cortex of macaque monkeys. Journal of Comparative Neurology, 371, 179–207. doi:10.1002/(SICI)1096-9861(19960722)371:2<179::AID-CNE1>3.0.CO;2-#
14. Chiu, P. H., Lohrenz, T. M., & Montague, P. R. (2008). Smokers’ brains compute, but ignore, a fictive error signal in a sequential investment task. Nature Neuroscience, 11, 514–520. doi:10.1038/nn2067
15. Chowdhury, R., Guitart-Masip, M., Lambert, C., Dayan, P., Huys, Q., Duzel, E., & Dolan, R. J. (2013). Dopamine restores reward prediction errors in old age. Nature Neuroscience, 16, 648–653. doi:10.1038/nn.3364
16. Clithero, J. A., & Rangel, A. (2014). Informatic parcellation of the network involved in the computation of subjective value. Social Cognitive and Affective Neuroscience, 9, 1289–1302. doi:10.1093/scan/nst106
17. Cohen, M. X. (2007). Individual differences and the neural representations of reward expectation and reward prediction error. Social Cognitive and Affective Neuroscience, 2, 20–30. doi:10.1093/scan/nsl021
18. Collins, A. G., & Frank, M. J. (2012). How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. European Journal of Neuroscience, 35, 1024–1035. doi:10.1111/j.1460-9568.2011.07980.x
19. Corbit, L. H., & Balleine, B. W. (2011). The general and outcome-specific forms of Pavlovian–instrumental transfer are differentially mediated by the nucleus accumbens core and shell. Journal of Neuroscience, 31, 11786–11794. doi:10.1523/JNEUROSCI. 2711-11.2011
20. Coricelli, G., Critchley, H. D., Joffily, M., O’Doherty, J. P., Sirigu, A., & Dolan, R. J. (2005). Regret and its avoidance: A neuroimaging study of choice behavior. Nature Neuroscience, 8, 1255–1262. doi:10.1038/nn1514
21. Critchley, H. D., & Rolls, E. T. (1996). Hunger and satiety modify the responses of olfactory and visual neurons in the primate orbitofrontal cortex. Journal of Neurophysiology, 75, 1673–1686.
22. D’Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008). BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science, 319, 1264–1267.
23. Daw, N. D. (2011). Trial-by-trial data analysis using computational models. In M. R. Delgado, E. A. Phelps, & T. W. Robbins (Eds.), Decision making, affect, and learning: Attention and performance XXIII (pp. 3–38). Oxford, UK: Oxford University Press.
24. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876–879. doi:10.1038/nature04766
25. Dayan, P., & Walton, M. E. (2012). A step-by-step guide to dopamine. Biological Psychiatry, 71, 842–843. doi:10.1016/j.biopsych.2012.03.008
26. Diekhof, E. K., Kaps, L., Falkai, P., & Gruber, O. (2012). The role of the human ventral striatum and the medial orbitofrontal cortex in the representation of reward magnitude—An activation likelihood estimation meta-analysis of neuroimaging studies of passive reward expectancy and outcome processing. Neuropsychologia, 50, 1252–1266. doi:10.1016/j.neuropsychologia.2012.02.007
27. Dombrovski, A. Y., Szanto, K., Clark, L., Reynolds, C. F., III, & Siegle, G. J. (2013). Reward signals, attempted suicide, and impulsivity in late-life depression. JAMA Psychiatry, 70, 1020–1030. doi:10.1001/jamapsychiatry.2013.75
28. Dosenbach, N. U., Visscher, K. M., Palmer, E. D., Miezin, F. M., Wenger, K. K., Kang, H. C., & Petersen, S. E. (2006). A core system for the implementation of task sets. Neuron, 50, 799–812. doi:10.1016/j.neuron.2006.04.031
29. Eickhoff, S. B., Bzdok, D., Laird, A. R., Kurth, F., & Fox, P. T. (2012). Activation likelihood estimation meta-analysis revisited. NeuroImage, 59, 2349–2361. doi:10.1016/j.neuroimage.2011.09.017
30. Eickhoff, S. B., Bzdok, D., Laird, A. R., Roski, C., Caspers, S., Zilles, K., & Fox, P. T. (2011). Co-activation patterns distinguish cortical modules, their connectivity and functional differentiation. NeuroImage, 57, 938–949. doi:10.1016/j.neuroimage.2011.05.021
31. Eickhoff, S. B., Laird, A. R., Grefkes, C., Wang, L. E., Zilles, K., & Fox, P. T. (2009). Coordinate-based activation likelihood estimation meta-analysis of neuroimaging data: A random-effects approach based on empirical estimates of spatial uncertainty. Human Brain Mapping, 30, 2907–2926. doi:10.1002/hbm.20718
32. Erdeniz, B., Rohe, T., Done, J., & Seidler, R. D. (2013). A simple solution for model comparison in bold imaging: The special case of reward prediction error and reward outcomes. Frontiers in Neuroscience, 7, 116. doi:10.3389/fnins.2013.00116
33. Estes, W. K., & Maddox, W. T. (2005). Risks of drawing inferences about cognitive processes from model fits to individual versus average performance. Psychonomic Bulletin & Review, 12, 403–408.
34. Fareri, D. S., Chang, L. J., & Delgado, M. R. (2012). Effects of direct social experience on trust decisions and neural reward circuitry. Frontiers in Neuroscience, 6, 148. doi:10.3389/fnins.2012.00148
35. Fellows, L. K. (2011). Orbitofrontal contributions to value-based decision making: Evidence from humans with frontal lobe damage. Annals of the New York Academy of Sciences, 1239, 51–58. doi:10.1111/j.1749-6632.2011.06229.x
36. FitzGerald, T. H., Friston, K. J., & Dolan, R. J. (2012). Action-specific value signals in reward-related regions of the human brain. Journal of Neuroscience, 32, 16417–16423. doi:10.1523/JNEUROSCI. 3254-12.2012
37. Frank, G. K., Reynolds, J. R., Shott, M. E., & O’Reilly, R. C. (2011). Altered temporal difference learning in bulimia nervosa. Biological Psychiatry, 70, 728–735. doi:10.1016/j.biopsych.2011.05.011
38. Frank, M. J. (2005). Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience, 17, 51–72. doi:10.1162/0898929052880093
39. Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T., & Hutchison, K. E. (2007). Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proceedings of the National Academy of Sciences, 104, 16311–16316.
40. Gallagher, M., McMahan, R. W., & Schoenbaum, G. (1999). Orbitofrontal cortex and representation of incentive value in associative learning. Journal of Neuroscience, 19, 6610–6614.
41. Gamez, D. (2012). From Baconian to Popperian neuroscience. Neural Systems and Circuits, 2, 2. doi:10.1186/2042-1001-2-2
42. Garrison, J., Erdeniz, B., & Done, J. (2013). Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies. Neuroscience & Biobehavioral Reviews, 37, 1297–1310. doi:10.1016/j.neubiorev.2013.03.023
43. Gershman, S. J., Pesaran, B., & Daw, N. D. (2009). Human reinforcement learning subdivides structured action spaces by learning effector-specific values. Journal of Neuroscience, 29, 13524–13531. doi:10.1523/JNEUROSCI. 2469-09.2009
44. Glascher, J., & Buchel, C. (2005). Formal learning theory dissociates brain regions with different temporal integration. Neuron, 47, 295–306. doi:10.1016/j.neuron.2005.06.008
45. Glascher, J., Hampton, A. N., & O’Doherty, J. P. (2009). Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. Cerebral Cortex, 19, 483–495.
46. Grabenhorst, F., & Rolls, E. T. (2011). Value, pleasure and choice in the ventral prefrontal cortex. Trends in Cognitive Sciences, 15, 56–67. doi:10.1016/j.tics.2010.12.004
47. Gradin, V. B., Kumar, P., Waiter, G., Ahearn, T., Stickle, C., Milders, M., & Steele, J. D. (2011). Expected value and prediction error abnormalities in depression and schizophrenia. Brain, 134, 1751–1764. doi:10.1093/brain/awr059
48. Graham, J., Salimi-Khorshidi, G., Hagan, C., Walsh, N., Goodyer, I., Lennox, B., & Suckling, J. (2013). Meta-analytic evidence for neuroimaging models of depression: State or trait? Journal of Affective Disorders, 151, 423–431. doi:10.1016/j.jad.2013.07.002
49. Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20, 2369–2382.
50. Haber, S. N., & Knutson, B. (2010). The reward circuit: Linking primate anatomy and human imaging. Neuropsychopharmacology, 35, 4–26. doi:10.1038/npp.2009.129
51. Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006). The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience, 26, 8360–8367. doi:10.1523/JNEUROSCI. 1010-06.2006
52. Hayasaka, S., & Nichols, T. E. (2003). Validating cluster size inference: Random field and permutation methods. NeuroImage, 20, 2343–2356.
53. Hertwig, R., & Erev, I. (2009). The description-experience gap in risky choice. Trends in Cognitive Sciences, 13, 517–523. doi:10.1016/j.tics.2009.09.004
54. Holroyd, C. B., & Coles, M. G. (2008). Dorsal anterior cingulate cortex integrates reinforcement history to guide voluntary behavior. Cortex, 44, 548–559. doi:10.1016/j.cortex.2007.08.013
55. Howard-Jones, P. A., Bogacz, R., Yoo, J. H., Leonards, U., & Demetriou, S. (2010). The neural mechanisms of learning from competitors. NeuroImage, 53, 790–799. doi:10.1016/j.neuroimage.2010.06.027
56. Izquierdo, A., Suda, R. K., & Murray, E. A. (2004). Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. Journal of Neuroscience, 24, 7540–7548. doi:10.1523/JNEUROSCI. 1921-04.2004
57. Jocham, G., Klein, T. A., & Ullsperger, M. (2011). Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. Journal of Neuroscience, 31, 1606–1613. doi:10.1523/JNEUROSCI. 3904-10.2011
58. Jones, R. M., Somerville, L. H., Li, J., Ruberry, E. J., Libby, V., Glover, G., & Casey, B. J. (2011). Behavioral and neural properties of social reinforcement learning. Journal of Neuroscience, 31, 13039–13045. doi:10.1523/JNEUROSCI. 2972-11.2011
59. Kahnt, T., Park, S. Q., Cohen, M. X., Beck, A., Heinz, A., & Wrase, J. (2009). Dorsal striatal–midbrain connectivity in humans predicts how reinforcements are used to guide decisions. Journal of Cognitive Neuroscience, 21, 1332–1345. doi:10.1162/jocn.2009.21092
60. Kamin, L. J. (1968). Predictability, surprise, attention, and conditioning. In B. A. Campbell & R. M. Church (Eds.), Punishment and aversive behavior (pp. 279–296). New York, NY: Appleton-Century-Crofts.
61. Kennerley, S. W., Dahmubed, A. F., Lara, A. H., & Wallis, J. D. (2009). Neurons in the frontal lobe encode the value of multiple decision variables. Journal of Cognitive Neuroscience, 21, 1162–1178. doi:10.1162/jocn.2009.21100
62. Kennerley, S. W., & Wallis, J. D. (2009a). Encoding of reward and space during a working memory task in the orbitofrontal cortex and anterior cingulate sulcus. Journal of Neurophysiology, 102, 3352–3364. doi:10.1152/jn.00273.2009
63. Kennerley, S. W., & Wallis, J. D. (2009b). Evaluating choices by single neurons in the frontal lobe: Outcome value encoded across multiple decision variables. European Journal of Neuroscience, 29, 2061–2073. doi:10.1111/j.1460-9568.2009.06743.x
64. Kim, H., Shimojo, S., & O’Doherty, J. P. (2006). Is avoiding an aversive outcome rewarding? Neural substrates of avoidance learning in the human brain. PLoS Biology, 4, e233. doi:10.1371/journal.pbio.0040233
65. Klein, T. A., Neumann, J., Reuter, M., Hennig, J., von Cramon, D. Y., & Ullsperger, M. (2007). Genetically determined differences in learning from errors. Science, 318, 1642–1645. doi:10.1126/science.1145044
66. Kobayashi, S., Pinto de Carvalho, O., & Schultz, W. (2010). Adaptation of reward sensitivity in orbitofrontal neurons. Journal of Neuroscience, 30, 534–544. doi:10.1523/JNEUROSCI. 4009-09.2010
67. Krigolson, O. E., Hassall, C. D., & Handy, T. C. (2014). How we learn to make decisions: Rapid propagation of reinforcement learning prediction errors in humans. Journal of Cognitive Neuroscience, 26, 635–644. doi:10.1162/jocn_a_00509
68. Kumar, P., Waiter, G., Ahearn, T., Milders, M., Reid, I., & Steele, J. D. (2008). Abnormal temporal difference reward-learning signals in major depression. Brain, 131, 2084–2093.
69. Lea, S. (1978). The psychology and economics of demand. Psychological Bulletin, 85, 441–466. doi:10.1037/0033-2909.85.3.441
70. Leathers, M. L., & Olson, C. R. (2012). In monkeys making value-based decisions, LIP neurons encode cue salience and not action value. Science, 338, 132–135. doi:10.1126/science.1226405
71. Levy, D. J., & Glimcher, P. W. (2012). The root of all value: A neural common currency for choice. Current Opinion in Neurobiology, 22, 1027–1038. doi:10.1016/j.conb.2012.06.001
72. Li, J., McClure, S. M., King-Casas, B., & Montague, P. R. (2006). Policy adjustment in a dynamic economic game. PLoS ONE, 1, e103. doi:10.1371/journal.pone.0000103
73. Li, J., Schiller, D., Schoenbaum, G., Phelps, E. A., & Daw, N. D. (2011). Differential roles of human striatum and amygdala in associative learning. Nature Neuroscience, 14, 1250–1252. doi:10.1038/nn.2904
74. Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011). Common and distinct networks underlying reward valence and processing stages: A meta-analysis of functional neuroimaging studies. Neuroscience & Biobehavioral Reviews, 35, 1219–1236. doi:10.1016/j.neubiorev.2010.12.012
75. Logothetis, N. K., & Pfeuffer, J. (2004). On the nature of the BOLD fMRI contrast mechanism. Magnetic Resonance Imaging, 22, 1517–1531. doi:10.1016/j.mri.2004.10.018
76. Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008). Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Computation, 20, 3034–3054. doi:10.1162/neco.2008.11-07-654
77. Maddux, J. M., Kerfoot, E. C., Chatterjee, S., & Holland, P. C. (2007). Dissociation of attention in learning and action: Effects of lesions of the amygdala central nucleus, medial prefrontal cortex, and posterior parietal cortex. Behavioral Neuroscience, 121, 63–79. doi:10.1037/0735-7044.121.1.63
78. Madlon-Kay, S., Pesaran, B., & Daw, N. D. (2013). Action selection in multi-effector decision making. NeuroImage, 70, 66–79. doi:10.1016/j.neuroimage.2012.12.001
79. Mathys, C., Daunizeau, J., Friston, K. J., & Stephan, K. E. (2011). A Bayesian foundation for individual learning under uncertainty. Frontiers in Human Neuroscience, 5, 39. doi:10.3389/fnhum.2011.00039
80. McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. (2011). Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. Journal of Neuroscience, 31, 2700–2705. doi:10.1523/jneurosci. 5499-10.2011
81. Metereau, E., & Dreher, J. C. (2013). Cerebral correlates of salient prediction error for different rewards and punishments. Cerebral Cortex, 23, 477–487. doi:10.1093/cercor/bhs037
82. Miller, R. R., Barnet, R. C., & Grahame, N. J. (1995). Assessment of the Rescorla–Wagner model. Psychological Bulletin, 117, 363–386.
83. Morrison, S. E., & Salzman, C. D. (2009). The convergence of information about rewarding and aversive stimuli in single neurons. Journal of Neuroscience, 29, 11471–11483. doi:10.1523/Jneurosci. 1815-09.2009
84. Murray, G. K., Corlett, P. R., Clark, L., Pessiglione, M., Blackwell, A. D., Honey, G., & Fletcher, P. C. (2008). Substantia nigra/ventral tegmental reward prediction error disruption in psychosis. Molecular Psychiatry, 13(239), 267–276. doi:10.1038/sj.mp.4002058
85. Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190–204. doi:10.1006/jmps.1999.1283
86. Niv, Y., Edlund, J. A., Dayan, P., & O’Doherty, J. P. (2012). Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. Journal of Neuroscience, 32, 551–562. doi:10.1523/JNEUROSCI. 5498-10.2012
87. Noonan, M. P., Walton, M. E., Behrens, T. E., Sallet, J., Buckley, M. J., & Rushworth, M. F. (2010). Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex. Proceedings of the National Academy of Sciences, 107, 20547–20552. doi:10.1073/pnas.1012246107
88. O’Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452–454. doi:10.1126/science.1094285
89. O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 38, 329–337.
90. O’Sullivan, N., Szczepanowski, R., El-Deredy, W., Mason, L., & Bentall, R. P. (2011). fMRI evidence of a relationship between hypomania and both increased goal-sensitivity and positive outcome-expectancy bias. Neuropsychologia, 49, 2825–2835. doi:10.1016/j.neuropsychologia.2011.06.008
91. Ongur, D., & Price, J. L. (2000). The organization of networks within the orbital and medial prefrontal cortex of rats, monkeys and humans. Cerebral Cortex, 10, 206–219.
92. Padoa-Schioppa, C., & Assad, J. A. (2006). Neurons in the orbitofrontal cortex encode economic value. Nature, 441, 223–226.
93. Padoa-Schioppa, C., & Assad, J. A. (2008). The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nature Neuroscience, 11, 95–102. doi:10.1038/nn2020
94. Park, S. Q., Kahnt, T., Beck, A., Cohen, M. X., Dolan, R. J., Wrase, J., & Heinz, A. (2010). Prefrontal cortex fails to learn from reward prediction errors in alcohol dependence. Journal of Neuroscience, 30, 7749–7753. doi:10.1523/JNEUROSCI. 5587-09.2010
95. Parkinson, J. A., Olmstead, M. C., Burns, L. H., Robbins, T. W., & Everitt, B. J. (1999). Dissociation in effects of lesions of the nucleus accumbens core and shell on appetitive Pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by D-amphetamine. Journal of Neuroscience, 19, 2401–2411.
96. Paxinos, G., & Huang, X.-F. (1995). Atlas of the human brain stem. San Diego, CA: Academic Press.
97. Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review, 87, 532–552. doi:10.1037/0033-295X.87.6.532
98. Peters, J., & Buchel, C. (2010). Neural representations of subjective reward value. Behavioural Brain Research, 213, 135–141. doi:10.1016/j.bbr.2010.04.031
99. Petrides, M., & Pandya, D. (1994). Comparative architectonic analysis of the human and the macaque frontal cortex. In F. Boller & J. Grafman (Eds.), Handbook of neuropsychology (Vol. 9, pp. 17–58). Amsterdam, The Netherlands: Elsevier.
100. Platt, M. L., & Glimcher, P. W. (1999). Neural correlates of decision variables in parietal cortex. Nature, 400, 233–238. doi:10.1038/22268
101. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). New York, NY: Appleton-Century-Crofts.
102. Robinson, O. J., Overstreet, C., Charney, D. R., Vytal, K., & Grillon, C. (2013). Stress increases aversive prediction error signal in the ventral striatum. Proceedings of the National Academy of Sciences, 110, 4129–4133. doi:10.1073/pnas.1213923110
103. Rodriguez, P. F. (2009). Stimulus-outcome learnability differentially activates anterior cingulate and hippocampus at feedback processing. Learning and Memory, 16, 324–331. doi:10.1101/lm.1191609
104. Rodriguez, P. F., Aron, A. R., & Poldrack, R. A. (2006). Ventral-striatal/nucleus-accumbens sensitivity to prediction errors during classification learning. Human Brain Mapping, 27, 306–313. doi:10.1002/hbm.20186
105. Roesch, M. R., Calu, D. J., Esber, G. R., & Schoenbaum, G. (2010). All that glitters.. dissociating attention and outcome expectancy from prediction errors signals. Journal of Neurophysiology, 104, 587–595. doi:10.1152/jn.00173.2010
106. Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10, 1615–1624. doi:10.1038/nn2013
107. Roesch, M. R., & Olson, C. R. (2004). Neuronal activity related to reward value and motivation in primate frontal cortex. Science, 304, 307–310. doi:10.1126/science.1093223
108. Roesch, M. R., & Olson, C. R. (2005). Neuronal activity in primate orbitofrontal cortex reflects the value of time. Journal of Neurophysiology, 94, 2457–2471. doi:10.1152/jn.00373.2005
109. Rohe, T., Weber, B., & Fliessbach, K. (2012). Dissociation of BOLD responses to reward prediction errors and reward receipt by a model comparison. European Journal of Neuroscience, 36, 2376–2382. doi:10.1111/j.1460-9568.2012.08125.x
110. Rottschy, C., Langner, R., Dogan, I., Reetz, K., Laird, A. R., Schulz, J. B., & Eickhoff, S. B. (2012). Modelling neural correlates of working memory: A coordinate-based meta-analysis. NeuroImage, 60, 830–846. doi:10.1016/j.neuroimage.2011.11.050
111. Rudebeck, P. H., Behrens, T. E., Kennerley, S. W., Baxter, M. G., Buckley, M. J., Walton, M. E., & Rushworth, M. F. S. (2008). Frontal cortex subregions play distinct roles in choices between actions and stimuli. Journal of Neuroscience, 28, 13775–13785. doi:10.1523/jneurosci. 3541-08.2008
112. Rudebeck, P. H., Buckley, M. J., Walton, M. E., & Rushworth, M. F. (2006). A role for the macaque anterior cingulate gyrus in social valuation. Science, 313, 1310–1312. doi:10.1126/science.1128197
113. Rudebeck, P. H., & Murray, E. A. (2011). Dissociable effects of subtotal lesions within the macaque orbital prefrontal cortex on reward-guided behavior. Journal of Neuroscience, 31, 10569–10578. doi:10.1523/Jneurosci. 0091-11.2011
114. Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010). Testing the reward prediction error hypothesis with an axiomatic model. Journal of Neuroscience, 30, 13525–13536. doi:10.1523/JNEUROSCI. 1747-10.2010
115. Sacchet, M. D., & Knutson, B. (2013). Spatial smoothing systematically biases the localization of reward-related brain activity. NeuroImage, 66, 270–277. doi:10.1016/j.neuroimage.2012.10.056
116. Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science, 310, 1337–1340.
117. Schlagenhauf, F., Rapp, M. A., Huys, Q. J., Beck, A., Wustenberg, T., Deserno, L., & Heinz, A. (2012). Ventral striatal prediction error signaling is associated with dopamine synthesis capacity and fluid intelligence. Human Brain Mapping. doi:10.1002/hbm.22000
118. Schoenbaum, G., Takahashi, Y., Liu, T. L., & McDannald, M. A. (2011). Does the orbitofrontal cortex signal value? Critical Contributions of the Orbitofrontal Cortex to Behavior, 1239, 87–99. doi:10.1111/j.1749-6632.2011.06210.x
119. Schonberg, T., Daw, N. D., Joel, D., & O’Doherty, J. P. (2007). Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. Journal of Neuroscience, 27, 12860–12867. doi:10.1523/JNEUROSCI. 2496-07.2007
120. Schonberg, T., O’Doherty, J. P., Joel, D., Inzelberg, R., Segev, Y., & Daw, N. D. (2010). Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson’s disease patients: Evidence from a model-based fMRI study. NeuroImage, 49, 772–781. doi:10.1016/j.neuroimage.2009.08.011
121. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599. doi:10.1126/science.275.5306.1593
122. Seger, C. A., Peterson, E. J., Cincotta, C. M., Lopez-Paniagua, D., & Anderson, C. W. (2010). Dissociating the contributions of independent corticostriatal systems to visual categorization learning through the use of reinforcement learning modeling and Granger causality modeling. NeuroImage, 50, 644–656. doi:10.1016/j.neuroimage.2009.11.083
123. Sescousse, G., Caldu, X., Segura, B., & Dreher, J. C. (2013). Processing of primary and secondary rewards: A quantitative meta-analysis and review of human functional neuroimaging studies. Neuroscience & Biobehavioral Reviews, 37, 681–696. doi:10.1016/j.neubiorev.2013.02.002
124. Seymour, B., O’Doherty, J. P., Koltzenburg, M., Wiech, K., Frackowiak, R., Friston, K., & Dolan, R. (2005). Opponent appetitive-aversive neural processes underlie predictive learning of pain relief. Nature Neuroscience, 8, 1234–1240. doi:10.1038/nn1527
125. Simmons, J. M., Ravel, S., Shidara, M., & Richmond, B. J. (2007). A comparison of reward-contingent neuronal activity in monkey orbitofrontal cortex and ventral striatum: Guiding actions toward rewards. Annals of the New York Academy of Sciences, 1121, 376–394. doi:10.1196/annals.1401.028
126. Strait, C. E., Blanchard, T. C., & Hayden, B. Y. (2014). Reward value comparison via mutual inhibition in ventromedial prefrontal cortex. Neuron, 82, 1357–1366. doi:10.1016/j.neuron.2014.04.032
127. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, UK: Cambridge Univ Press.
128. Takahashi, Y. K., Roesch, M. R., Stalnaker, T. A., Haney, R. Z., Calu, D. J., Taylor, A. R., & Schoenbaum, G. (2009). The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron, 62, 269–280. doi:10.1016/j.neuron.2009.03.005
129. Takemura, H., Samejima, K., Vogels, R., Sakagami, M., & Okuda, J. (2011). Stimulus-dependent adjustment of reward prediction error in the midbrain. PLoS One, 6, e28337. doi:10.1371/journal.pone.0028337
130. Tanaka, S. C., Samejima, K., Okada, G., Ueda, K., Okamoto, Y., Yamawaki, S., & Doya, K. (2006). Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Networks, 19, 1233–1241. doi:10.1016/j.neunet.2006.05.039
131. Tobler, P. N., Dickinson, A., & Schultz, W. (2003). Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. Journal of Neuroscience, 23, 10402–10410.
132. Tobler, P. N., O’Doherty, J. P., Dolan, R. J., & Schultz, W. (2006). Human neural learning depends on reward prediction errors in the blocking paradigm. Journal of Neurophysiology, 95, 301–310. doi:10.1152/jn.00762.2005
133. Tremblay, L., & Schultz, W. (1999). Relative reward preference in primate orbitofrontal cortex. Nature, 398, 704–708. doi:10.1038/19525
134. Turkeltaub, P. E., Eden, G. F., Jones, K. M., & Zeffiro, T. A. (2002). Meta-analysis of the functional neuroanatomy of single-word reading: Method and validation. NeuroImage, 16, 765–780.
135. Turkeltaub, P. E., Eickhoff, S. B., Laird, A. R., Fox, M., Wiener, M., & Fox, P. (2012). Minimizing within-experiment and within-group effects in activation likelihood estimation meta-analyses. Human Brain Mapping, 33, 1–13. doi:10.1002/hbm.21186
136. Valentin, V. V., & O’Doherty, J. P. (2009). Overlapping prediction errors in dorsal striatum during instrumental learning with juice and money reward in the human brain. Journal of Neurophysiology, 102, 3384–3391. doi:10.1152/jn.91195.2008
137. van den Bos, W., Cohen, M. X., Kahnt, T., & Crone, E. A. (2012). Striatum-medial prefrontal cortex connectivity predicts developmental changes in reinforcement learning. Cerebral Cortex, 22, 1247–1255. doi:10.1093/cercor/bhr198
138. Voorn, P., Vanderschuren, L. J., Groenewegen, H. J., Robbins, T. W., & Pennartz, C. M. (2004). Putting a spin on the dorsal-ventral divide of the striatum. Trends in Neurosciences, 27, 468–474. doi:10.1016/j.tins.2004.06.006
139. Waelti, P., Dickinson, A., & Schultz, W. (2001). Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412, 43–48.
140. Wallis, J. D. (2012). Cross-species studies of orbitofrontal cortex and value-based decision-making. Nature Neuroscience, 15, 13–19. doi:10.1038/nn.2956
141. Wallis, J. D., & Miller, E. K. (2003). Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. European Journal of Neuroscience, 18, 2069–2081. doi:10.1046/j.1460-9568.2003.02922.x
142. Watanabe, N., Sakagami, M., & Haruno, M. (2013). Reward prediction error signal enhanced by striatum-amygdala interaction explains the acceleration of probabilistic reward learning by emotion. Journal of Neuroscience, 33, 4487–4493. doi:10.1523/JNEUROSCI. 3400-12.2013
143. Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008). Striatal activity underlies novelty-based choice in humans. Neuron, 58, 967–973. doi:10.1016/j.neuron.2008.04.027
144. Wunderlich, K., Rangel, A., & O’Doherty, J. P. (2010). Economic choices can be made using only stimulus values. Proceedings of the National Academy of Sciences, 107, 15005–15010. doi:10.1073/pnas.1002258107
145. Yeung, N., Holroyd, C. B., & Cohen, J. D. (2005). ERP correlates of feedback and reward processing in the presence and absence of response choice. Cerebral Cortex, 15, 535–544.
146. Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005). The role of the dorsomedial striatum in instrumental conditioning. European Journal of Neuroscience, 22, 513–523. doi:10.1111/j.1460-9568.2005.04218.x
147. Yue, Y., Loh, J. M., & Lindquist, M. A. (2010). Adaptive spatial smoothing of fMRI images. Statistics and its Interface, 3, 3–13.