States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning

Neuron

Volumn 66, Issue 4, 2010, Pages 585-595

  Gläscher, Jan ^a,b   Daw, Nathaniel ^c   Dayan, Peter ^d   O'Doherty, John P ^a,e  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY MEDICAL CENTER HAMBURG EPPENDORF   (Germany)

^c NEW YORK UNIVERSITY   (United States)

^d UNIVERSITY COLLEGE LONDON   (United Kingdom)

^e TRINITY COLLEGE DUBLIN   (Ireland)

Author keywords

Sysneuro

Indexed keywords

ADULT; ARTICLE; BRAIN REGION; COMPUTER MODEL; CORPUS STRIATUM; DECISION MAKING; FEMALE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; NERVE CELL NETWORK; NORMAL HUMAN; NUCLEAR MAGNETIC RESONANCE IMAGING; PREFRONTAL CORTEX; PRIORITY JOURNAL; PROBABILITY; REINFORCEMENT; SIGNAL DETECTION; SUBICULUM; TASK PERFORMANCE;

ADOLESCENT; ADULT; CHOICE BEHAVIOR; FEMALE; FORECASTING; HUMANS; LEARNING; MALE; MODELS, NEUROLOGICAL; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY); RESEARCH DESIGN; REWARD; YOUNG ADULT;

EID: 77953260848     PISSN: 08966273     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neuron.2010.04.016     Document Type: Article

References (55)

1. Balleine B.W., Delgado M.R., Hikosaka O. The role of the dorsal striatum in reward and decision-making. J. Neurosci. 2007, 27:8161-8165.
2. Barraclough D.J., Conroy M.L., Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 2004, 7:404-410.
3. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
4. Behrens T.E., Woolrich M.W., Walton M.E., Rushworth M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
5. Bucci D.J., Holland P.C., Gallagher M. Removal of cholinergic input to rat posterior parietal cortex disrupts incremental processing of conditioned stimuli. J. Neurosci. 1998, 18:8038-8046.
6. Camerer C., Ho T.H. Experience-Weighted Attraction Learning in Coordination Games: Probability Rules, Heterogeneity, and Time-Variation. J. Math. Psychol. 1998, 42:305-326.
7. Corbetta M., Kincade J.M., Ollinger J.M., McAvoy M.P., Shulman G.L. Voluntary orienting is dissociated from target detection in human posterior parietal cortex. Nat. Neurosci. 2000, 3:292-297.
8. Courchesne E., Hillyard S.A., Galambos R. Stimulus novelty, task relevance and the visual evoked potential in man. Electroencephalogr. Clin. Neurophysiol. 1975, 39:131-143.
9. Cui H., Andersen R.A. Posterior parietal cortex encodes autonomously selected motor plans. Neuron 2007, 56:552-559.
10. D'Ardenne K., McClure S.M., Nystrom L.E., Cohen J.D. BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 2008, 319:1264-1267.
11. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
12. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879.
13. Deichmann R., Gottfried J.A., Hutton C., Turner R. Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage 2003, 19:430-441.
14. Delgado M.R., Nystrom L.E., Fissell C., Noll D.C., Fiez J.A. Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol. 2000, 84:3072-3077.
15. Delgado M.R., Gillis M.M., Phelps E.A. Regulating the expectation of reward via cognitive strategies. Nat. Neurosci. 2008, 11:880-881.
16. Dickinson A. Causal learning: an associative analysis. Q. J. Exp. Psychol. 2001, 54B:3-25.
17. Dickinson A., Balleine B. The role of learning in motivation. Stevens' Handbook of Experimental Psychology 2002, Wiley, New York, NY. C. Gallistel (Ed.).
18. Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?. Neural Netw. 1999, 12:961-974.
19. Doya K., Samejima K., Katagiri K., Kawato M. Multiple model-based reinforcement learning. Neural Comput. 2002, 14:1347-1369.
20. Fabiani M., Friedman D. Changes in brain activity patterns in aging: the novelty oddball. Psychophysiology 1995, 32:579-594.
21. Fletcher P.C., Anderson J.M., Shanks D.R., Honey R., Carpenter T.A., Donovan T., Papadakis N., Bullmore E.T. Responses of human frontal cortex to surprising events are predicted by formal associative learning theory. Nat. Neurosci. 2001, 4:1043-1048.
22. Friston K.J., Ashburner J., Frith C.D., Poline J.B., Heather J.D., Frackowiak R.S. Spatial registration and normalization of images. Hum. Brain Mapp. 1995, 3:165-189.
23. Gershman S.J., Pesaran B., Daw N.D. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J. Neurosci. 2009, 29:13524-13531.
24. Gläscher J. Visualization of group inference data in functional neuroimaging. Neuroinformatics 2009, 7:73-82.
25. Gläscher J., Hampton A.N., O'Doherty J.P. Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. Cereb. Cortex 2009, 19:483-495.
26. Gold J.I., Shadlen M.N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 2002, 36:299-308.
27. Haber S.N. The primate basal ganglia: parallel and integrative networks. J. Chem. Neuroanat. 2003, 26:317-330.
28. Haruno M., Kawato M. Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J. Neurophysiol. 2006, 95:948-959.
29. Holroyd C.B., Coles M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 2002, 109:679-709.
30. Knutson B., Adams C.M., Fong G.W., Hommer D. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J. Neurosci. 2001, 21:RC159.
31. Knutson B., Taylor J., Kaufman M., Peterson R., Glover G. Distributed neural representation of expected value. J. Neurosci. 2005, 25:4806-4812.
32. Law C.T., Gold J.I. Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat. Neurosci. 2008, 11:505-513.
33. Lee D., Conroy M.L., McGreevy B.P., Barraclough D.J. Reinforcement learning and decision making in monkeys during a competitive game. Brain Res. Cogn. Brain Res. 2004, 22:45-58.
34. Logothetis N.K., Pauls J., Augath M., Trinath T., Oeltermann A. Neurophysiological investigation of the basis of the fMRI signal. Nature 2001, 412:150-157.
35. MacDonald A.W., Cohen J.D., Stenger V.A., Carter C.S. Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control. Science 2000, 288:1835-1838.
36. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
37. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
38. Mushiake H., Saito N., Sakamoto K., Itoyama Y., Tanji J. Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 2006, 50:631-641.
39. Nichols T., Brett M., Andersson J., Wager T., Poline J.B. Valid conjunction inference with the minimum statistic. Neuroimage 2005, 25:653-660.
40. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
41. Opitz B., Mecklinger A., Friederici A.D., von Cramon D.Y. The functional neuroanatomy of novelty processing: integrating ERP and fMRI results. Cereb. Cortex 1999, 9:379-391.
42. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 1980, 87:532-552.
43. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 1999, 400:233-238.
44. Samejima K., Doya K. Multiple representations of belief states and action values in corticobasal ganglia loops. Ann. N Y Acad. Sci. 2007, 1104:213-228.
45. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 1998, 80:1-27.
46. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
47. Seitz A.R., Dinse H.R. A common framework for perceptual learning. Curr. Opin. Neurobiol. 2007, 17:148-153.
48. Seymour B., O'Doherty J.P., Dayan P., Koltzenburg M., Jones A.K., Dolan R.J., Friston K.J., Frackowiak R.S. Temporal difference models describe higher-order learning in humans. Nature 2004, 429:664-667.
49. Strobel A., Debener S., Sorger B., Peters J.C., Kranczioch C., Hoechstetter K., Engel A.K., Brocke B., Goebel R. Novelty and target processing during an auditory novelty oddball: a simultaneous event-related potential and functional magnetic resonance imaging study. Neuroimage 2008, 40:869-883.
50. Sugrue L.P., Corrado G.S., Newsome W.T. Matching behavior and the representation of value in the parietal cortex. Science 2004, 304:1782-1787.
51. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
52. Thorndike E.L. A Proof of the Law of Effect. Science 1933, 77:173-175.
53. Tolman E.C. Cognitive maps in rats and men. Psychol. Rev. 1948, 55:189-208.
54. Tolman E.C., Honzik C.H. Introduction and Removal of Reward, and Maze Performance in Rats. University of California Publications in Psychology 1930, 4:257-275.
55. Yantis S., Schwarzbach J., Serences J.T., Carlson R.L., Steinmetz M.A., Pekar J.J., Courtney S.M. Transient neural activity in human parietal cortex during spatial attention shifts. Nat. Neurosci. 2002, 5:995-1002.