Striatal activations signal prediction errors on confidence in the absence of external feedback

NeuroImage

Volumn 59, Issue 4, 2012, Pages 3457-3467

  Daniel, Reka ^a   Pollmann, Stefan ^a  

^a OTTO VON GUERICKE UNIVERSITY   (Germany)

Author keywords

Feedback; Nucleus accumbens; Observational learning; Reinforcement learning; Striatum

Indexed keywords

ADULT; ANATOMICAL VARIATION; ARTICLE; ATTENTION; BRAIN FUNCTION; CONTROLLED STUDY; CORPUS STRIATUM; DOPAMINERGIC SYSTEM; ELECTROENCEPHALOGRAM; FEEDBACK SYSTEM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; IMAGE PROCESSING; LEARNING; MALE; NERVE STIMULATION; NEUROIMAGING; NORMAL HUMAN; NUCLEUS ACCUMBENS; PREDICTION; PRIORITY JOURNAL; PUTAMEN; TASK PERFORMANCE;

ADULT; CORPUS STRIATUM; FEEDBACK, PHYSIOLOGICAL; FEMALE; HUMANS; LEARNING; MALE; YOUNG ADULT;

EID: 84855170021     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2011.11.058     Document Type: Article

References (86)

1. Abler B., Walter H., Erk S., Kammerer H., Spitzer M. Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage 2006, 31:790-795.
2. Aharon I., Etcoff N., Ariely D., Chabris C.F., O'Connor E., Breiter H.C. Beautiful faces have variable reward value: FMRI and behavioral evidence. Neuron 2001, 32:537-551.
3. Akaike H. A new look at the statistical model identification. IEEE Trans. Autom. Control 1974, 19:716-723.
4. Allwood C.M., Granhag P.A., Johansson H. Realism in confidence judgements of performance based on implicit learning. Eur. J. Cogn. Psychol. 2000, 12(2):165-188.
5. Aron A.R., Shohamy D., Clark J., Myers C., Gluck M.A., Poldrack R.A. Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning. J. Neurophysiol. 2004, 92:1144-1152.
6. Ashby F.G. Multidimensional models of perception and cognition. Multivariate Probability Distributions 1992, 1-34. Erlbaum, Hillsdale, NJ, USA, chapter.
7. Ashby F.G., Gott R.E. Decision rules in the perception and categorization of multidimensional stimuli. J. Exp. Psychol. Learn. Mem. Cogn. 1988, 14:33-53.
8. Ashby F.G., Maddox W.T. Human category learning. Annu. Rev. Psychol. 2005, 56:149-178.
9. Ashby F.G., Alfonso-Reese L.A., Turken A.U., Waldron E.M. A neuropsychological theory of multiple systems in category learning. Psychol. Rev. 1998, 105:442-481.
10. Ashby F.G., Maddox W.T., Bohil C.J. Observational versus feedback training in rule-based and information-integration category learning. Mem. Cognit. 2002, 30:666-677.
11. Baranski J.V., Petrusic W.M. Realism of confidence in sensory discrimination. Atten. Percept. Psychophys. 1999, 61:1369-1383.
12. Behrens T.E.J., Hunt L.T., Woolrich M.W., Rushworth M.F.S. Associative learning of social value. Nature 2008, 456:245-249.
13. Berns G.S., McClure S.M., Pagnoni G., Montague P.R. Predictability modulates human brain response to reward. J. Neurosci. 2001, 21:2793.
14. Brovelli A., Laksiri N., Nazarian B., Meunier M., Boussaoud D. Understanding the neural computations of arbitrary visuomotor learning through fMRI and associative learning theory. Cereb. Cortex 2008, 18:1485.
15. Bunzeck N., Düzel E. Absolute coding of stimulus novelty in the human substantia nigra/VTA. Neuron 2006, 51:369-379.
16. Butterfield B., Metcalfe J. The correction of errors committed with high confidence. Metacogn. Learn. 2006, 1:69-84.
17. Cincotta C.M., Seger C.A. Dissociation between striatal regions while learning to categorize via feedback and via observation. J. Cogn. Neurosci. 2007, 19:249-265.
18. Craig A.D.B. How do you feel-now? The anterior insula and human awareness. Nat. Rev. Neurosci. 2009, 10:59-70.
19. D'Esposito M., Postle B.R., Rypma B. Prefrontal cortical contributions to working memory: evidence from event-related fmri studies. Exp. Brain Res. 2000, 133:3-11.
20. Daniel R., Pollmann S. Comparing the neural basis of monetary reward and cognitive feedback during information-integration category learning. J. Neurosci. 2010, 30:47-55.
21. Daniel R., Wagner G., Koch K., Reichenbach J.R., Sauer H., Schlösser R.G.M. Assessing the neural basis of uncertainty in perceptual category learning through varying levels of distortion. J. Cogn. Neurosci. 2011, 1781-1793.
22. Daw N.D. Trial-by-trial data analysis using computational models. Affect, Learning and Decision Making, Attention and Performance XXIII 2011, Oxford University Press, Oxford.
23. Daw N.D., Doya K. The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 2006, 16:199-204.
24. Daw N.D., Shohamy D. The cognitive neuroscience of motivation and learning. Soc. Cognition 2008, 26:593-620.
25. Daw N., O'Doherty J., Dayan P., Seymour B., Dolan R. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876.
26. Dayan P., Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr. Opin. Neurobiol. 2008, 18:185-196.
27. Debener S., Ullsperger M., Siegel M., Fiehler K., von Cramon D.Y., Engel A.K. Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J. Neurosci. 2005, 25:11730-11737.
28. Delgado M.R. Reward-related responses in the human striatum. Ann. N. Y. Acad. Sci. 2007, 1104:70-88.
29. Delgado M.R., Frank R.H., Phelps E.A. Perceptions of moral character modulate the neural systems of reward during the trust game. Nat. Neurosci. 2005, 8:1611-1618.
30. Elwin E., Juslin P., Olsson H., Enkvist T. Constructivist coding: learning from selective feedback. Psychol. Sci. 2007, 18:105-110.
31. Evans A.C., Collins D.L., Mills S.R., Brown E.D., Kelly R.L., Peters T.M. 3D statistical neuroanatomical models from 305 MRI volumes. IEEE Conference Record 1993, 3:1813-1817.
32. Filoteo J.V., Maddox W.T., Davis J.D. A possible role of the striatum in linear and nonlinear category learning: evidence from patients with Huntington's disease. Behav. Neurosci. 2001, 115:786-798.
33. Franke G. Die Symptom-Checkliste von Derogatis - Deutsche Version 1995, Beltz Test Gesellschaft, Göttingen.
34. Gentsch A., Ullsperger P., Ullsperger M. Dissociable medial frontal negativities from a common monitoring system for self- and externally caused failure of goal achievement. Neuroimage 2009, 47:2023-2030.
35. Gottfried J.A., O'Doherty J., Dolan R.J. Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging. J. Neurosci. 2002, 22:10829-10837.
36. Grinband J., Hirsch J., Ferrera V.P. A neural representation of categorization uncertainty in the human brain. Neuron 2006, 49:757-763.
37. Haber S.N., Knutson B. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 2009, 35:4-26.
38. Han S., Huettel S.A., Raposo A., Adcock R.A., Dobbins I.G. Functional significance of striatal responses during episodic decisions: recovery or goal attainment?. J. Neurosci. 2010, 30:4767-4775.
39. Hare T.A., O'Doherty J., Camerer C.F., Schultz W., Rangel A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci. 2008, 28:5623.
40. Haruno M., Kawato M. Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J. Neurophysiol. 2006, 95:948.
41. Heldmann M., Rüsseler J., Münte T.F. Internal and external information in error processing. BMC Neurosci. 2008, 9:33.
42. Henriksson M.P., Elwin E., Juslin P. What is coded into memory in the absence of outcome feedback?. J. Exp. Psychol. Learn. Mem. Cogn. 2010, 36:1-16.
43. Hogarth R.M. Is confidence in decisions related to feedback? Evidence from random samples of real-world behavior. Information Sampling and Adaptive Cognition 2006, 456-485. Cambridge University Press, New York, USA. K. Fiedler, P. Juslin (Eds.).
44. Holroyd C.B., Coles M.G.H. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 2002, 109:679-709.
45. Jocham G., Ullsperger M. Neuropharmacology of performance monitoring. Neurosci. Biobehav. Rev. 2009, 33:48-60.
46. Jones E., Oliphant T., Peterson P., et al. SciPy: open source scientific tools for Python URL. http://www.scipy.org.
47. Kaplan F., Oudeyer P.Y. Maximizing learning progress: an internal reward system for development. Embodied Artificial Intelligence 2004, 629ff. Springer Verlag.
48. King-Casas B., Tomlin D., Anen C., Camerer C.F., Quartz S.R., Montague P.R. Getting to know you: reputation and trust in a two-person economic exchange. Science 2005, 308:78-83.
49. Knutson B., Cooper J.C. Functional magnetic resonance imaging of reward prediction. Curr. Opin. Neurol. 2005, 18:411.
50. Knutson B., Gibbs S.E.B. Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology (Berl) 2007, 191:813-822.
51. Logothetis N.K. The underpinnings of the bold functional magnetic resonance imaging signal. J. Neurosci. 2003, 23:3963.
52. Maddox W.T., Ashby F.G. Comparing decision bound and exemplar models of categorization. Percept. Psychophys. 1993, 53:49-70.
53. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
54. Miller E.K., Cohen J.D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 2001, 24:167-202.
55. Niv Y., Montague P.R. Theoretical and empirical studies of learning. Neuroeconomics: Decision Making and the Brain 2009, 331-351. Academic, NY, USA. P.W. Glimcher, C.F. Camerer, E. Fehr, A. Poldrack (Eds.).
56. Niv Y., Schoenbaum G. Dialogues on prediction errors. Trends Cogn. Sci. 2008, 12:265-272.
57. Nomura E.M., Maddox W.T., Filoteo J.V., Ing A.D., Gitelman D.R., Parrish T.B., Mesulam M.M., Reber P.J. Neural correlates of rule-based and information-integration visual category learning. Cereb. Cortex 2007, 17:37-43.
58. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
59. O'Doherty J., Dayan P., Schultz J., Deichmann R., Friston K., Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452.
60. O'Doherty J.P., Hampton A., Kim H. Model-based fMRI and its application to reward learning and decision making. Ann. N. Y. Acad. Sci. 2007, 1104:35-53.
61. Oldfield R. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 1971, 9:97-113.
62. Pagnoni G., Zink C.F., Montague P.R., Berns G.S. Activity in human ventral striatum locked to errors of reward prediction. Nat. Neurosci. 2002, 5:97-98.
63. Pessiglione M., Seymour B., Flandin G., Dolan R., Frith C. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006, 442:1042-1045.
64. Rangel A., Camerer C., Montague P.R. A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 2008, 9:545-556.
65. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory 1972, 64-99. Appleton Century Crofts, New York, USA. A.H. Black, W.F. Prokasy (Eds.).
66. Ridderinkhof K.R., Ullsperger M., Crone E.A., Nieuwenhuis S. The role of the medial frontal cortex in cognitive control. Science 2004, 306:443-447.
67. Rodriguez P.F., Aron A.R., Poldrack R.A. Ventral-striatal/nucleus-accumbens sensitivity to prediction errors during classification learning. Hum. Brain Mapp. 2006, 27:306-313.
68. Roediger H.L., Watson J.M., McDermott K.B., Gallo D.A. Factors that determine false recall: a multiple regression analysis. Psychon. Bull. Rev. 2001, 8:385-407.
69. Rushworth M.F.S., Behrens T.E.J. Choice, uncertainty and value in prefrontal and cingulate cortex. Nat. Neurosci. 2008, 11:389-397.
70. Schönberg T., Daw N.D., Joel D., O'Doherty J.P. Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J. Neurosci. 2007, 27:12860.
71. Schott B.H., Minuzzi L., Krebs R.M., Elmenhorst D., Lang M., Winz O.H., Seidenbecher C.I., Coenen H.H., Heinze H.J., Zilles K., Düzel E., Bauer A. Mesolimbic functional magnetic resonance imaging activations during reward anticipation correlate with reward-related ventral striatal dopamine release. J. Neurosci. 2008, 28:14311-14319.
72. Schultz W. Behavioral dopamine signals. Trends Neurosci. 2007, 30:203-210.
73. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
74. Seger C.A., Poldrack R.A., Prabhakaran V., Zhao M., Glover G.H., Gabrieli J.D. Hemispheric asymmetries and individual differences in visual concept learning as measured by functional MRI. Neuropsychologia 2000, 38:1316-1324.
75. Seger C.A., Peterson E.J., Cincotta C.M., Lopez-Paniagua D., Anderson C.W. Dissociating the contributions of independent corticostriatal systems to visual categorization learning through the use of reinforcement learning modeling and granger causality modeling. Neuroimage 2010, 50:644-656.
76. Shohamy D., Myers C.E., Grossman S., Sage J., Gluck M.A., Poldrack R.A. Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. Brain 2004, 127:851-859.
77. Shohamy D., Myers C.E., Kalanithi J., Gluck M.A. Basal ganglia and dopamine contributions to probabilistic category learning. Neurosci. Biobehav. Rev. 2008, 32:219-236.
78. Sporer S.L., Penrod S., Read D., Cutler B. Choosing, confidence, and accuracy: a meta-analysis of the confidence-accuracy relation in eyewitness identification studies. Psychol. Bull. 1995, 118:315.
79. Sutton R. Learning to predict by the methods of temporal differences. Mach. Learn. 1988, 3:9-44.
80. Ullsperger M. Performance monitoring in neurological and psychiatric patients. Int. J. Psychophysiol. 2006, 59:59-69.
81. Ullsperger M. Genetic association studies of performance monitoring and learning from feedback: the role of dopamine and serotonin. Neurosci. Biobehav. Rev. 2010, 34:649-659.
82. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J. Neurosci. 2007, 27:4019-4026.
83. Watkins, C., 1989. Learning from delayed rewards. PhD thesis, Cambridge, England.
84. White C.M., Koehler D.J. Missing information in multiple-cue probability learning. Mem. Cognit. 2004, 32:1007-1018.
85. Worsley K.J., Marrett S., Neelin P., Vandal A.C., Friston K.J., Evans A.C. A unified statistical approach for determining significant signals in images of cerebral activation. Hum. Brain Mapp. 1996, 4:58-73.
86. Zakay D., Tuvia R. Choice latency times as determinants of post-decisional confidence. Acta Psychol. 1998, 98:103-115.