Cortical delta activity reflects reward prediction error and related behavioral adjustments, but at different times

NeuroImage

Volumn 110, Issue , 2015, Pages 205-216

  Cavanagh, James F ^a  

^a UNIVERSITY OF NEW MEXICO   (United States)

Author keywords

Decision making; Delta; Hierarchy; Prediction error; Reinforcement learning; Reward positivity

Indexed keywords

ADULT; ARTICLE; BEHAVIOR CHANGE; DECISION MAKING; DELTA RHYTHM; EVENT RELATED POTENTIAL; FEEDBACK SYSTEM; FEMALE; HUMAN; HUMAN EXPERIMENT; MALE; MOTOR CORTEX; NORMAL HUMAN; PARIETAL CORTEX; REINFORCEMENT; RESPONSE TIME; REWARD; BEHAVIOR; BRAIN CORTEX; ELECTROENCEPHALOGRAPHY; LEARNING; PHYSIOLOGY; PSYCHOLOGICAL ADJUSTMENT; PSYCHOMOTOR PERFORMANCE; REACTION TIME; STATISTICAL MODEL; YOUNG ADULT;

BEHAVIOR; CEREBRAL CORTEX; DELTA RHYTHM; ELECTROENCEPHALOGRAPHY; EMOTIONAL ADJUSTMENT; FEMALE; HUMANS; LEARNING; MALE; MODELS, STATISTICAL; PSYCHOMOTOR PERFORMANCE; REACTION TIME; REWARD; YOUNG ADULT;

EID: 84923034711     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2015.02.007     Document Type: Article

References (75)

1. Baker T.E., Holroyd C.B. Dissociated roles of the anterior cingulate cortex in reward and conflict processing as revealed by the feedback error-related negativity and N200. Biol. Psychol. 2011, 87(1):25-34.
2. Bernat E.M., Nelson L.D., Steele V.R., Gehring W.J., Patrick C.J. Externalizing psychopathology and gain-loss feedback in a simulated gambling task: dissociable components of brain response revealed by time-frequency analysis. J. Abnorm. Psychol. 2011, 120:352-364.
3. Botvinick M.M., Niv Y., Barto A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 2009, 113:262-280.
4. Bunge S.A. How we use rules to select actions: a review of evidence from cognitive neuroscience. Cogn. Affect. Behav. Neurosci. 2004, 4:564-579.
5. Buzsáki G., Draguhn A. Neuronal oscillations in cortical networks. Science 2004, 304:1926-1929.
6. Camerer C., Ho T.H. Experience-weighted attraction learning in normal form games. Econometrica 1999, 67:827-874.
7. Canolty R.T., Edwards E., Dalal S.S., Soltani M., Nagarajan S.S., Kirsch H.E., Berger M.S., Barbaro N.M., Knight R.T. High gamma power is phase-locked to theta oscillations in human neocortex. Science 2006, 313:1626-1628.
8. Caplin A., Dean M. Axiomatic methods, dopamine and reward prediction error. Curr. Opin. Neurobiol. 2008, 18:197-202.
9. Cavanagh J.F., Frank M.J. Frontal theta as a mechanism for cognitive control. Trends Cogn. Sci. 2014, 1-8.
10. Cavanagh J.F., Shackman A.J. Frontal midline theta reflects anxiety and cognitive control: meta-analytic evidence. J. Physiol. Paris 2014, (in press). 10.1016/j.jphysparis.2014.04.003.
11. Cavanagh J.F., Frank M.J., Klein T.J., Allen J.J.B. Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. Neuroimage 2010, 49:3198-3209.
12. Cavanagh J.F., Figueroa C.M., Cohen M.X., Frank M.J. Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb. Cortex 2012, 22:2575-2586.
13. Cavanagh J.F., Zambrano-Vazquez L., Allen J.J.B. Theta lingua franca: a common mid-frontal substrate for action monitoring processes. Psychophysiology 2012, 49:220-238.
14. Cavanagh J.F., Masters S.E., Bath K., Frank M.J. Conflict acts as an implicit cost in reinforcement learning. Nat. Commun. 2014, 5:5394.
15. Chase H.W., Swainson R., Durham L., Benham L., Cools R. Feedback-related negativity codes prediction error but not behavioral adjustment during probabilistic reversal learning. J. Cogn. Neurosci. 2010, 23(4):936-946.
16. Cheadle S., Wyart V., Tsetsos K., Myers N., de Gardelle V., Herce Castañón S., Summerfield C. Adaptive gain control during human perceptual choice. Neuron 2014, 81:1429-1441.
17. Cohen M.X. It's about time. Front. Hum. Neurosci. 2011, 5:2.
18. Cohen M.X. Analyzing Neural Time Series Data: Theory and Practice 2014, MIT Press, Cambridge, MA.
19. Cohen M.X., Cavanagh J.F. Single-trial regression elucidates the role of prefrontal theta oscillations in response conflict. Front. Psychol. 2011, 2:30.
20. Cohen M.X., Elger C.E., Ranganath C. Reward expectation modulates feedback-related negativity and EEG spectra. Neuroimage 2007, 35:968-978.
21. Collins A.G.E., Frank M.J. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychol. Rev. 2013, 120:190-229.
22. Collins A., Koechlin E. Reasoning, learning, and creativity: frontal lobe function and human decision-making. PLoS Biol. 2012, 10:e1001293.
23. Collins A.G.E., Cavanagh J.F., Frank M.J. Human EEG uncovers latent generalizable rule structure during learning. J. Neurosci. 2014, 34:4677-4685.
24. Daw N.D. Trial by trial data analysis using computational models. Decision Making, Affect, and Learning: Attention and Performance XXIII 2011, 1-26. Oxford University Press. M.R. Delgado, E.A. Phelps, T.W. Robbins (Eds.).
25. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879.
26. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
27. Delorme A., Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 2004, 134:9-21.
28. Den Ouden H.E.M., Kok P., de Lange F.P. How prediction errors shape perception, attention, and motivation. Front. Psychol. 2012, 3:548.
29. Doll B.B., Simon D.A., Daw N.D. The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 2012, 22:1075-1081.
30. Engel A.K., Fries P. Beta-band oscillations-signalling the status quo?. Curr. Opin. Neurobiol. 2010, 20:156-165.
31. Fischer A.G., Ullsperger M. Real and fictive outcomes are processed differently but converge on a common adaptive mechanism. Neuron 2013, 79:1243-1255.
32. Frank M.J., Moustafa A.A., Haughey H.M., Curran T., Hutchison K.E. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. U. S. A. 2007, 104:16311-16316.
33. Friedman D., Cycowicz Y.M., Gaeta H. The novelty P3: an event-related brain potential (ERP) sign of the brain's evaluation of novelty. Neurosci. Biobehav. Rev. 2001, 25:355-373.
34. Friston K. Learning and inference in the brain. Neural Netw. 2003, 16:1325-1352.
35. Friston K. A theory of cortical responses. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2005, 360:815-836.
36. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
37. Gold J.I., Shadlen M.N. The neural basis of decision making. Annu. Rev. Neurosci. 2007, 30:535-574.
38. Hajihosseini A., Holroyd C.B. Frontal midline theta and N200 amplitude reflect complementary information about expectancy and outcome evaluation. Psychophysiology 2013, 50(6):550-562.
39. HajiHosseini A., Rodríguez-Fornells A., Marco-Pallarés J. The role of beta-gamma oscillations in unexpected rewards processing. Neuroimage 2012, 60:1678-1685.
40. Hauser T.U., Iannaccone R., Stämpfli P., Drechsler R., Brandeis D., Walitza S., Brem S. The feedback-related negativity (FRN) revisited: new insights into the localization, meaning and network organization. Neuroimage 2014, 84:159-168.
41. Holroyd C.B., Coles M.G. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 2002, 109:679-709.
42. Holroyd CBC.B., Mcclure SMS.M. Hierarchical control over effortful behavior by rodent medial frontal cortex. Psychol. Rev. 2015, 122(1):54-83.
43. Holroyd C.B., Pakzad-Vaezi K.L., Krigolson O.E. The feedback correct-related positivity: sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology 2008, 45:688-697.
44. Ichikawa N., Siegle G.J., Dombrovski A., Ohira H. Subjective and model-estimated reward prediction: association with the feedback-related negativity (FRN) and reward prediction error in a reinforcement learning task. Int. J. Psychophysiol. 2010, 78:273-283.
45. Jepma M., Nieuwenhuis S. Pupil diameter predicts changes in the exploration-exploitation trade-off: evidence for the adaptive gain theory. J. Cogn. Neurosci. 2011, 23:1587-1596.
46. Kelly S.P., O'Connell R.G. Internal and external influences on the rate of sensory evidence accumulation in the human brain. J. Neurosci. 2013, 33:19434-19441.
47. Krugel L.K., Biele G., Mohr P.N.C., Li S.C., Heekeren H.R. Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions. Proc. Natl. Acad. Sci. U. S. A. 2009, 106:17951-17956.
48. Kujawa A., Proudfit G.H., Klein D.N. Neural reactivity to rewards and losses in offspring of mothers and fathers with histories of depressive and anxiety disorders. J. Abnorm. Psychol. 2014, 123:287-297.
49. Lachaux J.P., Rodriguez E., Martinerie J., Varela F.J. Measuring phase synchrony in brain signals. Hum. Brain Mapp. 1999, 8:194-208.
50. Lakatos P., Karmos G., Mehta A.D., Ulbert I., Schroeder C.E. Entrainment of neuronal oscillations as a mechanism of attentional selection. Science 2008, 320:110-113.
51. Luck S. An Introduction to the Event-Related Potential Technique (Cognitive Neuroscience) 2005, A Bradford Book. 1st ed.
52. Lukie C.N., Montazer-Hojat S., Holroyd C.B. Developmental changes in the reward positivity: an electrophysiological trajectory of reward processing. Dev. Cogn. Neurosci. 2014, 9:191-199.
53. Luu P., Tucker D.M., Makeig S. Frontal midline theta and the error-related negativity: neurophysiological mechanisms of action regulation. Clin. Neurophysiol. 2004, 115:1821-1835.
54. Marco-Pallares J., Cucurell D., Cunillera T., Garcia R., Andres-Pueyo A., Munte T.F., Rodriguez-Fornells A. Human oscillatory activity associated to reward processing in a gambling task. Neuropsychologia 2008, 46:241-248.
55. Nieuwenhuis S., Aston-Jones G., Cohen J.D. Decision making, the P3, and the locus coeruleus-norepinephrine system. Psychol. Bull. 2005, 131:510-532.
56. Nolan H., Whelan R., Reilly R.B. FASTER: fully automated statistical thresholding for EEG artifact rejection. J. Neurosci. Methods 2010, 192:152-162.
57. O'Connell R.G., Dockree P.M., Kelly S.P. A supramodal accumulation-to-bound signal that determines perceptual decisions in humans. Nat. Neurosci. 2012, 15:1729-1735.
58. O'Reilly J.X. Making predictions in a changing world-inference, uncertainty, and learning. Front. Neurosci. 2013, 7:105.
59. Payzan-LeNestour E., Bossaerts P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 2011, 7:e1001048.
60. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 1980, 87:532-552.
61. Pearson J.M., Hayden B.Y., Raghavachari S., Platt M.L. Neurons in posterior cingulate cortex signal exploratory decisions in a dynamic multioption choice task. Curr. Biol. 2009, 19:1532-1537.
62. Pearson J.M., Heilbronner S.R., Barack D.L., Hayden B.Y., Platt M.L. Posterior cingulate cortex: adapting behavior to a changing world. Trends Cogn. Sci. 2011, 15:143-151.
63. Philiastides M.G., Biele G., Vavatzanidis N., Kazzer P., Heekeren H.R. Temporal dynamics of prediction error processing during reward-based decision making. Neuroimage 2010, 53:221-232.
64. Polich J. Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol. 2007, 118:2128-2148.
65. Ribas-Fernandes J.J.F., Solway A., Diuk C., McGuire J.T., Barto A.G., Niv Y., Botvinick M.M. A neural signature of hierarchical reinforcement learning. Neuron 2011, 71:370-379.
66. Schroeder C.E., Lakatos P. Low-frequency neuronal oscillations as instruments of sensory selection. Trends Neurosci. 2009, 32:9-18.
67. Stephan K.E., Penny W.D., Daunizeau J., Moran R.J., Friston K.J. Bayesian model selection for group studies. Neuroimage 2009, 46:1004-1017.
68. Sugrue L.P., Corrado G.S., Newsome W.T. Choosing the greater of two goods: neural currencies for valuation and decision making. Nat. Rev. Neurosci. 2005, 6:363-375.
69. Talmi D., Atkinson R., El-Deredy W. The feedback-related negativity signals salience prediction errors, not reward prediction errors. J. Neurosci. 2013, 33:8264-8269.
70. Trujillo L.T., Allen J.J. Theta EEG dynamics of the error-related negativity. Clin. Neurophysiol. 2007, 118:645-668.
71. Uhlhaas P.J., Roux F., Rodriguez E., Rotarska-Jagiela A., Singer W. Neural synchrony and the development of cortical networks. Trends Cogn. Sci. 2010, 14:72-80.
72. Walsh, M.M., Anderson, J.R., 2011. Modulation of the feedback-related negativity by instruction and experience. 2011.
73. Walsh M.M., Anderson J.R. Learning from experience: event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neurosci. Biobehav. Rev. 2012, 36:1870-1884.
74. Weinberg A., Riesel A., Proudfit G.H. Show me the money: the impact of actual rewards and losses on the feedback negativity. Brain Cogn. 2014, 87:134-139.
75. Wyart V., de Gardelle V., Scholl J., Summerfield C. Rhythmic fluctuations in evidence accumulation during decision making in the human brain. Neuron 2012, 76:847-858.