Modulation of feedback-related negativity during trial-and-error exploration and encoding of behavioral shifts

Frontiers in Neuroscience

Volumn , Issue 7 NOV, 2013, Pages

  Sallet, Jérôme ^a,b,c   Camille, Nathalie ^d   Procyk, Emmanuel ^a,b  

^a STEM CELL AND BRAIN RESEARCH INSTITUTE   (France)

^b UNIVERSITÉ LYON 1   (France)

^c UNIVERSITY OF OXFORD   (United Kingdom)

^d CNRS   (France)

Author keywords

Cingulate cortex; Feedback related negativity; Reward prediction error; Trial and error exploration

Indexed keywords

ADULT; ARTICLE; BEHAVIOR CHANGE; BRAIN ELECTROPHYSIOLOGY; CINGULATE GYRUS; CLINICAL ARTICLE; ELECTRODE; ELECTROENCEPHALOGRAM; EVENT RELATED POTENTIAL; EXPLORATORY BEHAVIOR; FEEDBACK RELATED NEGATIVITY; HUMAN; MALE; MENTAL TASK; NEUROMODULATION; PREDICTIVE VALUE; PROBLEM SOLVING; REACTION TIME; REWARD; VISUAL STIMULATION;

EID: 84888810924     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2013.00209     Document Type: Article

References (68)

1. Alexander, W. H., and Brown, J. W. (2011). Medial prefrontal cortex as an action-outcome predictor. Nat. Neurosci. 14, 1338-1344. doi:10.1038/nn.2921
2. Amiez, C., Joseph, J. P., and Procyk, E. (2005). Anterior cingulate error-related activity is modulated by predicted reward. Eur. J. Neurosci. 21, 3447-3452. doi:10.1111/j.1460-9568.2005.04170.x
3. Amiez, C., Joseph, J. P., and Procyk, E. (2006). Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex. 16, 1040-1055. doi:10.1093/cercor/bhj046
4. Amiez, C., Neveu, R., Warrot, D., Petrides, M., Knoblauch, K., and Procyk, E. (2013). The location of feedback-related activity in the midcingulate cortex is predicted by local morphology. J. Neurosci. 33, 2217-2228. doi:10.1523/JNEUROSCI.2779-12.2013
5. Amiez, C., Sallet, J., Procyk, E., and Petrides, M. (2012). Modulation of feedback related activity in the rostral anterior cingulate cortex during trial and error exploration. Neuroimage. 63, 1078-1090. doi:10.1016/j.neuroimage.2012.06.023
6. Bellebaum, C., and Daum, I. (2008). Learning-related changes in reward expectancy are reflected in the feedback-related negativity. Eur. J. Neurosci. 27, 1823-1835. doi:10.1111/j.1460-9568.2008.06138.x
7. Bellebaum, C., Polezzi, D., and Daum, I. (2010). It is less than you expected: the feedback-related negativity reflects violations of reward magnitude expectations. Neuropsychologia. 48, 3343-3350. doi:10.1016/j.neuropsychologia.2010.07.023
8. Beste, C., Saft, C., Andrich, J., Gold, R., and Falkenstein, M. (2006). Error processing in Huntington's disease. PLoS ONE 1:e86. doi:10.1371/journal.pone.0000086
9. Bromberg-Martin, E. S., and Hikosaka, O. (2011). Lateral habenula neurons signal errors in the prediction of reward information. Nat. Neurosci. 14, 1209-1216. doi:10.1038/nn.2902
10. Bromberg-Martin, E. S., Matsumoto, M., and Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron. 68, 815-834. doi:10.1016/j.neuron.2010.11.022
11. Cavanagh, J. F., Figueroa, C. M., Cohen, M. X., and Frank, M. J. (2012). Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb. Cortex. 22, 2575-2586. doi:10.1093/cercor/bhr332
12. Cavanagh, J. F., Frank, M. J., Klein, T. J., and Allen, J. J. (2010). Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. Neuroimage. 49, 3198-3209. doi:10.1016/j.neuroimage.2009.11.080
13. Chase, H. W., Swainson, R., Durham, L., Benham, L., and Cools, R. (2011). Feedback-related negativity codes prediction error but not behavioral adjustment during probabilistic reversal learning. J. Cogn. Neurosci. 23, 936-946. doi:10.1162/jocn.2010.21456
14. Cohen, M. X., Elger, C. E., and Ranganath, C. (2007). Reward expectation modulates feedback-related negativity and EEG spectra. Neuroimage. 35, 968-978. doi:10.1016/j.neuroimage.2006.11.056
15. Cohen, M. X., Wilmes, K., and Vijver, I. (2011). Cortical electrophysiological network dynamics of feedback learning. Trends Cogn. Sci. 15, 558-566. doi:10.1016/j.tics.2011.10.004
16. De Bruijn, E. R., Sabbe, B. G., Hulstijn, W., Ruigt, G. S., and Verkes, R. J. (2006). Effects of antipsychotic and antidepressant drugs on action monitoring in healthy volunteers. Brain Res. 1105, 122-129. doi:10.1016/j.brainres.2006.01.006
17. Falkenstein, M., Hielscher, H., Dziobek, I., Schwarzenau, P., Hoormann, J., Sunderman, B., et al. (2001). Action monitoring, error detection, and the basal ganglia: an ERP study. Neuroreport. 12, 157-161. doi:10.1097/00001756-200101220-00039
18. Ferdinand, N. K., Mecklinger, A., Kray, J., and Gehring, W. J. (2012). The processing of unexpected positive response outcomes in the mediofrontal cortex. J. Neurosci. 32, 12087-12092. doi:10.1523/JNEUROSCI.1410-12.2012
19. Gehring, W. J., and Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science. 295, 2279-2282. doi:10.1126/science.1066893
20. Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. U.S.A. 108(Suppl. 3), 15647-15654. doi:10.1073/pnas.1014269108
21. Hajihosseini, A., and Holroyd, C. B. (2013). Frontal midline theta and N200 amplitude reflect complementary information about expectancy and outcome evaluation. Psychophysiology. 50, 550-562. doi:10.1111/psyp.12040
22. Haruno, M., and Kawato, M. (2006). Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw. 19, 1242-1254. doi:10.1016/j.neunet.2006.06.007
23. Hayden, B. Y., Heilbronner, S. R., Pearson, J. M., and Platt, M. L. (2011a). Surprise signals in anterior cingulate cortex: neuronal encoding of unsigned reward prediction errors driving adjustment in behavior. J. Neurosci. 31, 4178-4187. doi:10.1523/JNEUROSCI.4652-10.2011
24. Hayden, B. Y., Pearson, J. M., and Platt, M. L. (2011b). Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 14, 933-939. doi:10.1038/nn.2856
25. Hayden, B. Y., and Platt, M. L. (2010). Neurons in anterior cingulate cortex multiplex information about reward and action. J. Neurosci. 30, 3339-3346. doi:10.1523/JNEUROSCI.4874-09.2010
26. Holroyd, C. B., and Coles, M. G. (2002). The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679-709. doi:10.1037/0033-295X.109.4.679
27. Holroyd, C. B., Krigolson, O. E., Baker, R., Lee, S., and Gibson, J. (2009). When is an error not a prediction error? An electrophysiological investigation. Cogn. Affect. Behav. Neurosci. 9, 59-70. doi:10.3758/CABN.9.1.59
28. Holroyd, C. B., Krigolson, O. E., and Lee, S. (2011). Reward positivity elicited by predictive cues. Neuroreport. 22, 249-252. doi:10.1097/WNR.0b013e328345441d
29. Holroyd, C. B., Nieuwenhuis, S., Yeung, N., and Cohen, J. D. (2003). Errors in reward prediction are reflected in the event-related brain potential. Neuroreport. 14, 2481-2484. doi:10.1097/00001756-200312190-00037
30. Holroyd, C. B., Nieuwenhuis, S., Yeung, N., Nystrom, L., Mars, R. B., Coles, M. G., et al. (2004). Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nat. Neurosci. 7, 497-498. doi:10.1038/nn1238
31. Ito, S., Stuphorn, V., Brown, J. W., and Schall, J. D. (2003). Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science. 302, 120-122. doi:10.1126/science.1087847
32. Johansen, J. P., and Fields, H. L. (2004). Glutamatergic activation of anterior cingulate cortex produces an aversive teaching signal. Nat. Neurosci. 7, 398-403. doi:10.1038/nn1207
33. Karlsson, M. P., Tervo, D. G., and Karpova, A. Y. (2012). Network resets in medial prefrontal cortex mark the onset of behavioral uncertainty. Science. 338, 135-139. doi:10.1126/science.1226518
34. Kennerley, S. W., Behrens, T. E., and Wallis, J. D. (2011). Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat. Neurosci. 14, 1581-1589. doi:10.1038/nn.2961
35. Kennerley, S. W., and Wallis, J. D. (2009). Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables. Eur. J. Neurosci. 29, 2061-2073. doi:10.1111/j.1460-9568.2009.06743.x
36. Khamassi, M., Enel, P., Dominey, P. F., and Procyk, E. (2013). Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters. Prog. Brain Res. 202, 441-464. doi:10.1016/B978-0-444-62604-2.00022-8
37. Kolling, N., Behrens, T. E., Mars, R. B., and Rushworth, M. F. (2012). Neural mechanisms of foraging. Science. 336, 95-98. doi:10.1126/science.1216930
38. Kvitsiani, D., Ranade, S., Hangya, B., Taniguchi, H., Huang, J. Z., and Kepecs, A. (2013). Distinct behavioural and network correlates of two interneuron types in prefrontal cortex. Nature. 498, 363-366. doi:10.1038/nature12176
39. Matsumoto, M., Matsumoto, K., Abe, H., and Tanaka, K. (2007). Medial prefrontal cell activity signaling prediction errors of action values. Nat. Neurosci. 10, 647-656. doi:10.1038/nn1890
40. Nakamura, K., Roesch, M. R., and Olson, C. R. (2005). Neuronal activity in macaque SEF and ACC during performance of tasks involving conflict. J. Neurophysiol. 93, 884-908. doi:10.1152/jn.00305.2004
41. Oliveira, F. T., McDonald, J. J., and Goodman, D. (2007). Performance monitoring in the anterior cingulate is not all error related: expectancy deviation and the representation of action-outcome associations. J. Cogn. Neurosci. 19, 1994-2004. doi:10.1162/jocn.2007.19.12.1994
42. Peterson, D. A., Lotz, D. T., Halgren, E., Sejnowski, T. J., and Poizner, H. (2011). Choice modulates the neural dynamics of prediction error processing during rewarded learning. Neuroimage. 54, 1385-1394. doi:10.1016/j.neuroimage.2010.09.051
43. Pfabigan, D. M., Alexopoulos, J., Bauer, H., and Sailer, U. (2011). Manipulation of feedback expectancy and valence induces negative and positive reward prediction error signals manifest in event-related brain potentials. Psychophysiology. 48, 656-664. doi:10.1111/j.1469-8986.2010.01136.x
44. Procyk, E., Tanaka, Y. L., and Joseph, J. P. (2000). Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat. Neurosci. 3, 502-508. doi:10.1038/74880
45. Quilodran, R., Rothe, M., and Procyk, E. (2008). Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron. 57, 314-325. doi:10.1016/j.neuron.2007.11.031
46. Rothe, M., Quilodran, R., Sallet, J., and Procyk, E. (2011). Coordination of high gamma activity in anterior cingulate and lateral prefrontal cortical areas during adaptation. J. Neurosci. 31, 11110-11117. doi:10.1523/JNEUROSCI.1016-11.2011
47. Rushworth, M. F., Kolling, N., Sallet, J., and Mars, R. B. (2012). Valuation and decision-making in frontal cortex: one or many serial or parallel systems? Curr. Opin. Neurobiol. 22, 946-955. doi:10.1016/j.conb.2012.04.011
48. Rutledge, R. B., Dean, M., Caplin, A., and Glimcher, P. W. (2010). Testing the reward prediction error hypothesis with an axiomatic model. J. Neurosci. 30, 13525-13536. doi:10.1523/JNEUROSCI.1747-10.2010
49. Sailer, U., Fischmeister, F. P., and Bauer, H. (2010). Effects of learning on feedback-related brain potentials in a decision-making task. Brain Res. 1342, 85-93. doi:10.1016/j.brainres.2010.04.051
50. Sallet, J., Quilodran, R., Rothé, M., Vezoli, J., Joseph, J. P., and Procyk, E. (2007). Expectations, gains, and losses in the anterior cingulate cortex. Cogn. Affect. Behav. Neurosci. 7, 327-336. doi:10.3758/CABN.7.4.327
51. San Martin, R. (2012). Event-related potential studies of outcome processing and feedback-guided learning. Front. Hum. Neurosci. 6:304. doi:10.3389/fnhum.2012.00304
52. San Martin, R., Manes, F., Hurtado, E., Isla, P., and Ibanez, A. (2010). Size and probability of rewards modulate the feedback error-related negativity associated with wins but not losses in a monetarily rewarded gambling task. Neuroimage. 51, 1194-1204. doi:10.1016/j.neuroimage.2010.03.031
53. Seo, H., and Lee, D. (2009). Behavioral and neural changes after gains and losses of conditioned reinforcers. J. Neurosci. 29, 3627-3641. doi:10.1523/JNEUROSCI.4726-08.2009
54. Shima, K., and Tanji, J. (1998). Role for cingulate motor area cells in voluntary movement selection based on reward. Science. 282, 1335-1338. doi:10.1126/science.282.5392.1335
55. Talmi, D., Atkinson, R., and El-Deredy, W. (2013). The feedback-related negativity signals salience prediction errors, not reward prediction errors. J. Neurosci. 33, 8264-8269. doi:10.1523/JNEUROSCI.5695-12.2013
56. Ullsperger, M., and Von Cramon, D. Y. (2001). Subprocesses of performance monitoring: a dissociation of error processing and response competition revealed by event-related fMRI and ERPs. Neuroimage. 14, 1387-1401. doi:10.1006/nimg.2001.0935
57. Van De Vijver, I., Ridderinkhof, K. R., and Cohen, M. X. (2011). Frontal oscillatory dynamics predict feedback learning and action adjustment. J. Cogn. Neurosci. 23, 4106-4121. doi:10.1162/jocn_a_00110
58. Van Der Helden, J., and Boksem, M. A. (2012). Medial frontal negativity reflects learning from positive feedback. Psychophysiology. 49, 1109-1113. doi:10.1111/j.1469-8986.2012.01388.x
59. Vezoli, J., and Procyk, E. (2009). Frontal feedback-related potentials in nonhuman primates: modulation during learning and under haloperidol. J. Neurosci. 29, 15675-15683. doi:10.1523/JNEUROSCI.4943-09.2009
60. Vogt, B. A. (2009a). "Architecture, neurocytology and comparative organization of monkey and human cingulate cortices," in Cingulate Neurobiology and Disease, ed B. A. Vogt (Oxford: Oxford University Press), 65-93.
61. Vogt, B. A. (2009b). "Regions and subregions of the cingulate cortex," in Cingulate Neurobiology and Disease, ed B. A. Vogt (Oxford: Oxford University Press), 113-144.
62. Walsh, M. M., and Anderson, J. R. (2011). Modulation of the feedback-related negativity by instruction and experience. Proc. Natl. Acad. Sci. U.S.A. 108, 19048-19053. doi:10.1073/pnas.1117189108
63. Walsh, M. M., and Anderson, J. R. (2012). Learning from experience: event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neurosci. Biobehav. Rev. 36, 1870-1884. doi:10.1016/j.neubiorev.2012.05.008
64. Williams, Z. M., Bush, G., Rauch, S. L., Cosgrove, G. R., and Eskandar, E. N. (2004). Human anterior cingulate neurons and the integration of monetary reward with motor responses. Nat. Neurosci. 7, 1370-1375. doi:10.1038/nn1354
65. Womelsdorf, T., Johnston, K., Vinck, M., and Everling, S. (2010). Theta-activity in anterior cingulate cortex predicts task rules and their adjustments following errors. Proc. Natl. Acad. Sci. U.S.A. 107, 5248-5253. doi:10.1073/pnas.0906194107
66. Yasuda, A., Sato, A., Miyawaki, K., Kumano, H., and Kuboki, T. (2004). Error-related negativity reflects detection of negative reward prediction error. Neuroreport. 15, 2561-2565. doi:10.1097/00001756-200411150-00027
67. Yeung, N., and Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. J. Neurosci. 24, 6258-6264. doi:10.1523/JNEUROSCI.4537-03.2004
68. Zirnheld, P. J., Carroll, C. A., Kieffaber, P. D., O'donnell, B. F., Shekhar, A., and Hetrick, W. P. (2004). Haloperidol impairs learning and error-related negativity in humans. J. Cogn. Neurosci. 16, 1098-1112. doi:10.1162/0898929041502779