Surprise signals in anterior cingulate cortex: Neuronal encoding of unsigned reward prediction errors driving adjustment in behavior

Journal of Neuroscience

Volumn 31, Issue 11, 2011, Pages 4178-4187

  Hayden, Benjamin Y ^a,b   Heilbronner, Sarah R ^a,b   Pearson, John M ^a,b   Platt, Michael L ^a,b  

^a DUKE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL EXPERIMENT; ANTERIOR CINGULATE; ARTICLE; ATTENTION; BEHAVIOR; CONFLICT; CONTROLLED STUDY; DECISION MAKING; DOPAMINERGIC NERVE CELL; ERROR; LEARNING; MACACA MULATTA; MALE; MOTOR CONTROL; NERVE CELL; NEUROMODULATION; NONHUMAN; PREDICTION; PRIORITY JOURNAL; REWARD;

ACTION POTENTIALS; ADAPTATION, PSYCHOLOGICAL; ANIMALS; ATTENTION; BEHAVIOR, ANIMAL; CHOICE BEHAVIOR; ELECTROPHYSIOLOGY; GYRUS CINGULI; MACACA MULATTA; MICROELECTRODES; NEURONS; PHOTIC STIMULATION; PROBABILITY LEARNING; REWARD;

EID: 79952746993     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.4652-10.2011     Document Type: Article

References (84)

1. Adrian ED, Matthews R (1927) The action of light on the eye. I. The discharge of impulses in the optic nerve and its relation to the electric changes in the retina. J Physiol 63:378-414.
2. Amiez C, Joseph JP, Procyk E (2005) Anterior cingulate error-related activity is modulated by predicted reward. Eur J Neurosci 21:3447-3452. (Pubitemid 40961439)
3. Amiez C, Joseph JP, Procyk E (2006) Reward encoding in the monkey anterior cingulate cortex. Cereb Cortex 16:1040-1055. (Pubitemid 43992453)
4. Aston-Jones G, Cohen JD (2005) An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu Rev Neurosci 28:403-450. (Pubitemid 41098911)
5. Aston-Jones G, Ennis M, Pieribone VA, Nickell WT, Shipley MT (1986) The brain nucleus locus coeruleus: restricted afferent control of a broad efferent network. Science 234:734-737. (Pubitemid 17184220)
6. Aston-Jones G, Rajkowski J, Kubiak P, Alexinsky T (1994) Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task. J Neurosci 14:4467-4480. (Pubitemid 24200566)
7. Aston-Jones G, Rajkowski J, Kubiak P (1997) Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 80:697-715. (Pubitemid 27334615)
8. Badgaiyan RD, Posner MI (1998) Mapping the cingulate cortex in response selection and monitoring. Neuroimage 7:255-260. (Pubitemid 28372795)
9. Barlow HB (1972) Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1:371-394.
10. Barraclough DJ, Conroy ML, Lee D (2004) Prefrontal cortex and decision making in a mixed-strategy game. Nat Neurosci 7:404-410. (Pubitemid 38429181)
11. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141. (Pubitemid 40922855)
12. Belova MA, Paton JJ, Morrison SE, Salzman CD (2007) Expectation modulates responses to pleasant and aversive stimuli in primate amygdala. Neuron 55:970-984. (Pubitemid 47391187)
13. Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD (2001) Conflict monitoring and cognitive control. Psychol Rev 108:624-652. (Pubitemid 33604139)
14. Brainard DH (1997) The Psychophysics Toolbox. Spat Vis 10:433-436.
15. Carter CS, Macdonald AM, Botvinick M, Ross LL, Stenger VA, Noll D, Cohen JD (2000) Parsing executive processes: strategic vs. evaluative functions of the anterior cingulate cortex. Proc Natl Acad Sci U S A 97:1944-1948. (Pubitemid 30118548)
16. Cornelissen FW, Peters EM, Palmer J (2002) The Eyelink Toolbox: eye tracking with MATLAB and the Psychophysics Toolbox. Behav Res Methods Instrum Comput 34:613-617.
17. Courville AC, Daw ND, Touretzky DS (2006) Bayesian theories of conditioning in a changing world. Trends Cogn Sci 10:294-300. (Pubitemid 44109995)
18. Davis KD, Hutchison WD, Lozano AM, Tasker RR, Dostrovsky JO (2000) Human anterior cingulate cortex neurons modulated by attention-demanding tasks. J Neurophysiol 83:3575-3577. (Pubitemid 30408888)
19. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441:876-879.
20. Desimone R, Duncan J (1995) Neural mechanisms of selective visual attention. Annu Rev Neurosci 18:193-222.
21. Egeth HE, Yantis S (1997) Visual attention: control, representation, and time course. Annu Rev Psychol 48:269-297. (Pubitemid 127485178)
22. Ellsberg D (1961) Risk, ambiguity, and the savage axioms. Q J Econ 75:643-669.
23. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898-1902. (Pubitemid 36356659)
24. Fox CR, Tversky A (1995) Ambiguity aversion and comparative ignorance. Q J Econ 110:585-603.
25. Gehring WJ, Willoughby AR (2002) The medial frontal cortex and the rapid processing of monetary gains and losses. Science 295:2279-2282. (Pubitemid 34258845)
26. Hall G, Pearce JM (1979) Latent inhibition of a CS during CS-US pairings. J Exp Psychol Anim Behav Process 5:31-42.
27. Hayden BY, Nair AC, McCoy AN, Platt ML (2008) Posterior cingulate cortex mediates outcome-contingent allocation of behavior. Neuron 60:19-25.
28. Hayden BY, Pearson JM, Platt ML (2009) Fictive reward signals in the anterior cingulate cortex. Science 324:948-950.
29. Hayden BY, Heilbronner SR, Platt ML (2010) Ambiguity aversion in rhesus macaques. Front Neurosci 4:pii:166.
30. Holland PC, Gallagher M (1999) Amygdala circuitry in attentional and representational processes. Trends Cogn Sci 3:65-73. (Pubitemid 29099880)
31. Holland PC, Gallagher M (2006) Different roles for amygdala central nucleus and substantia innominata in the surprise-induced enhancement of learning. J Neurosci 26:3791-3797.
32. Holland PC, Han JS, Gallagher M (2000) Lesions of the amygdala central nucleus impair performance on a selective attention task. J Neurosci 20:6701-6706.
33. Holroyd CB, Coles MG (2002) The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev 109:679-709.
34. Hsu M, Bhatt M, Adolphs R, Tranel D, Camerer CF (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science 310:1680-1683. (Pubitemid 41780790)
35. Huettel SA, Stowe CJ, Gordon EM, Warner BT, Platt ML (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron 49:765-775. (Pubitemid 43290135)
36. Ito S, Stuphorn V, Brown JW, Schall JD (2003) Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302:120-122. (Pubitemid 37210702)
37. Kaye H, Pearce JM (1984) The strength of the orienting response during blocking. Q J Exp Psychol B 36:131-144.
38. Kennerley SW, Wallis JD (2009) Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables. Eur J Neurosci 29:2061-2073.
39. Kennerley SW, Walton ME, Behrens TE, Buckley MJ, Rushworth MF (2006) Optimal decision making and the anterior cingulate cortex. Nat Neurosci 9:940-947. (Pubitemid 43980509)
40. Kennerley SW, Dahmubed AF, Lara AH, Wallis JD (2009) Neurons in the frontal lobe encode the value of multiple decision variables. J Cogn Neurosci 21:1162-1178.
41. Kerns JG, Cohen JD, MacDonald AW 3rd, Cho RY, Stenger VA, Carter CS (2004) Anterior cingulate conflict monitoring and adjustments in control. Science 303:1023-1026. (Pubitemid 38209708)
42. Kondo H, Osaka N, Osaka M (2004) Cooperation of the anterior cingulate cortex and dorsolateral prefrontal cortex for attention shifting. Neuroimage 23:670-679. (Pubitemid 39370618)
43. Lauwereyns J, Takikawa Y, Kawagoe R, Kobayashi S, Koizumi M, Coe B, Sakagami M, Hikosaka O (2002) Feature-based anticipation of cues that predict reward in monkey caudate nucleus. Neuron 33:463-473. (Pubitemid 34158848)
44. Lee D, Seo H (2007) Mechanisms of reinforcement learning and decision making in the primate dorsolateral prefrontal cortex. Ann N Y Acad Sci 1104:108-122. (Pubitemid 47092733)
45. Lin SC, Nicolelis MA (2008) Neuronal ensemble bursting in the basal forebrain encodes salience irrespective of valence. Neuron 59:138-149.
46. Mackintosh NJ (1975) A theory of attention: variations in the associability of stimuli with reinforcement. Psychol Rev 82:276-298.
47. Matsumoto M, Hikosaka O (2007) Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447:1111-1115. (Pubitemid 47014424)
48. Matsumoto M, Hikosaka O (2009) Representation of negative motivational value in the primate lateral habenula. Nat Neurosci 12:77-84.
49. Matsumoto M, Matsumoto K, Abe H, Tanaka K (2007) Medial prefrontal cell activity signaling prediction errors of action values. Nat Neurosci 10:647-656. (Pubitemid 46652446)
50. McGaugh JL (2004) The amygdala modulates the consolidation of memories of emotionally arousing experiences. Annu Rev Neurosci 27:1-28.
51. Mesulam MM (1981) A cortical network for directed attention and unilateral neglect. Ann Neurol 10:309-325.
52. Mesulam MM (1999) Spatial attention and neglect: parietal, frontal and cingulate contributions to the mental representation and attentional targeting of salient extrapersonal events. Philos Trans R Soc Lond B Biol Sci 354:1325-1346.
53. Mesulam MM, Nobre AC, Kim YH, Parrish TB, Gitelman DR (2001) Heterogeneity of cingulate contributions to spatial attention. Neuroimage 13:1065-1072. (Pubitemid 32553537)
54. Montague PR, Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377:725-728. (Pubitemid 26450238)
55. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936-1947. (Pubitemid 26145969)
56. Morecraft RJ, Van Hoesen GW (1993) Frontal granual cortex input to the cingulate (M3), supplementary (M2) and primary (M1) motor cortices in the rhesus monkey. J Comp Neurol 337:669-689.
57. Nakamura K, Roesch MR, Olson CR (2005) Neuronal activity in macaque SEF and ACC during performance of tasks involving conflict. J Neurophysiol 93:884-908.
58. Paus T (2001) Primate anterior cingulate cortex: where motor control, drive and cognition interface. Nat Rev Neurosci 2:417-424.
59. Pearce JM, Hall G (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 87:532-552.
60. Pearson JM, Hayden BY, Raghavachari S, Platt ML (2009) Neurons in posterior cingulate cortex signal exploratory decisions in a dynamic multioption choice task. Current Biol 19:1532-1537.
61. Poldrack RA, Packard MG (2003) Competition among multiple memory systems: converging evidence from animal and human brain studies. Neuropsychologia 41:245-251. (Pubitemid 35365954)
62. Posner MI, Petersen SE (1990) The attention system of the human brain. Annu Rev Neurosci 13:25-42. (Pubitemid 20100639)
63. Procyk E, Tanaka YL, Joseph JP (2000) Anterior cingulate activity during routine and non-routine sequential behaviors in macaques. Nat Neurosci 3:502-508. (Pubitemid 30495634)
64. Quilodran R, Rothé M, Procyk E (2008) Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 57:314-325.
65. Rescorla RA, and Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical conditioning II: current research and theory (Black AH, Prokasy WF, eds.). New York: Appleton-Century-Crofts.
66. Roesch MR, Calu DJ, Esber GR, Schoenbaum G (2010) All that glitters: dissociating attention and outcome expectancy from prediction error signals. J Neurophysiol 104:587-595.
67. Rushworth MF, Kennerley SW, Walton ME (2005) Cognitive neuroscience: resolving conflict in and over the medial frontal cortex. Curr Biol 15:R54-R56.
68. Rushworth MF, Buckley MJ, Behrens TE, Walton ME, Bannerman DM (2007) Functional organization of the medial frontal cortex. Curr Opin Neurobiol 17:220-227. (Pubitemid 46575241)
69. Sallet J, Quilodran R, Rothé M, Vezoli J, Joseph JP, Procyk E (2007) Expectations, gains, and losses in the anterior cingulate cortex. Cogn Affect Behav Neurosci 7:327-336.
70. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310:1337-1340. (Pubitemid 41698867)
71. Schultz W (2006) Behavioral theories and the neurophysiology of reward. Annu Rev Psychol 57:87-115.
72. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599. (Pubitemid 27120526)
73. Seo H, Lee D (2007) Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J Neurosci 27:8366-8377. (Pubitemid 47219624)
74. Shidara M, Richmond BJ (2002) Anterior cingulate: single neuronal signals related to degree of reward expectancy. Science 296:1709-1711. (Pubitemid 34579174)
75. Shima K, Tanji J (1998) Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282:1335-1338. (Pubitemid 28524499)
76. Stuphorn V, Taylor TL, Schall JD (2000) Performance monitoring by the supplementary eye field. Nature 408:857-860. (Pubitemid 32016095)
77. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Cambridge, MA: MIT.
78. Swan JA, Pearce JM (1988) The orienting response as an index of stimulus associability in rats. J Exp Psychol Anim Behav Process 14:292-301.
79. van Veen V, Cohen JD, Botvinick MM, Stenger VA, Carter CS (2001) Anterior cingulate cortex, conflict monitoring, and levels of processing. Neuroimage 14:1302-1308. (Pubitemid 33124178)
80. Weissman DH, Giesbrecht B, Song AW, Mangun GR, Woldorff MG (2003) Conflict monitoring in the human anterior cingulate cortex during selective attention to global and local object features. Neuroimage 19:1361-1368. (Pubitemid 37025837)
81. White NM, McDonald RJ (2002) Multiple parallel memory systems in the brain of the rat. Neurobiol Learn Mem 77:125-184. (Pubitemid 34971650)
82. Williams ZM, Bush G, Rauch SL, Cosgrove GR, Eskandar EN (2004) Human anterior cingulate neurons and the integration of monetary reward with motor responses. Nat Neurosci 7:1370-1375. (Pubitemid 41057760)
83. Yin HH, Knowlton BJ (2006) The role of the basal ganglia in habit formation. Nat Rev Neurosci 7:464-476. (Pubitemid 44012286)
84. Yu AJ, Dayan P (2005) Uncertainty, neuromodulation, and attention. Neuron 46:681-692. (Pubitemid 40797391)