Cortical Signals for Rewarded Actions and Strategic Exploration

Neuron

Volumn 80, Issue 1, 2013, Pages 223-234

  Donahue, Christopher H ^a   Seo, Hyojung ^a   Lee, Daeyeol ^a,b  

^a YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b YALE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL EXPERIMENT; ANTERIOR CINGULATE; ARTICLE; CONTROLLED STUDY; DECISION MAKING; DORSAL ANTERIOR CINGULATE CORTEX; EXPLORATORY BEHAVIOR; FEMALE; FRONTAL CORTEX; LATERAL INTRAPARIETAL AREA; LEARNING; MALE; NONHUMAN; PARIETAL CORTEX; PREFRONTAL CORTEX; PRIORITY JOURNAL; REINFORCEMENT; REWARD; RHESUS MONKEY; SUPPLEMENTARY EYE FIELD; TASK PERFORMANCE;

ACTION POTENTIALS; ANIMALS; BEHAVIOR, ANIMAL; CHOICE BEHAVIOR; FEMALE; MACACA MULATTA; MALE; MOTOR CORTEX; NEURONS; PSYCHOMOTOR PERFORMANCE; REWARD; SACCADES; VISUAL FIELDS;

EID: 84884733456     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2013.07.040     Document Type: Article

References (63)

1. Aston-Jones G., Cohen J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 2005, 28:403-450.
2. Averbeck B.B., Sohn J.W., Lee D. Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 2006, 9:276-282.
3. Baeg E.H., Kim Y.B., Huh K., Mook-Jung I., Kim H.T., Jung M.W. Dynamics of population code for working memory in the prefrontal cortex. Neuron 2003, 40:177-188.
4. Barraclough D.J., Conroy M.L., Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 2004, 7:404-410.
5. Behrens T.E.J., Woolrich M.W., Walton M.E., Rushworth M.F.S. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
6. Bernacchia A., Seo H., Lee D., Wang X.J. A reservoir of time constants for memory traces in cortical neurons. Nat. Neurosci. 2011, 14:366-372.
7. Brown J.W., Braver T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 2005, 307:1118-1121.
8. Chen L.L., Wise S.P. Neuronal activity in the supplementary eyefield during acquisition of conditional oculomotor associations. J.Neurophysiol. 1995, 73:1101-1121.
9. Chen L.L., Wise S.P. Evolution of directional preferences in the supplementary eye field during acquisition of conditional oculomotor associations. J.Neurosci. 1996, 16:3067-3081.
10. Chen L.L., Wise S.P. Conditional oculomotor learning: population vectors in the supplementary eye field. J.Neurophysiol. 1997, 78:1166-1169.
11. Coe B., Tomihara K., Matsuzawa M., Hikosaka O. Visual and anticipatory bias in three cortical eye fields of the monkey during an adaptive decision-making task. J.Neurosci. 2002, 22:5081-5090.
12. Curtis C.E., Lee D. Beyond working memory: the role of persistent activity in decision making. Trends Cogn. Sci. 2010, 14:216-222.
13. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879.
14. Dorris M.C., Glimcher P.W. Activity in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron 2004, 44:365-378.
15. Frank M.J., Seeberger L.C., O'reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
16. Haggard P. Human volition: towards a neuroscience of will. Nat. Rev. Neurosci. 2008, 9:934-946.
17. Hayden B.Y., Pearson J.M., Platt M.L. Fictive reward signals in the anterior cingulate cortex. Science 2009, 324:948-950.
18. Histed M.H., Pasupathy A., Miller E.K. Learning substrates in the primate prefrontal cortex and striatum: sustained activity related to successful actions. Neuron 2009, 63:244-253.
19. Husain M., Parton A., Hodgson T.L., Mort D., Rees G. Self-control during response conflict by human supplementary eye field. Nat. Neurosci. 2003, 6:117-118.
20. Isoda M., Hikosaka O. Switching from automatic to controlled action by monkey medial frontal cortex. Nat. Neurosci. 2007, 10:240-248.
21. Kennerley S.W., Dahmubed A.F., Lara A.H., Wallis J.D. Neurons in the frontal lobe encode the value of multiple decision variables. J.Cogn. Neurosci. 2009, 21:1162-1178.
22. Kennerley S.W., Behrens T.E.J., Wallis J.D. Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat. Neurosci. 2011, 14:1581-1589.
23. Kim H., Sul J.H., Huh N., Lee D., Jung M.W. Role of striatum in updating values of chosen actions. J.Neurosci. 2009, 29:14701-14712.
24. Kim H., Lee D., Jung M.W. Signals for previous goal choice persist in the dorsomedial, but not dorsolateral striatum of rats. J.Neurosci. 2013, 33:52-63.
25. Kravitz A.V., Tye L.D., Kreitzer A.C. Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 2012, 15:816-818.
26. Lau B., Glimcher P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J.Exp. Anal. Behav. 2005, 84:555-579.
27. Lee D., Seo H. Mechanisms of reinforcement learning and decision making in the primate dorsolateral prefrontal cortex. Ann. N Y Acad. Sci. 2007, 1104:108-122.
28. Lee D., Conroy M.L., McGreevy B.P., Barraclough D.J. Reinforcement learning and decision making in monkeys during a competitive game. Brain Res. Cogn. Brain Res. 2004, 22:45-58.
29. Lee D., Seo H., Jung M.W. Neural basis of reinforcement learning and decision making. Annu. Rev. Neurosci. 2012, 35:287-308.
30. Lee H., Ghim J.-W., Kim H., Lee D., Jung M.W. Hippocampal neural correlates for values of experienced events. J.Neurosci. 2012, 32:15053-15065.
31. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
32. Middlebrooks P.G., Sommer M.A. Neuronal correlates of metacognition in primate frontal cortex. Neuron 2012, 75:517-530.
33. Nakatani Y., Matsumoto Y., Mori Y., Hirashima D., Nishino H., Arikawa K., Mizunami M. Why the carrot is more effective than the stick: different dynamics of punishment memory and reward memory and its possible biological basis. Neurobiol. Learn. Mem. 2009, 92:370-380.
34. Nash J.F. Equilibrium points in n-person games. Proc. Natl. Acad. Sci. USA 1950, 36:48-49.
35. Olson C.R., Gettner S.N. Object-centered direction selectivity in the macaque supplementary eye field. Science 1995, 269:985-988.
36. Olson C.R., Gettner S.N., Ventura V., Carta R., Kass R.E. Neuronal activity in macaque supplementary eye field during planning of saccades in response to pattern and spatial cues. J.Neurophysiol. 2000, 84:1369-1384.
37. Parton A., Nachev P., Hodgson T.L., Mort D., Thomas D., Ordidge R., Morgan P.S., Jackson S., Rees G., Husain M. Role of the human supplementary eye field in the control of saccadic eye movements. Neuropsychologia 2007, 45:997-1008.
38. Pawitan Y. In All Likelihood: Statistical Modeling and Inference Using Likelihood 2001, Oxford University Press, Oxford.
39. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 1999, 400:233-238.
40. Purcell B.A., Weigand P.K., Schall J.D. Supplementary eye field during visual search: salience, cognitive control, and performance monitoring. J.Neurosci. 2012, 32:10273-10285.
41. Quilodran R., Rothé M., Procyk E. Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 2008, 57:314-325.
42. Roitman J.D., Shadlen M.N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J.Neurosci. 2002, 22:9475-9489.
43. Rushworth M.F.S., Behrens T.E.J. Choice, uncertainty and value in prefrontal and cingulate cortex. Nat. Neurosci. 2008, 11:389-397.
44. Schlag J., Schlag-Rey M. Evidence for a supplementary eye field. J.Neurophysiol. 1987, 57:179-200.
45. Schlag-Rey M., Amador N., Sanchez H., Schlag J. Antisaccade performance predicted by neuronal activity in the supplementary eye field. Nature 1997, 390:398-401.
46. Seo H., Lee D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J.Neurosci. 2007, 27:8366-8377.
47. Seo H., Lee D. Cortical mechanisms for reinforcement learning in competitive games. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2008, 363:3845-3857.
48. Seo H., Lee D. Behavioral and neural changes after gains and losses of conditioned reinforcers. J.Neurosci. 2009, 29:3627-3641.
49. Seo H., Barraclough D.J., Lee D. Dynamic signals related to choices and outcomes in the dorsolateral prefrontal cortex. Cereb. Cortex 2007, 17:i110-i117.
50. Seo H., Barraclough D.J., Lee D. Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game. J.Neurosci. 2009, 29:7278-7289.
51. Singer A.C., Frank L.M. Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron 2009, 64:910-921.
52. Stuphorn V., Taylor T.L., Schall J.D. Performance monitoring by the supplementary eye field. Nature 2000, 408:857-860.
53. Stuphorn V., Brown J.W., Schall J.D. Role of supplementary eye field in saccade initiation: executive, not direct, control. J.Neurophysiol. 2010, 103:801-816.
54. Sugrue L.P., Corrado G.S., Newsome W.T. Matching behavior and the representation of value in the parietal cortex. Science 2004, 304:1782-1787.
55. Sul J.H., Kim H., Huh N., Lee D., Jung M.W. Distinct roles of rodent orbitofrontal and medial prefrontal cortex in decision making. Neuron 2010, 66:449-460.
56. Sul J.H., Jo S., Lee D., Jung M.W. Role of rodent secondary motor cortex in value-based action selection. Nat. Neurosci. 2011, 14:1202-1208.
57. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
58. Tanji J. New concepts of the supplementary motor area. Curr. Opin. Neurobiol. 1996, 6:782-787.
59. Thevarajah D., Webb R., Ferrall C., Dorris M.C. Modeling thevalue of strategic actions in the superior colliculus. Front. Behav. Neurosci. 2010, 3:57.
60. Vickery T.J., Chun M.M., Lee D. Ubiquity and specificity of reinforcement signals throughout the human brain. Neuron 2011, 72:166-177.
61. Wallis J.D., Kennerley S.W. Heterogeneous reward signals in prefrontal cortex. Curr. Opin. Neurobiol. 2010, 20:191-198.
62. Wirth S., Avsar E., Chiu C.C., Sharma V., Smith A.C., Brown E., Suzuki W.A. Trial outcome and associative learning signals in the monkey hippocampus. Neuron 2009, 61:930-940.
63. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46:681-692.