Predictive decision making driven by multiple time-linked reward representations in the anterior cingulate cortex

Nature Communications

Volumn 7, Issue 1, 2016, Pages

  Wittmann, Marco K ^a   Kolling, Nils ^a   Akaishi, Rei ^a,b   Chau, Bolton K H ^a,c   Brown, Joshua W ^d   Nelissen, Natalie ^a   Rushworth, Matthew F S ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF ROCHESTER   (United States)

^c HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

^d INDIANA UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; DECISION MAKING; DORSAL ANTERIOR CINGULATE CORTEX; EXPECTATION; HUMAN; PREDICTION; REINFORCEMENT; REWARD; VENTRAL STRIATUM; VENTROMEDIAL PREFRONTAL CORTEX; ATTENTION; BIOLOGICAL MODEL; CINGULATE GYRUS; DIAGNOSTIC IMAGING; FEMALE; LEARNING; MALE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; PREFRONTAL CORTEX; TIME FACTOR; YOUNG ADULT;

ADULT; ATTENTION; CHOICE BEHAVIOR; FEMALE; GYRUS CINGULI; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; MODELS, BIOLOGICAL; PREFRONTAL CORTEX; REWARD; TIME FACTORS; YOUNG ADULT;

EID: 85048086791     PISSN: None     EISSN: 20411723     Source Type: Journal    
DOI: 10.1038/ncomms12327     Document Type: Article

References (56)

1. Charnov, E. Optimal foraging: the marginal value theorem. Theor. Popul. Biol. 9, 129–136 (1976).
2. Stephens, D. W. & Krebs, J. R. Foraging Theory Princeton University Press (1986).
3. Kira, S., Yang, T. & Shadlen, M. N. A neural implementation of Wald’s sequential probability ratio test. Neuron 85, 861–873 (2015).
4. Boorman, E. D., Behrens, T. E. J., Woolrich, M. W. & Rushworth, M. F. S. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733–743 (2009).
5. FitzGerald, T. H. B., Seymour, B. & Dolan, R. J. The role of human orbitofrontal cortex in value comparison for incommensurable objects. J. Neurosci. 29, 8388–8395 (2009).
6. Basten, U., Biele, G., Heekeren, H. R. & Fiebach, C. J. How the brain integrates costs and benefits during decision making. Proc. Natl Acad. Sci. USA 107, 21767–21772 (2010).
7. Philiastides, M. G., Biele, G. & Heekeren, H. R. A mechanistic account of value computation in the human brain. Proc. Natl Acad. Sci. USA 107, 9430 (2010).
8. Hunt, L. T. et al. Mechanisms underlying cortical activity during value-guided choice. Nat. Neurosci. 15, 470–476 (2012).
9. Jocham, G., Hunt, L. T., Near, J. & Behrens, T. E. A mechanism for value-guided choice based on the excitation-inhibition balance in prefrontal cortex. Nat. Neurosci. 15, 960–961 (2012).
10. Strait, C. E., Blanchard, T. C. & Hayden, B. Y. Reward value comparison via mutual inhibition in ventromedial prefrontal cortex. Neuron 82, 1357–1366 (2014).
11. Noonan, M. P. et al. Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex. Proc. Natl Acad. Sci. USA 107, 20547–20552 (2010).
12. Hayden, B. Y., Pearson, J. M. & Platt, M. L. Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 14, 933–939 (2011).
13. Neta, M. et al. Spatial and temporal characteristics of error-related activity in the human brain. J. Neurosci. 35, 253–266 (2015).
14. Crespi, L. P. Quantitative variation of incentive and performance in the white rat. Am. J. Psychol. 55, 467–517 (1942).
15. McNamara, J. M., Fawcett, T. W. & Houston, A. I. An adaptive response to uncertainty generates positive and negative contrast effects. Science 340, 1084–1086 (2013).
16. Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J. & Rushworth, M. F. S. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).
17. Buckley, M. J. et al. Dissociable components of rule-guided behavior depend on distinct medial and prefrontal regions. Science 325, 52–58 (2009).
18. Bernacchia, A., Seo, H., Lee, D. & Wang, X. J. A reservoir of time constants for memory traces in cortical neurons. Nat. Neurosci. 14, 366–372 (2011).
19. Seo, H. & Lee, D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 27, 8366–8377 (2007).
20. Hikosaka, O. Basal ganglia mechanisms of reward-oriented eye movement. Ann. N. Y. Acad. Sci. 1104, 229–249 (2007).
21. Schultz, W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87–115 (2006).
22. Bromberg-Martin, E. S., Matsumoto, M., Nakahara, H. & Hikosaka, O. Multiple timescales of memory in lateral habenula and dopamine neurons. Neuron 67, 499–510 (2010).
23. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction MIT Press (1998).
24. Bayer, H. M. & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
25. Dayan, P. Prospective and retrospective temporal difference learning. Network (Bristol, England) 20, 32–46 (2009).
26. Niv, Y., Daw, N. D., Joel, D. & Dayan, P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 191, 507–520 (2007).
27. Constantino, S. M. & Daw, N. D. Learning the opportunity cost of time in a patch-foraging task. Cogn. Affect. Behav. Neurosci. 15, 837–853 (2015).
28. Kolling, N., Behrens, T. E., Mars, R. B. & Rushworth, M. F. Neural mechanisms of foraging. Science 336, 95–98 (2012).
29. Kolling, N., Behrens, T., Wittmann, M. K. & Rushworth, M. Multiple signals in anterior cingulate cortex. Curr. Opin. Neurobiol. 37, 36–43 (2016).
30. Kolling, N. et al. Anterior cingulate cortex and the value of the environment, search, persistence, and model updating. Nat. Neurosci. (in press).
31. Neubert, F. X., Mars, R. B., Thomas, A. G., Sallet, J. & Rushworth, M. F. Comparison of human ventral frontal cortex areas for cognitive control and language with areas in monkey frontal cortex. Neuron 81, 700–713 (2014).
32. Chau, B. K., Kolling, N., Hunt, L. T., Walton, M. E. & Rushworth, M. F. A neural mechanism underlying failure of optimal choice with multiple alternatives. Nat. Neurosci. 17, 463–470 (2014).
33. Plassmann, H., O'Doherty, J. & Rangel, A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 27, 9984–9988 (2007).
34. McGuire, J. T. & Kable, J. W. Medial prefrontal cortical activity reflects dynamic re-evaluation during voluntary persistence. Nat. Neurosci. 18, 760–766 (2015).
35. Neubert, F. X., Mars, R. B., Sallet, J. & Rushworth, M. F. Connectivity reveals relationship of brain areas for reward-guided learning and decision making in human and monkey frontal cortex. Proc. Natl Acad. Sci. USA 112, E2695–E2704 (2015).
36. Behrens, T. E. J., Hunt, L. T., Woolrich, M. W. & Rushworth, M. F. S. Associative learning of social value. Nature 456, 245–249 (2008).
37. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
38. Isoda, M. & Hikosaka, O. Switching from automatic to controlled action by monkey medial frontal cortex. Nat. Neurosci. 10, 240–248 (2007).
39. Mars, R. B. et al. Short-latency influence of medial frontal cortex on primary motor cortex during action selection under conflict. J. Neurosci. 29, 6926–6931 (2009).
40. Neubert, F. X., Mars, R. B., Buch, E. R., Olivier, E. & Rushworth, M. F. Cortical and subcortical interactions during action reprogramming and their related white matter pathways. Proc. Natl Acad. Sci. USA 107, 13240–13245 (2010).
41. De Martino, B., O'Doherty, J. P., Ray, D., Bossaerts, P. & Camerer, C. In the mind of the market: theory of mind biases value computation during financial bubbles. Neuron 79, 1222–1231 (2013).
42. Schuck, N. W. et al. Medial prefrontal cortex predicts internally driven strategy shifts. Neuron 86, 331–340 (2015).
43. Nakamura, K., Santos, G. S., Matsuzaki, R. & Nakahara, H. Differential reward coding in the subdivisions of the primate caudate during an oculomotor task. J. Neurosci. 32, 15963–15982 (2012).
44. Behrens, T. E. J., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. S. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).
45. Zhang, S. & Yu, A. J. Forgetful Bayes and myopic planning: Human learning and decision-making in a bandit setting. Adv. Neural Inf. Process. Syst. 26, 2607–2615 (2013).
46. Rutledge, R. B., Skandali, N., Dayan, P. & Dolan, R. J. A computational and neural model of momentary subjective well-being. Proc. Natl Acad. Sci. USA 111, 12252–12257 (2014).
47. Eldar, E. & Niv, Y. Interaction between emotional state and learning underlies mood instability. Nat. Commun. 6, 6149 (2015).
48. Alexander, W. H. & Brown, J. W. Hierarchical error representation: a computational model of anterior cingulate and dorsolateral prefrontal cortex. Neural Comput. 27, 1–57 (2015).
49. Daw, N., Gershman, S. & Seymour, B. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011).
50. Koechlin, E., Ody, C. & Kouneiher, F. The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).
51. Badre, D., Hoffman, J., Cooney, J. W. & D'Esposito, M. Hierarchical cognitive control deficits following damage to the human frontal lobe. Nat. Neurosci. 12, 515–522 (2009).
52. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B. & Dolan, R. J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
53. Doll, B. B., Duncan, K. D., Simon, D. A., Shohamy, D. & Daw, N. D. Model-based choices involve prospective neural activity. Nat. Neurosci. 18, 767–772 (2015).
54. Kolling, N., Wittmann, M. & Rushworth, M. F. Multiple neural mechanisms of decision making and their competition under changing risk pressure. Neuron 81, 1190–1202 (2014).
55. Boorman, E. D., Rushworth, M. F. & Behrens, T. E. Ventromedial prefrontal and anterior cingulate cortex adopt choice and default reference frames during sequential multi-alternative choice. J. Neurosci. 33, 2242–2253 (2013).
56. Scholl, J. et al. The good, the bad, and the irrelevant: neural mechanisms of learning real and hypothetical rewards and effort. J. Neurosci. 35, 11233–11251 (2015).