The good, the bad, and the irrelevant: Neural mechanisms of learning real and hypothetical rewards and effort

Journal of Neuroscience

Volumn 35, Issue 32, 2015, Pages 11233-11251

  Scholl, Jacqueline ^a   Kolling, Nils ^a   Nelissen, Natalie ^a   Wittmann, Marco K ^a   Harmer, Catherine J ^a   Rushworth, Matthew F S ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Effort; Frontal pole; Hypothetical; Learning; Reward; VmPFC

Indexed keywords

ADULT; AMYGDALOID NUCLEUS; ANTERIOR CINGULATE; ANTERIOR INSULA; ARTICLE; BAYESIAN LEARNING; BEHAVIOR ASSESSMENT; BRAIN FUNCTION; COMPUTER ANALYSIS; CONNECTOME; CONTROLLED STUDY; CORPUS STRIATUM; CORRELATION ANALYSIS; DECISION MAKING; FEMALE; FRONTAL OPERCULUM; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEMODYNAMICS; HUMAN; HUMAN EXPERIMENT; INFORMATION PROCESSING; LEARNING; MALE; MATHEMATICAL MODEL; MULTIVARIATE LOGISTIC REGRESSION ANALYSIS; NERVOUS SYSTEM FUNCTION; NORMAL HUMAN; OUTCOME ASSESSMENT; PREDICTION; PREFRONTAL CORTEX; PRIORITY JOURNAL; PSYCHOPHYSIOLOGY; TASK PERFORMANCE; BIOLOGICAL MODEL; BRAIN; BRAIN COMPUTER INTERFACE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; REWARD; YOUNG ADULT;

ADULT; BRAIN; BRAIN-COMPUTER INTERFACES; CHOICE BEHAVIOR; FEMALE; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; MODELS, NEUROLOGICAL; REWARD; YOUNG ADULT;

EID: 84939132898     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.0396-15.2015     Document Type: Article

References (62)

1. Ahn WY, Krawitz A, Kim W, Busmeyer JR, Brown JW (2011) A modelbased fMRI analysis with hierarchical Bayesian parameter estimation. J Neurosci Psychol Econ 4:95–110.CrossRef Medline
2. Beckmann CF, Smith SM (2004) Probabilistic independent component analysis for functional magnetic resonance imaging. IEEE Trans Med Imaging 23:137-152CrossRef Medline
3. Beckmann CF, Jenkinson M, Smith SM (2003) General multilevel linear modeling for group analysis in fMRI. Neuroimage 20:1052-1063CrossRef Medline
4. Beckmann M, Johansen-Berg H, Rushworth MF (2009) Connectivity-based parcellation of human cingulate cortex and its relation to functional specialization. J Neurosci 29:1175-1190CrossRef Medline
5. Behrens TE, Woolrich MW, Walton ME, Rushworth MF (2007) Learning the value of information in an uncertain world. Nat Neurosci 10:1214-1221CrossRef Medline
6. Bickel WK, Pitcock JA, Yi R, Angtuaco EJ (2009) Congruence of BOLD response across intertemporal choice conditions: fictive and real money gains and losses. J Neurosci 29:8839-8846CrossRef Medline
7. Boorman ED, Behrens TE, Woolrich MW, Rushworth MF (2009) Howgreenis the grassonthe other side?Frontopolarcortexandtheevidencein favor of alternative courses of action. Neuron 62:733-743CrossRef Medline
8. Boorman ED, Behrens TE, Rushworth MF (2011) Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex. PLoS Biol 9:e1001093CrossRef Medline
9. Boorman ED, Rushworth MF, Behrens TE (2013) Ventromedial prefrontal and anterior cingulate cortex adopt choice and default reference frames during sequential multialternative choice. J Neurosci 33:2242-2253CrossRef Medline
10. Bray S, Rangel A, Shimojo S, Balleine B, O’Doherty JP (2008) The neural mechanisms underlying the influence of pavlovian cues on human decision making. J Neurosci 28:5861-5866CrossRef Medline
11. Burke CJ, Brünger C, Kahnt T, Park SQ, Tobler PN (2013) Neural integration of risk and effort costs by the frontal pole: only upon request. J Neurosci 33:1706-1713CrossRef Medline
12. Carmichael ST, Price JL (1995) Limbic connections of the orbital and medial prefrontal cortex in macaque monkeys. J Comp Neurol 363:615-641CrossRef Medline
13. Chau BK, Kolling N, Hunt LT, Walton ME, Rushworth MF (2014) A neural mechanism underlying failure of optimal choice with multiple alternatives. Nat Neurosci 17:463-470CrossRef Medline
14. Chiu PH, Lohrenz TM, Montague PR (2008) Smokers’ brains compute, but ignore, a fictive error signal in a sequential investment task. Nat Neurosci 11:514-520CrossRef Medline
15. Croxson PL, Walton ME, O’Reilly JX, Behrens TE, Rushworth MF (2009) Effort-based cost-benefit valuation and the human brain. J Neurosci 29: 4531-4541CrossRef Medline
16. Daw ND, O’Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441:876-879CrossRef Medline
17. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:1204-1215CrossRef Medline
18. Deichmann R, Gottfried JA, Hutton C, Turner R (2003) Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage 19:430-441CrossRef Medline
19. de Martino B, Kumaran D, Seymour B, Dolan RJ (2006) Frames, biases, and rational decision-making in the human brain. Science 313:684-687CrossRef Medline
20. Donoso M, Collins AG, Koechlin E (2014) Human cognition: foundations of human reasoning in the prefrontal cortex. Science 344:1481-1486CrossRef Medline
21. Friston KJ, Frith CD, Liddle PF, Dolan RJ, Lammertsma AA, Frackowiak RS (1990) The relationship between global and local changes in PET scans. J Cereb Blood Flow Metab 10:458-466CrossRef Medline
22. Geurts DE, Huys QJ, den Ouden HE, Cools R (2013) Aversive Pavlovian control of instrumental behavior in humans. J Cogn Neurosci 25:1428-1441CrossRef Medline
23. Jenkinson M, Beckmann CF, Behrens TE, Woolrich MW, Smith SM (2012) Fsl. Neuroimage 62:782-790CrossRef Medline
24. Kelly RE Jr, Alexopoulos GS, Wang Z, Gunning FM, Murphy CF, Morimoto SS, Kanellopoulos D, Jia Z, Lim KO, Hoptman MJ (2010) Visual inspection of independent components: defining a procedure for artifact removal from fMRI data. J Neurosci Methods 189:233-245CrossRef Medline
25. Kolling N, Behrens TE, Mars RB, Rushworth MF (2012) Neural mechanisms of foraging. Science 336:95-98CrossRef Medline
26. Kolling N, Wittmann M, Rushworth MF (2014) Multiple neural mechanisms of decision making and their competition under changing risk pressure. Neuron 81:1190-1202CrossRef Medline
27. Kurniawan IT, Guitart-Masip M, Dayan P, Dolan RJ (2013) Effort and valuation in the brain: the effects of anticipation and execution. J Neurosci 33:6160-6169CrossRef Medline
28. Lee MD, Wagenmakers E-J (2014) Bayesian cognitive modeling: a practical course. Cambridge: Cambridge UP. Lohrenz T, McCabe K, Camerer CF, Montague PR (2007) Neural signature of fictive learning signals in a sequential investment task. Proc Natl Acad Sci U S A 104:9493-9498CrossRef Medline
29. Mars RB, Jbabdi S, Sallet J, O’Reilly JX, Croxson PL, Olivier E, Noonan MP, Bergmann C, Mitchell AS, Baxter MG, Behrens TE, Johansen-Berg H, Tomassini V, Miller KL, Rushworth MF (2011) Diffusion-weighted imaging tractography-based parcellation of the human parietal cortex and comparison with human and macaque resting-state functional connectivity. J Neurosci 31:4087-4100CrossRef Medline
30. Neubert FX, Mars RB, Thomas AG, Sallet J, Rushworth MF (2014) Comparison of human ventral frontal cortex areas for cognitive control and language with areas in monkey frontal cortex. Neuron 81:700-713CrossRef Medline
31. Neubert FX, Mars RB, Sallet J, Rushworth MFS (2015) Connectivity profiles reveal relationship of brain areas for reward-guided learning and decision making in human and monkey frontal cortex. Proc Natl Acad Sci U S A 112:E2695-E2704CrossRef Medline
32. Nilsson H, Rieskamp J, Wagenmakers EJ (2011) Hierarchical Bayesian parameter estimation for cumulative prospect theory. J Math Psychol 55: 84-93CrossRef
33. Noonan MP, Kolling N, Walton ME, Rushworth MF (2012) Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement. Eur J Neurosci 35:997-1010CrossRef Medline
34. O’Reilly JX, Woolrich MW, Behrens TE, Smith SM, Johansen-Berg H (2012) Tools of the trade: psychophysiological interactions and functional connectivity. Soc Cogn Affect Neurosci 7:604-609CrossRef Medline
35. Palminteri S, Justo D, Jauffret C, Pavlicek B, Dauta A, Delmaire C, Czernecki V, Karachi C, Capelle L, Durr A, Pessiglione M (2012) Critical roles for anterior insula and dorsal striatum in punishment-based avoidance learning. Neuron 76:998-1009CrossRef Medline
36. Pessiglione M, Schmidt L, Draganski B, Kalisch R, Lau H, Dolan RJ, Frith CD (2007) How the brain translates money into force: a neuroimaging study of subliminal motivation. Science 316:904-906CrossRef Medline
37. Plummer M (2003) JAGS: a program for analysis of Bayesian graphical models using Gibbs sampling. In: Proceedings of the 3rd international workshop on distributed statistical computing, p 125. Vienna.
38. Preacher KJ, Hayes AF (2004) SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behav Res Methods Instrum Comput 36:717-731CrossRef Medline
39. Prévost C, Pessiglione M, Météreau E, Cléry-Melin ML, Dreher JC (2010) Separate valuation subsystems for delay and effort decision costs. J Neurosci 30:14080-14090CrossRef Medline
40. Prévost C, Liljeholm M, Tyszka JM, O’Doherty JP (2012) Neural correlates of specific and general Pavlovian-to-Instrumental Transfer within human amygdalar subregions: a high-resolution fMRI study. J Neurosci 32:8383-8390CrossRef Medline
41. Rangel A, Hare T (2010) Neural computations associated with goal-directed choice. Curr Opin Neurobiol 20:262-270CrossRef Medline
42. Rogers BP, Morgan VL, Newton AT, Gore JC (2007) Assessing functional connectivity in the human brain by fMRI. Magn Reson Imaging 25:1347-1357CrossRef Medline
43. Roiser JP, de Martino B, Tan GC, Kumaran D, Seymour B, Wood NW, Dolan RJ (2009) A genetically mediated bias in decision making driven by failure of amygdala control. J Neurosci 29:5985-5991CrossRef Medline
44. Rudebeck PH, Walton ME, Smyth AN, Bannerman DM, Rushworth MF (2006) Separate neural pathways process different decision costs. Nat Neurosci 9:1161-1168CrossRef Medline
45. Rushworth MF, Kolling N, Sallet J, Mars RB (2012) Valuation and decisionmaking in frontal cortex: one or many serial or parallel systems? Curr Opin Neurobiol 22:946-955CrossRef Medline
46. Salamone JD, Cousins MS, McCullough LD, Carriero DL, Berkowitz RJ (1994) Nucleus accumbens dopamine release increases during instrumental lever pressing for food but not free food consumption. Pharmacol Biochem Behav 49:25-31CrossRef Medline
47. Salamone JD, Correa M, Mingote S, Weber SM (2003) Nucleus accumbens dopamine and the regulation of effort in food-seeking behavior: implications for studies of natural motivation, psychiatry, and drug abuse. J Pharmacol Exp Ther 305:1-8CrossRef Medline
48. Sallet J, Mars RB, Noonan MP, Neubert FX, Jbabdi S, O’Reilly JX, Filippini N, Thomas AG, Rushworth MF (2013) The organization of dorsal frontal cortex in humans and macaques. J Neurosci 33:12255-12274CrossRef Medline
49. Scholl J, Günthner J, Kolling N, Favaron E, Rushworth MF, Harmer CJ, Reinecke A (2014) A role beyond learning for NMDA receptors in reward-based decisionmaking-a pharmacological study using d-cycloserine. Neuropsychopharmacology 39:2900-2909CrossRef Medline
50. Shiffrin RM, Lee MD, Kim W, Wagenmakers EJ (2008) A survey of model evaluation approaches with a tutorial on hierarchical bayesian methods. Cogn Sci 32:1248-1284CrossRef Medline
51. Skvortsova V, Palminteri S, Pessiglione M (2014) Learning to minimize efforts versus maximizing rewards: computational principles and neural correlates. J Neurosci 34:15621-15630CrossRef Medline
52. Smith SM (2002) Fast robust automated brain extraction. HumBrain Mapp 17:143-155CrossRef Medline
53. Smith SM, Jenkinson M, Woolrich MW, Beckmann CF, Behrens TE, Johansen-Berg H, Bannister PR, De Luca M, Drobnjak I, Flitney DE, Niazy RK, Saunders J, Vickers J, Zhang Y, De Stefano N, Brady JM, Matthews PM (2004) Advances in functional and structuralMRimage analysis and implementation as FSL. Neuroimage 23 [1]:S208-S219.
54. Spiegelhalter DJ, Best NG, Carlin BP, van der Linde A (2002) Bayesian measures of model complexity and fit. J R Statist Soc B 64:583-639
55. CrossRef Talmi D, Seymour B, Dayan P, Dolan RJ (2008) Human Pavlovian-instrumental transfer. J Neurosci 28:360-368CrossRef Medline
56. Tanaka SC, Balleine BW, O’Doherty JP (2008) Calculating consequences: brain systems that encode the causal effects of actions. J Neurosci 28: 6750-6755CrossRef Medline
57. Vickery TJ, Chun MM, Lee D (2011) Ubiquity and specificity of reinforcement signals throughout the human brain. Neuron 72:166-177CrossRef Medline
58. Walton ME, Bannerman DM, Rushworth MF (2002) Therole of rat medial frontal cortex in effort-based decision making. J Neurosci 22:10996-11003. Medline
59. Walton ME, Bannerman DM, Alterescu K, Rushworth MF (2003) Functional specialization within medial frontal cortex of the anterior cingulate for evaluating effort-related decisions. J Neurosci 23:6475-6479. Medline
60. Walton ME, Kennerley SW, Bannerman DM, Phillips PE, Rushworth MF (2006) Weighingupthe benefits of work: behavioralandneural analyses of effort-related decision making. Neural Netw 19:1302-1314CrossRef Medline
61. Woolrich MW, Ripley BD, Brady M, Smith SM (2001) Temporal autocorrelation in univariate linear modeling of fMRI data. Neuroimage 14: 1370-1386CrossRef Medline
62. Wunderlich K, Dayan P, Dolan RJ (2012) Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 15:786-791CrossRef Medline