Learning to minimize efforts versus maximizing rewards: Computational principles and neural correlates

Journal of Neuroscience

Volumn 34, Issue 47, 2014, Pages 15621-15630

  Skvortsova, Vasilisa ^a,b   Palminteri, Stefano ^a,b,c,d   Pessiglione, Mathias ^a,b  

^a Brain and Wellness Institute   (United States)

^b INSTITUT DU CERVEAU ET DE LA MOELLE ÉPINIÈRE   (France)

^c INSERM   (France)

^d ECOLE NORMALE SUPÉRIEURE   (France)

Author keywords

Computational modeling; Effort; Reinforcement learning; Reward; Ventromedial prefrontal cortex

Indexed keywords

ADULT; ANTERIOR CINGULATE; ANTERIOR INSULA; ARTICLE; BEHAVIOR CHANGE; BOLD SIGNAL; DECISION MAKING; FEMALE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; MOTOR CONTROL; NERVE CELL NETWORK; NEUROIMAGING; NORMAL HUMAN; POSTERIOR PARIETAL CORTEX; REINFORCEMENT; REWARD; TASK PERFORMANCE; VENTROMEDIAL PREFRONTAL CORTEX; ADOLESCENT; ASSOCIATION; BIOLOGICAL MODEL; COMPUTER SIMULATION; EXERCISE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; YOUNG ADULT;

ADOLESCENT; ADULT; CHOICE BEHAVIOR; COMPUTER SIMULATION; CUES; FEMALE; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; MODELS, NEUROLOGICAL; PHYSICAL EXERTION; REWARD; YOUNG ADULT;

EID: 84911425685     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.1350-14.2014     Document Type: Article

References (84)

1. Bartra O, McGuire JT, Kable JW (2013) The valuation system: a coordinate-based meta analysis of BOLD fMRI experiments examining neural correlates of subjective value. Neuroimage 76:412- 427. CrossRef Medline
2. Bautista LM, Tinbergen J, Kacelnik A (2001) To walk or to fly? How birds choose among foraging modes. Proc Natl Acad Sci U S A 98:1089 -1094. CrossRef Medline
3. Behrens TE, Woolrich MW, Walton ME, Rushworth MF (2007) Learning the value of information in an uncertain world. Nat Neurosci 10:1214- 1221. CrossRef Medline
4. Behrens TE, Hunt LT, Woolrich MW, Rushworth MF (2008) Associative learning of social value. Nature 456:245-249. CrossRef Medline
5. Berns GS, Laibson D, Loewenstein G (2007) Intertemporal choice-toward an integrative framework. Trends Cogn Sci 11:482- 488. CrossRef Medline
6. Bryden DW, Johnson EE, Tobia SC, Kashtelyan V, Roesch MR (2011) Attention for learning signals in anterior cingulate cortex. J Neurosci 31: 18266-18274. CrossRef Medline
7. Büchel C, Morris J, Dolan RJ, Friston KJ (1998) Brain systems mediating aversive conditioning: an event-related fMRI study. Neuron 20:947-957. CrossRef Medline
8. Burke CJ, Brünger C, Kahnt T, Park SQ, Tobler PN (2013) Neural integration of risk and effort costs by the frontal pole: only upon request. J Neurosci 33:1706 -1713. CrossRef Medline
9. Bush G, Vogt BA, Holmes J, Dale AM, Greve D, Jenike MA, Rosen BR (2002) Dorsal anterior cingulate cortex: a role in reward-based decision making. Proc Natl Acad Sci U S A 99:523-528. CrossRef Medline
10. Camille N, Griffiths CA, Vo K, Fellows LK, Kable JW (2011) Ventromedial frontal lobe damage disrupts value maximization in humans. J Neurosci 31:7527-7532. CrossRef Medline
11. Croxson PL, Walton ME, O'Reilly JX, Behrens TE, Rushworth MF (2009) Effort-based cost-benefit valuation and the human brain. J Neurosci 29: 4531-4541. CrossRef Medline
12. Daunizeau J, Adam V, Rigoux L (2014) VBA: a probabilistic treatment of nonlinear models for neurobiological and behavioural data. PLoS Comput Biol 10:e1003441. CrossRef Medline
13. Dayan P, BalleineBW (2002) Reward, motivation and reinforcement learning. Neuron 36:285-298. CrossRef
14. Deichmann R, Gottfried JA, Hutton C, Turner R (2003) Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage 19:430-441. CrossRef Medline
15. den Ouden HE, Friston KJ, Daw ND, McIntosh AR, Stephan KE (2009) A dual role for prediction error in associative learning. Cereb Cortex 19: 1175-1185. CrossRef Medline
16. Doll BB, Simon DA, Daw ND (2012) The ubiquity of model-based reinforcement learning. Curr Opin Neurobiol 22:1075-1081. CrossRef Medline
17. Doya K (2002) Metalearning and neuromodulation. Neural Netw 15:495- 506. CrossRef Medline
18. Frank MJ, Seeberger LC, O'reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306:1940 -1943. CrossRef Medline
19. Friston K (2005) A theory of cortical responses. Philos Trans R Soc Lond B Biol Sci 360:815- 836. CrossRef Medline
20. Friston KJ, Stephan KE (2007) Free-energy and the brain. Synthese 159: 417-458. CrossRef Medline
21. Galvan A, Hare TA, Davidson M, Spicer J, Glover G, Casey BJ (2005) The role of ventral frontostriatal circuitry in reward-based learning in humans. J Neurosci 25:8650-8656. CrossRef Medline
22. Gan JO, Walton ME, Phillips PE (2010) Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat Neurosci 13:25-27. CrossRef Medline
23. Gershman SJ, Pesaran B, Daw ND (2009) Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J Neurosci 29:13524 -13531. CrossRef Medline
24. Green L, Fristoe N, Myerson J (1994) Temporal discounting and preference reversals in choice between delayed outcomes. Psychon Bull Rev 1:383- 389. CrossRef Medline
25. Haber SN, Knutson B (2010) The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35:4 -26. CrossRef Medline
26. Hampton AN, Bossaerts P, O'Doherty JP (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 26:8360-8367. CrossRef Medline
27. Hare TA, O'Doherty J, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 28:5623-5630. CrossRef Medline
28. Hare TA, Schultz W, Camerer CF, O'Doherty JP, Rangel A (2011) Transformation of stimulus value signals into motor commands during simple choice. Proc Natl Acad Sci U S A 108:18120 -18125. CrossRef Medline
29. Hartmann MN, Hager OM, Tobler PN, Kaiser S (2013) Parabolic discounting of monetary rewards by physical effort. Behav Processes 100:192-196. CrossRef Medline
30. Hayden BY, Heilbronner SR, Pearson JM, Platt ML (2011) Surprise signals in anterior cingulate cortex: neuronal encoding of unsigned reward prediction errors driving adjustment behavior. J Neurosci 31:4178-4187. CrossRef Medline
31. Kable JW, Glimcher PW (2007) The neural correlates of subjective value during intertemporal choice. Nat Neurosci 10:1625-1633. CrossRef Medline
32. Kalenscher T, Pennartz CM (2008) Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making. Prog Neurobiol 84:284 -315. CrossRef Medline
33. Khamassi M, Enel P, Dominey PF, Procyk E (2013) Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters. Prog Brain Res 202:441- 464. CrossRef Medline
34. Klavir O, Genud-Gabai R, Paz R (2013) Functional connectivity between amygdala and cingulate cortex for adaptive aversive learning. Neuron 80:1290 -1300. CrossRef Medline
35. Kurniawan IT, Guitart-Masip M, Dayan P, Dolan RJ (2013) Effort and valuation in the brain: the effects of anticipation and execution. J Neurosci 33:6160-6169. CrossRef Medline
36. Law CT, Gold JI (2009) Reinforcement learning can account for associative and perceptual learning on a visual-decision task. Nat Neurosci 12:655- 663. CrossRef Medline
37. Levy DJ, Glimcher PW (2012) The root of all value: a neural common currency for choice. Curr Opin Neurobiol 22:1027-1038. CrossRef Medline
38. Loewenstein G, Prelec D (1992) Anomalies in intertemporal choice: evidence and interpretation. Q J Econ 107:573-597. CrossRef
39. Madlon-Kay S, Pesaran B, Daw ND (2013) Action selection in multieffector decision making. Neuroimage 70:66 -79. CrossRef Medline
40. Mathys C, Daunizeau J, Friston KJ, Stephan KE (2011) A bayesian foundation for individual learning under uncertainty. FrontHumNeurosci 5:39. CrossRef Medline
41. Meyniel F, Sergent C, Rigoux L, Daunizeau J, Pessiglione M (2013) Neurocomputational account of how the human brain decides when to have a break. Proc Natl Acad Sci U S A 110:2641-2646. CrossRef Medline
42. Minamimoto T, Hori Y, Richmond BJ (2012) Is working more costly than waiting in monkeys? PLoS One 7:e48434. CrossRef Medline
43. Nassar MR, Wilson RC, Heasly B, Gold JI (2010) An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J Neurosci 30:12366 -12378. CrossRef Medline
44. Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005) Valid conjunction inference with the minimum statistic. Neuroimage 25:653- 660. CrossRef Medline
45. Nitschke JB, Dixon GE, Sarinopoulos I, Short SJ, Cohen JD, Smith EE, Kosslyn SM, Rose RM, Davidson RJ (2006) Altering expectancy dampens neural response to aversive taste in primary taste cortex. Nat Neurosci 9:435- 442. CrossRef Medline
46. Noonan MP, Kolling N, Walton ME, Rushworth MF (2012) Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement. Eur J Neurosci 35:997-1010. CrossRef Medline
47. O' Doherty J, Kringelbach ML, Rolls ET, Hornak J, Andrews C (2001) Abstract reward and punishment representations in the human orbitofrontal cortex. Nat Neurosci 4:95-102. CrossRef Medline
48. Pagnoni G, Zink CF, Montague PR, Berns GS (2002) Activity in human ventral striatum locked to errors of reward prediction. Nat Neurosci 5:97- 98. CrossRef Medline
49. Palminteri S, Boraud T, Lafargue G, Dubois B, Pessiglione M (2009) Brain hemispheres selectively track the expected value of contralateral options. J Neurosci 29:13465-13472. CrossRef Medline
50. Palminteri S, Justo D, Jauffret C, Pavlicek B, Dauta A, Delmaire C, Czernecki V, Karachi C, Capelle L, Durr A, Pessiglione M (2012) Critical roles for anterior insula and dorsal striatum in punishment-based avoidance learning. Neuron 76:998 -1009. CrossRef Medline
51. Pasquereau B, Turner RS (2013) Limited encoding of effort by dopamine neurons in a cost-benefit trade-off task. J Neurosci 33:8288-8300. CrossRef Medline
52. Pearce JM, Hall G (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stumuli. Psychol Rev 87:532-552. CrossRef Medline
53. Penny WD (2012) Comparing dynamic causal models using AIC, BIC and free energy. Neuroimage 59:319 -330. CrossRef Medline
54. PennyWD, Stephan KE, Daunizeau J, Rosa MJ, Friston KJ, Schofield TM, Leff AP (2010) Comparing families of dynamic causal models. PLoS Comput Biol 6:e1000709. CrossRef Medline
55. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042-1045. CrossRef Medline
56. Peters J, Büchel C (2009) Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making. J Neurosci 29:15727-15734. CrossRef Medline
57. Peters J, Büchel C (2010) Neural representations of subjective reward value. Behav Brain Res 213:135-141. CrossRef Medline
58. Peters J, Büchel C (2011) The neural mechanisms of inter-temporal decision- making: understanding variability. Trends Cogn Sci 15:227-239. CrossRef Medline
59. Plassmann H, O'Doherty JP, Rangel A (2010) Appetitive and aversive goal values are encoded in the medial orbitofrontal cortex at the time of decision making. J Neurosci 30:10799 -10808. CrossRef Medline
60. Preuschoff K, Quartz SR, Bossaerts P (2008) Human insula activation reflects risk prediction errors as well as risk. J Neurosci 28:2745-2752. CrossRef Medline
61. Prévost C, Pessiglione M, Météreau E, Cléry-Melin ML, Dreher JC (2010) Separate valuation subsystems for delay and effort decision costs. J Neurosci 30:14080 -14090. CrossRef Medline
62. Rangel A, Camerer C, Montague PR (2008) A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci 9:545- 556. CrossRef Medline
63. Rao RP, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2:79-87. CrossRef Medline
64. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical conditioning II: current research and theory (Black AH, Prokasy WF, ed), pp 64-99. New York: Appleton Century Crofts.
65. Rigoux L, Stephan KE, Friston KJ, Daunizeau J (2014) Bayesian model selection for group studies-revisited. Neuroimage 84:971-985. CrossRef Medline
66. Roesch MR, Taylor AR, Schoenbaum G (2006) Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51:509-520. CrossRef Medline
67. Rudebeck PH, Walton ME, Smyth AN, Bannerman DM, Rushworth MF (2006) Separate neural pathways process different decision costs. Nat Neurosci 9:1161-1168. CrossRef Medline
68. Rudebeck PH, Mitz AR, Chacko RV, Murray EA (2013) Effects of amygdala lesions on reward-value coding in orbital and medial prefrontal cortex. Neuron 80:1519 -1531. CrossRef Medline
69. Rutledge RB, Dean M, Caplin A, Glimcher PW (2010) Testing the reward prediction error hypothesis with an axiomatic model. J Neurosci 30: 13525-13536. CrossRef Medline
70. Samanez-Larkin GR, Hollon NG, Carstensen LL, Knutson B (2008) Individual differences in insular sensitivity during loss anticipation predict avoidance learning. Psychol Sci 19:320 -323. CrossRef Medline
71. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599. CrossRef Medline
72. Seymour B, Daw N, Dayan P, Singer T, Dolan R (2007) Differential encoding of losses and gains in the human striatum. J Neurosci 27:4826-4831. CrossRef Medline
73. SpratlingMW (2010) Predictive coding as a model of response properties in cortical area V1. J Neurosci 30:3531-3543. CrossRef Medline
74. Stephan KE, PennyWD,Daunizeau J, Moran RJ, Friston KJ (2009) Bayesian model selection for group studies. Neuroimage 46:1004 -1017. CrossRef Medline
75. Sutton RS, BartoAG (1998) Reinforcement learning. Cambridge, MA: MIT.
76. Talmi D, Dayan P, Kiebel SJ, Frith CD, Dolan RJ (2009) How humans integrate the prospects of pain and reward during choice. J Neurosci 29: 14617-14626. CrossRef Medline
77. Trimmer PC, McNamara JM, Houston AI, Marshall JA. R (2012) Does natural selection favour the Rescorla-Wagner rule? J Theor Biol 302:39 -52. CrossRef Medline
78. Walton ME, Bannerman DM, Alterescu K, Rushworth MF (2003) Functional specialization within medial frontal cortex of the anterior cingulate for evaluating effort-related decisions. J Neurosci 23:6475- 6479. Medline
79. Walton ME, Kennerley SW, Bannerman DM, Phillips PE, Rushworth MF (2006) Weighing up the benefits of work: behavioral and neural analyses of effort-related decision making. Neural Netw 19:1302-1314. CrossRef Medline
80. Walton ME, Croxson PL, Rushworth MF, Bannerman DM (2005) The mesocortical dopamine projection to anterior cingulate cortex plays no role in guiding effort-related decisions. Behav Neurosci 119:323-328. CrossRef Medline
81. Weiskopf N, Hutton C, Josephs O, Deichmann R (2006) Optimal EPI parameters for reduction of susceptibility-induced BOLD sensitivity losses: a whole-brain analysis at 3 T and 1.5 T. Neuroimage 33:493-504. CrossRef Medline
82. Wilson RC, Nassar MR, Gold JI (2013) A mixture of delta-rules approximation to bayesian inference in change-point problems. PLoS Comput Biol 9:e1003150. CrossRef Medline
83. Wunderlich K, Rangel A, O'Doherty JP (2010) Economic choices can be made using only stimulus values. Proc Natl Acad Sci U S A 107:15005- 15010. CrossRef Medline
84. Zhu L, Mathewson KE, HsuM (2012) Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning. Proc Natl Acad Sci U S A 109:1419 -1424. CrossRef Medline