Multiple Systems for Value Learning

Neuroeconomics: Decision Making and the Brain: Second Edition

Volumn , Issue , 2013, Pages 393-410

  Daw, Nathaniel D ^a   O'Doherty, John P ^b  

^a NEW YORK UNIVERSITY   (United States)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Goal directed; Habitual; Model based learning; Model free learning; Pavlovian; Reinforcement learning

Indexed keywords

EID: 84885024210     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-12-416008-8.00021-8     Document Type: Chapter

References (110)

1. Adams C. Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q. J. Exp. Psychol. Sect. B 1982, 34(2):77-98.
2. Aggarwal M., Hyland B.I., Wickens J.R. Neural control of dopamine neurotransmission: implications for reinforcement learning. Eur. J. Neurosci. 2012, 35(7):1115-1123.
3. Alexander G.E., Crutcher M.D. Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci. 1990, 13(7):266-271.
4. Balleine B.W., Daw N.D., O'Doherty J.P. Multiple forms of value learning and the function of dopamine. Neuroeconomics: Decision Making and the Brain 2008, Academic Press, New York. P.W. Glimcher, C.F. Camerer, R.A. Poldrack, E. Fehr (Eds.).
5. Balleine B.W., Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37(4-5):407-419.
6. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010, 35(1):48-69.
7. Barto A.G. Adaptive critics and the basal ganglia. Models of Information Processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
8. Beierholm U.R., Anen C., Quartz S., Bossaerts P. Separate encoding of model-based and model-free valuations in the human brain. NeuroImage 2011, 58(3):955-962.
9. Berg H.C., Brown D.A. Chemotaxis in escherichia coli analysed by three-dimensional tracking. Nature 1972, 239(5374):500-504.
10. Bindra D. A Theory of Intelligent Behavior 1976, Wiley-Interscience, Oxford, England.
11. Bornstein A.M., Daw N.D. Dissociating hippocampal and striatal contributions to sequential prediction learning. European J. Neurosci. 2012, 35(7):1011-1023.
12. Bray S., Rangel A., Shimojo S., Balleine B., O'Doherty J.P. The neural mechanisms underlying the influence of Pavlovian cues on human decision making. J. Neurosci. 2008, 28(22):5861-5866.
13. Breland K., Breland M. The misbehavior of organisms. Am. psychol. 1961, 16(11):681.
14. Brembs B., Lorenzetti F.D., Reyes F.D., Baxter D.A., Byrne J.H. Operant reward learning in Aplysia: neuronal correlates and mechanisms. Science 2002, 296(5573):1706-1709.
15. Cador M., Robbins T., Everitt B. Involvement of the amygdala in stimulus-reward associations: interaction with the ventral striatum. Neuroscience 1989, 30(1):77-86.
16. Chib V.S., De Martino B., Shimojo S., O'Doherty J.P. Neural mechanisms underlying paradoxical performance for monetary incentives are driven by loss aversion. Neuron 2012, 74(3):582-594.
17. Cohen N.J., Eichenbaum H. Memory, amnesia, and the hippocampal system 1993, MIT press, Cambridge, MA.
18. Cook D.G., Carew T.J. Operant conditioning of head waving in Aplysia. Proc. Natl. Acad. Sci. U.S.A. 1986, 83(4):1120-1124.
19. Corbit L.H., Balleine B.W. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 2003, 146(1-2):145-157.
20. Corbit L.H., Muir J.L., Balleine B.W. The role of the nucleus accumbens in instrumental conditioning: Evidence of a functional dissociation between accumbens core and shell. J. Neurosci. 2001, 21(9):3251-3260.
21. Corbit L.H., Ostlund S.B., Balleine B.W. Sensitivity to instrumental contingency degradation is mediated by the entorhinal cortex and its efferents via the dorsal hippocampus. J. Neurosci. 2002, 22(24):10976-10984.
22. Damasio A.R. Descartes' Error: Emotion, Rationality and the Human Brain 1994, Putnam, New York.
23. Davey G.C. Classical conditioning and the acquisition of human fears and phobias: A review and synthesis of the literature. Adv. Behav. Res. Ther. 1992, 14(1):29-66.
24. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69(6):1204-1215.
25. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8(12):1704-1711.
26. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Netw. 2006, 19(8):1153-1160.
27. De Martino B., Kumaran D., Holt B., Dolan R.J. The neurobiology of reference-dependent value computation. J. Neurosci. 2009, 29(12):3833-3842.
28. De Martino B., Kumaran D., Seymour B., Dolan R.J. Frames, biases, and rational decision-making in the human brain. Science 2006, 313(5787):684-687.
29. de Wit S., Corlett P.R., Aitken M.R., Dickinson A., Fletcher P.C. Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans. J. Neurosci. 2009, 29(36):11330-11338.
30. de Wit S., Watson P., Harsay H.A., Cohen M.X., van de Vijver I., Ridderinkhof K.R. Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. J. Neurosci. 2012, 32(35):12066-12075.
31. Dickinson A. Actions and habits: the development of a behavioural autonomy. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1985, 308:67-78.
32. Dickinson A. Bidirectional instrumental conditioning. Q. J. Exp. Psychol. Sect. B 1996, 49(4):289-306.
33. Everitt B.J., Robbins T.W. Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 2005, 8(11):1481-1489.
34. Faure A., Haberland U., Conde F., El Massioui N. Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J. Neurosci. 2005, 25:2771-2780.
35. Freud S. The Ego and the Id 1961, vol. 19. Hogarth Press, London.
36. Gillan C.M., Papmeyer M., Morein-Zamir S., et al. Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. Am. J. Psychiatry 2011, 168(7):718-726.
37. Glascher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66(4):585-595.
38. Gottfried J.A., O'Doherty J., Dolan R.J. Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging. J. Neurosci. 2002, 22(24):10829-10837.
39. Gottfried J.A., O'Doherty J., Dolan R.J. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 2003, 301(5636):1104-1107.
40. Groenewegen H.J., Russchen F.T. Organization of the efferent projections of the nucleus accumbens to pallidal, hypothalamic, and mesencephalic structures: a tracing and immunohistochemical study in the cat. J. Comp. Neurol. 1984, 223(3):347-367.
41. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 2006, 26(32):8360-8367.
42. Hampton A.N., Bossaerts P., O'Doherty J.P. Neural correlates of mentalizing-related computations during strategic interactions in humans. Proc. Natl. Acad. Sci. U.S.A. 2008, 105(18):6741-6746.
43. Hartley T., Burgess N. Complementary memory systems: competition, cooperation and compensation. Trends Neurosci. 2005, 28(4):169-170.
44. Hatfield T., Han J.S., Conley M., Gallagher M., Holland P. Neurotoxic lesions of basolateral, but not central, amygdala interfere with Pavlovian second-order conditioning and reinforcer devaluation effects. J. Neurosci. 1996, 16(16):5256-5265.
45. Hershberger W.A. An approach through the looking glass. Learn. Behav. 1986, 14(4):443-451.
46. Holland P.C. CS-US interval as a determinant of the form of Pavlovian appetitive conditioned responses. J. Exp. Psychol. Anim. Behav. Process. 1980, 6(2):155-174.
47. Holland P.C. Relations between Pavlovian-instrumental transfer and reinforcer devaluation. J. Exp. Psychol. Animal Behav. Proc. 2004, 30:104-117.
48. Hull C. Principles of Behavior 1943, Appleton-Century-Crofts, New York, NY.
49. James W. The Principles of Psychology 1950, vol. 1. Dover, New York.
50. Johnson A., Redish A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 2007, 27(45):12176-12189.
51. Jueptner M., Frith C.D., Brooks D.J., Frackowiak R.S., Passingham R.E. Anatomy of motor learning. II. Subcortical structures and learning by trial and error. J. Neurophysiol. 1997, 77:1325-1337.
52. Kable J.W., Glimcher P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 2007, 10(12):1625-1633.
53. Kahneman D. A perspective on judgment and choice. Am. Psychol. 2003, 58:697-720.
54. Keramati M., Dezfouli A., Piray P. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 2011, 7(5):e1002055.
55. Killcross S., Coutureau E. Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 2003, 13(4):400-408.
56. Killcross S., Robbins T.W., Everitt B.J. Different types of fear-conditioned behaviour mediated by separate nuclei within amygdala. Nature 1997, 388(6640):377-380.
57. Konorski J. Conditioned Reflexes and Neuron Organization 1948, Cambridge University Press, Cambridge.
58. Laibson D. Golden eggs and hyperbolic discounting. Q. J. Econ. 1997, 112(2):443-477.
59. Lehéricy S., Benali H., Van de Moortele P.F., Pélégrini-Issac M., Waechter T., Ugurbil K. Distinct basal ganglia territories are engaged in early and advanced motor sequence learning. PNAS 2005, 102:12566-12571.
60. Lengyel M., Dayan P. Hippocampal contributions to control: the third way. Adv. Neural Inf. Process. Syst. 2007, 20:889-896.
61. Liljeholm M., Tricomi E., O'Doherty J.P., Balleine B.W. Neural correlates of instrumental contingency learning: differential effects of action-reward conjunction and disjunction. J. Neurosci. 2011, 31(7):2474-2480.
62. Loewenstein G. Out of control: visceral influences on behavior. Organ. Behav. Hum. Decis. Process. 1996, 65(3):272-292.
63. Loewenstein G., O'Donoghue T., 2004. Animal spirits: affective and deliberative processes in economic behavior. Working Paper 04-14, Center for Analytic Economics, Cornell University.
64. McClure S.M., Laibson D.I., Loewenstein G., Cohen J.D. Separate neural systems value immediate and delayed monetary rewards. Science 2004, 306(5695):503-507.
65. Mischel W. Processes in delay of gratification. Advances in Experimental Social Psychology 1974, vol. 7. Academic Press, New York.
66. Mobbs D., Hassabis D., Seymour B., et al. Choking on the money: reward-based performance decrements are associated with midbrain activity. Psychol. Sci. 2009, 20(8):955-962.
67. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16(5):1936-1947.
68. Morrison S.E., Salzman C.D. Representations of appetitive and aversive information in the primate orbitofrontal cortex. Ann. N. Y. Acad. Sci. 2011, 1239:59-70.
69. Mowrer O.H. On the dual nature of learning-a re-interpretation of "conditioning" and "problem-solving. Harv. Educ. Rev. 1947, 17:102-148.
70. Norman D.A., Shallice T. Attention to action: Willed and Automatic control of behaviour. Consciousness and Self-regulation 1986, Plenum Press, New York, NY. G.E. Schwartz, D. Shapiro (Eds.).
71. O'Doherty J., Deichmann R., Critchley H.D., Dolan R.J. Neural responses during anticipation of a primary taste reward. Neuron 2002, 33(5):815-826.
72. O'Keefe J., Nadel L. The Hippocampus as a Cognitive Map 1978, Clarendon, Oxford.
73. Ostlund S.B., Balleine B.W. Lesions of medial prefrontal cortex disrupt the acquisition but not the expression of goal-directed learning. J. Neurosci. 2005, 25(34):7763-7770.
74. Ostlund S.B., Balleine B.W. Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental conditioning. J. Neurosci. 2007, 27(18):4819-4825.
75. Pare D., Quirk G.J., Ledoux J.E. New vistas on amygdala networks in conditioned fear. J. Neurophysiol. 2004, 92(1):1-9.
76. Parkinson J., Dalley J., Cardinal R., et al. Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function. Behav. Brain Res. 2002, 137(1-2):149-163.
77. Parkinson J.A., Olmstead M.C., Burns L.H., Robbins T.W., Everitt B.J. Dissociation in effects of lesions of the nucleus accumbens core and shell on appetitive Pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by D-amphetamine. J. Neurosci. 1999, 19(6):2401-2411.
78. Paton J.J., Belova M.A., Morrison S.E., Salzman C.D. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 2006, 439(7078):865-870.
79. Pavlov I.P. Conditioned Reflexes 1927, Oxford University Press, Oxford.
80. Plato Phaedrus 1995, Hackett Publishing Company, Indianapolis, Indiana, ca. 428-347 B.C.
81. Prevost C., Liljeholm M., Tyszka J.M., O'Doherty J.P. Neural correlates of specific and general Pavlovian-to-Instrumental Transfer within human amygdalar subregions: a high-resolution fMRI study. J. Neurosci. 2012, 32(24):8383-8390.
82. Price J.L., Amaral D.G. An autoradiographic study of the projections of the central nucleus of the monkey amygdala. J. Neurosci. 1981, 1(11):1242-1259.
83. Redish A.D., Jensen S., Johnson A. A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci. 2008, 31(4):415-437. (discussion 437-487).
84. Reynolds J.N., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 2002, 15(4-6):507-521.
85. Schacter D., Addis D., Buckner R. Remembering the past to imagine the future: the prospective brain. Nat. Rev. Neurosci. 2007, 8(9):657-661.
86. Schneider W., Shiffrin R.M. Controlled and automatic human information processing: I. Detection, search, and attention. Psychol. Rev. 1977, 84(1):1-66.
87. Schoenbaum G., Chiba A.A., Gallagher M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat. Neurosci. 1998, 1(2):155-159.
88. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275(5306):1593-1599.
89. Sheffield F.D. Relation betwen classical and instrumental conditioning. Classical Conditioning 1965, 302-322. Appleton-Century-Crofts, New York, NY. W.F. Prokasy (Ed.).
90. Sherrington C. The Integrative Action of the Nervous System 1906, Yale University Press, New Haven.
91. Simon D.A., Daw N.D. Environmental statistics and the trade-off between model-based and TD learning in humans. Adv. Neural Inf. Process. Syst. 2011, 24.
92. Simon D.A., Daw N.D. Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci. 2011, 31(14):5526-5539.
93. Skinner B.F. Two "synthetic social relations". J. Exp. Anal. Behav. 1962, 5(4):531-533.
94. Squire L.R. Declarative and nondeclarative memory: multiple brain systems supporting learning and memory. J. Cogn. Neurosci. 1992, 4(3):232-243.
95. Sutton R.S. Learning to predict by the methods of temporal differences. Mach. Learn. 1988, 3(1):9-44.
96. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press.
97. Talmi D., Seymour B., Dayan P., Dolan R.J. Human Pavlovian-instrumental transfer. J. Neurosci. 2008, 28(2):360-368.
98. Tanaka S.C., Balleine B.W., O'Doherty J.P. Calculating consequences: brain systems that encode the causal effects of actions. J. Neurosci. 2008, 28(26):6750-6755.
99. Thaler R.H., An H.M.S. Economic Theory of Self Control. J. Polit. Econ. 1981, 89(2):392-406.
100. Thorndike, E.L., 1898. Animal Intelligence: An Experimental Study of the Associative Processes in Animals. Vol Monograph Supplements, No. 8. Macmillan, New York.
101. Tolman E. Cognitive maps in rats and men. Psychol. Rev. 1948, 55(4):189-208.
102. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 2009, 29:2225-2232.
103. Tully T., Quinn W.G. Classical conditioning and retention in normal and mutant Drosophila melanogaster. J. Comp. Physiol. A 1985, 157(2):263-277.
104. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J. Neurosci. 2007, 27(15):4019-4026.
105. Walters E.T., Carew T.J., Kandel E.R. Associative learning in aplysia: evidence for conditioned fear in an invertebrate. Science 1981, 211(4481):504-506.
106. Wang, L.P., Li, F., Wang, D., Xie,K., Shen, X., Tsien, J.Z., 2011. NMDA receptors in dopaminergic neurons are crucial for habit learning. Neuron. 72 (6), 1055-1066.
107. Weber E.U., Johnson E.J. Mindful judgment and decision making. Annu. Rev. Psychol. 2009, 60:53-85.
108. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 15(5):786-791.
109. Yin HH, Knowlton BJ, Balleine BW. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 2004, 19(1):181-189.
110. Yin H.H., Ostlund S.B., Knowlton B.J., Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 2005, 22(2):513-523.