Models of Value and Choice

Neuroscience of Preference and Choice

Volumn , Issue , 2012, Pages 33-52

  Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Model based control; Model free control; Motivation; Pavlovian control; Reinforcement learning; Utility

Indexed keywords

EID: 84882459170     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-12-381431-9.00002-4     Document Type: Chapter

References (89)

1. Ainslie G. Breakdown of will 2001, Cambridge University Press, Cambridge, UK.
2. Alexander G.E., Crutcher M.D. Functional architecture of basal ganglia circuits: Neural substrates of parallel processing. Trends in Neurosciences 1990, 13(7):266-271.
3. Balleine B.W. Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits. Physiology & Behavior 2005, 86(5):717-730.
4. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: Corti-costriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010, 35(1):48-69.
5. Barto A. Adaptive critics and the basal ganglia. Models of information processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge MA. J. Houk, J. Davis, D. Beiser (Eds.).
6. Barto A., Sutton R., Anderson C. Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics 1983, 13(5):834-846.
7. Barto, A., Singh, S., & Chentanez, N. (2004). Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of international conference of developmental learning, San Diego, CA.
8. Berridge K.C. Motivation concepts in behavioral neuroscience. Physiology & Behavior 2004, 81:179-209.
9. Berridge K.C. The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology (Berl) 2007, 191(3):391-431.
10. Berridge K.C., Robinson T.E., Aldridge J.W. Dissecting components of reward: "Liking", "wanting", and learning. Current Opinion in Pharmacology 2009, 9(1):65-73.
11. Blanchard D.C., Blanchard R.J. Ethoexperimental approaches to the biology of emotion. Annual Review of Psychology 1988, 39:43-68.
12. Bogacz R., Brown E., Moehlis J., Holmes P., Cohen J.D. The physics of optimal decision making: A formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review 2006, 113(4):700-765.
13. Bolles R.C. Species-specific defense reactions and avoidance learning. Psychological Review 1970, 77:32-48.
14. Boureau Y.-L., Dayan P. Opponency revisited: Competition and cooperation between dopamine and serotonin. Neuropsychopharmacology 2011, 36:74-97.
15. Bushong B., King L., Camerer C., Rangel A. Pavlovian processes in consumer choice: The physical presence of a good increases willingness-to-pay. American Economic Review 2010, 100:1-18.
16. Cardinal R.N., Parkinson J.A., Hall J., Everitt B.J. Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience and Biobehavioral Reviews 2002, 26(3):321-352.
17. Correia S.P.C., Dickinson A., Clayton N.S. Western scrub-jays anticipate future needs independently of their current motivational state. Current Biology 2007, 17(10):856-861.
18. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69(6):1204-1215.
19. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 2005, 8(12):1704-1711.
20. Dayan P., Niv Y., Seymour B., Daw N.D. The misbehavior of value and the discipline of the will. Neural Networks 2006, 19(8):1153-1160.
21. Dayan P. The role of value systems in decision-making. Better than conscious: Decision making, the human mind, and implications for institutions 2008, 51-70. MIT Press, Ernst Strüngmann Forum, Cambridge, MA. C. Engel, W. Singer (Eds.).
22. Dayan P., Daw N.D. Decision theory, reinforcement learning, and the brain. Cognitive, Affective & Behavioral Neuroscience 2008, 8(4):429-453.
23. Dayan P., Huys Q.J.M. Serotonin, inhibition, and negative mood. PLoS Computational Biology 2008, 4(2):e4.
24. Dayan P., Seymour B. Values and actions in aversion. Neuroeconomics: Decision making and the brain 2008, 175-191. Academic Press, New York, NY. P. Glimcher, C. Camerer, R. Poldrack, E. Fehr (Eds.).
25. Dayan P., Huys Q.J.M. Serotonin in affective control. Annual Review of Neuroscience 2009, 32:95-126.
26. De Martino B., Kumaran D., Seymour B., Dolan R.J. Frames, biases, and rational decision-making in the human brain. Science 2006, 313(5787):684-687.
27. Dearden, R., Friedman, N., & Russell, S. (1998). Bayesian Q-learning. In Proceedings of the fiteenth National/tenth Conference on Artificial intelligence/Innovative Applications of Artificial Intelligence Table of Contents, pp. 761-768. Menlo Park, CA: American Association for Artificial Intelligence.
28. Dickinson A. Contemporary animal learning theory 1980, Cambridge University Press, Cambridge, UK.
29. Dickinson A., Balleine B. The role of learning in the operation of motivational systems. Stevens' handbook of experimental psychology 2002, Vol. 3:497-534. Wiley, New York, NY.
30. Elster J. Ulysses and the sirens: Studies in rationality and irrationality 1979, Cambridge University Press, Cambridge, UK.
31. Faure A., Reynolds S.M., Richard J.M., Berridge K.C. Mesolimbic dopamine in desire and dread: Enabling motivation to be generated by localized glutamate disruptions in nucleus accumbens. Journal of Neuroscience 2008, 28(28):7184-7192.
32. Foster D.J., Wilson M.A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 2006, 440(7084):680-683.
33. Frank M.J., Seeberger L.C., O'Reilly R.C. By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science 2004, 306(5703):1940-1943.
34. Frank M.J., Claus E.D. Anatomy of a decision: Striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review 2006, 113(2):300-326.
35. Fudenberg D., Tirole J. Game theory 1994, MIT Press, Cambridge, MA.
36. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66(4):585-595.
37. Gupta A.S., van der Meer M.A.A., Touretzky D.S., Redish A.D. Hippocampal replay is not a simple function of experience. Neuron 2010, 65(5):695-705.
38. Hazy T.E., Frank M.J., O'Reilly R.C. Towards an executive without a homunculus: Computational models of the prefrontal cortex/basal ganglia system. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 2007, 362(1485):1601-1613.
39. Hershberger W.A. An approach through the looking-glass. Animal Learning & Behavior 1986, 14:443-451.
40. Hertwig R., Barron G., Weber E.U., Erev I. Decisions from experience and the effect of rare events in risky choice. Psychological Science 2004, 15(8):534-539.
41. Hertwig R., Erev I. The description-experience gap in risky choice. Trends in Cognitive Sciences 2009, 13:517-523.
42. Holland P.C. Relations between Pavlovian-instrumental transfer and reinforcer devaluation. Journal of Experimental Psychology. Animal Behavior Processes 2004, 30(2):104-117.
43. Huys Q.J.M., Dayan P. A Bayesian formulation of behavioral control. Cognition 2009, 113:314-328.
44. Kahneman D. A perspective on judgment and choice: Mapping bounded rationality. American Psychologist 2003, 58(9):697-720.
45. Kahneman D., Frederick S. Frames and brains: Elicitation and control of response tendencies. Trends in Cognitive Sciences 2007, 11:45-46.
46. Killcross S., Coutureau E. Coordination of actions and habits in the medial prefrontal cortex of rats. Cerebral Cortex 2003, 13(4):400-408.
47. Lengyel M., Dayan P. Hippocampal contributions to control: A normative perspective. Computational and Systems Neuroscience 2007.
48. Mackintosh N.J. Conditioning and associative learning 1983, Oxford University Press, Oxford, UK.
49. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of Neuroscience 1996, 16(5):1936-1947.
50. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience 2006, 9(8):1057-1063.
51. Morrison S.E., Salzman C.D. Re-valuing the amygdala. Current Opinion in Neurobiology 2010, 20(2):221-230.
52. Niv Y. Reinforcement learning in the brain. Journal of Mathematical Psychology 2009, 53(3):139-154.
53. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berl) 2007, 191(3):507-520.
54. O'Doherty J.P. Reward representations and reward-related learning in the human brain: Insights from neuroimaging. Current Opinion in Neurobiology 2004, 14(6):769-776.
55. O'Doherty J.P. Lights, camembert, action! The role of human orbitofrontal cortex in encoding stimuli, rewards, and choices. Annals of the New York Academy of Sciences, USA 2007, 1121:254-272.
56. Oudeyer P., Kaplan F., Hafner V. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 2007, 11(2):265-286.
57. Padoa-Schioppa C. Orbitofrontal cortex and the computation of economic value. Annals of the New York Academy of Sciences, USA 2007, 1121:232-253.
58. Peciña S. Opioid reward "liking" and "wanting" in the nucleus accumbens. Physiology & Behavior 2008, 94(5):675-680.
59. Peciña S., Berridge K.C. Hedonic hot spot in nucleus accumbens shell: Where do mu-opioids cause increased hedonic impact of sweetness?. Journal of Neuroscience 2005, 25(50):11777-11786.
60. Puterman M.L. Markov decision processes: Discrete stochastic dynamic programming (Wiley series in probability and statistics) 2005, Wiley-Interscience.
61. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. Classical conditioning II: Current theory and research 1972, 64-99. Appleton-Century-Crofts, New York, NY. A.H. Black, W.F. Prokasy (Eds.).
62. Reynolds S.M., Berridge K.C. Fear and feeding in the nucleus accumbens shell: Rostrocaudal segregation of GABA-elicited defensive behavior versus eating behavior. Journal of Neuroscience 2001, 21(9):3261-3270.
63. Reynolds S.M., Berridge K.C. Positive and negative motivation in nucleus accum-bens shell: Bivalent rostrocaudal gradients for GABA-elicited eating, taste "liking"/"disliking" reactions, place preference/avoidance, and fear. Journal of Neuroscience 2002, 22(16):7308-7320.
64. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience 2007, 10(12):1615-1624.
65. Rummery G., Niranjan M. On-line Q-learning using connectionist systems 1994, Cambridge University, Cambrdige, UK, Technical Report CUED/F-INFENG-TR 166.
66. Samejima K., Ueda Y., Doya K., Kimura M. Representation of action-specific reward values in the striatum. Science 2005, 310(5752):1337-1340.
67. Samuel A. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development 1959, 3:210-229.
68. Schultz W. Getting formal with dopamine and reward. Neuron 2002, 36(2):241-263.
69. Sheffield F. Relation between classical conditioning and instrumental learning. Classical conditioning 1965, 302-322. Appleton-Century-Crofts, New York, NY. W. Prokasy (Ed.).
70. Singh S., Barto A., Chentanez N. Intrinsically motivated reinforcement learning. Advances in Neural Information Processing Systems 2005, 1281-1288. MIT Press, Cambridge, MA.
71. Singh, S., Lewis, R., & Barto, A. (2009). Where do rewards come from? In Proceedings of the thirty-first Annual Conference of the Cognitive Science Society (pp. 2601-2606). Amsterdam, The Netherlands.
72. Smith Y., Villalba R. Striatal and extrastriatal dopamine in the basal ganglia: An overview of its anatomical organization in normal and parkinsonian brains. Movement Disorders 2008, 23(Suppl 3):S534-S547.
73. Suddendorf T., Corballis M.C. Mental time travel and the evolution of the human mind. Genetic, Social, and General Psychology Monographs 1997, 123(2):133-167.
74. Suri R.E., Schultz W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 1999, 91(3):871-890.
75. Sutton R. Learning to predict by the methods of temporal differences. Machine Learning 1988, 3(1):9-44.
76. Sutton, R. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the seventh international conference on machine learning, 216: 224.
77. Sutton R.S., Barto A.G. Reinforcement learning: An introduction 1998, The MIT Press, Cambridge, MA, Adaptive Computation and Machine Learning.
78. Todorov E. Efficient computation of optimal actions. Proceedings of the National Academy of Sciences of the United States of America 2009, 106(28):11478-11483.
79. Tolman E. There is more than one kind of learning. Psychological Review 1949, 56:144-155.
80. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. The European Journal of Neuroscience 2009, 29(11):2225-2232.
81. Tversky A., Thaler R.H. Anomalies: Preference reversals. Journal of Economic Perspective 1990, 4:201-211.
82. Usher M., McClelland J.L. The time course of perceptual choice: The leaky, competing accumulator model. Psychological Review 2001, 108(3):550-592.
83. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. Journal of Neuroscience 2007, 27(15):4019-4026.
84. Wallis J.D., Kennerley S.W. Heterogeneous reward signals in prefrontal cortex. Current Opinion in Neurobiology 2010, 20(2):191-198.
85. Watkins, C. (1989). Learning from delayed rewards. PhD thesis, Cambridge, UK: University of Cambridge.
86. Weber E.U., Shafir S., Blais A.-R. Predicting risk sensitivity in humans and lower animals: Risk as variance or coefficient of variation. Psychological Review 2004, 111(2):430-445.
87. Widrow, B., & Hoff, M. (1960). Adaptive switching circuits. In Western Electric Show and Convention Record (Vol. 4, pp. 96-104). New York, NY.
88. Williams D.R., Williams H. Auto-maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement. Journal of the Experimental Analysis of Behavior 1969, 12(4):511-520.
89. Wittmann B.C., Daw N.D., Seymour B., Dolan R.J. Striatal activity underlies novelty-based choice in humans. Neuron 2008, 58(6):967-973.