Rationalizable irrationalities of choice

Topics in Cognitive Science

Volumn 6, Issue 2, 2014, Pages 204-228

  Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Bounded rationality; Model based; Model free; Noisy decision making; Pavlovian; Pruning; Reinforcement learning

Indexed keywords

ALGORITHM; DECISION MAKING; DEFENSE MECHANISM; HUMAN; PHYSIOLOGY; THEORETICAL MODEL; UNCERTAINTY;

ALGORITHMS; CHOICE BEHAVIOR; DECISION MAKING; HUMANS; MODELS, THEORETICAL; RATIONALIZATION; UNCERTAINTY;

EID: 84899639676     PISSN: 17568757     EISSN: 17568765     Source Type: Journal    
DOI: 10.1111/tops.12082     Document Type: Article

References (115)

1. Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9(1), 147-169.
2. Balleine, B. W. (2005). Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits. Physiology and Behavior, 86(5), 717-730.
3. Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321(5890), 851-854.
4. Beck, J. M., Ma, W. J., Pitkow, X., Latham, P. E., & Pouget, A. (2012). Not noisy, just wrong: The role of suboptimal inference in behavioral variability. Neuron, 74(1), 30-39.
5. Berns, G. S., McClure, S. M., Pagnoni, G., & Montague, P. R. (2001). Predictability modulates human brain response to reward. International Journal of Neuroscience, 21(8), 2793-2798.
6. Berridge, K. C. (2007). The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology (Berl), 191(3), 391-431.
7. Blanchard, D. C., & Blanchard, R. J. (1988). Ethoexperimental approaches to the biology of emotion. Annual Review of Public Psychology, 39:43-68.
8. Bolles, R. C. (1970). Species-specific defense reactions and avoidance learning. Psychological Review, 77, 32-48.
9. Boureau, Y.-L., & Dayan, P. (2011). Opponency revisited: Competition and cooperation between dopamine and serotonin. Neuropsychopharmacology, 36(1), 74-97.
10. Breland, K., & Breland, M. (1961). The misbehavior of organisms. American Psychologist, 16(9), 681-84.
11. Bushong, B., King, L., Camerer, C., & Rangel, A. (2010). Pavlovian processes in consumer choice: The physical presence of a good increases willingness-to-pay. The American Economic Review, 100(4), 1556-1571.
12. Cools, R., Nakamura, K., & Daw, N. D. (2011). Serotonin and dopamine: Unifying affective, activational, and decision functions. Neuropsychopharmacology, 36(1), 98-113.
13. Corcoran, R., Rowse, G., Moore, R., Blackwood, N., Kinderman, P., Howard, R., Cummins, S., & Bentall, R. P. (2008). A transdiagnostic investigation of "theory of mind" and "jumping to conclusions" in patients with persecutory delusions. Psychological Medicine, 38(11), 1577-1583.
14. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6), 1204-1215.
15. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704-1711.
16. Dayan, P. (1993). Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4), 613-624.
17. Dayan, P. (2008). The role of value systems in decision making. In C. Engel & W. Singer (Eds.), Better than conscious? Decision making, the human mind, and implications for institutions (pp. 51-70). Cambridge, MA: MIT Press.
18. Dayan, P. (2009). Goal-directed control and its antipodes. Neural Networks, 22(3), 213-219.
19. Dayan, P. (2012). Robust neural decision-making. In P. Hammerstein & J. Stevens (Eds.), Evolving the mechanisms of decision making (pp. 151-158). Cambridge, MA: MIT Press.
20. Dayan, P., & Huys, Q. J. M. (2008). Serotonin, inhibition, and negative mood. PLoS Comput Biology, 4(2):e4.
21. Dayan, P., Kakade, S., & Montague, P. R. (2000). Learning and selective attention. Nature Neuroscience, 3 Suppl, 1218-1223.
22. Dayan, P., Niv, Y., Seymour, B., & Daw, N. D. (2006). The misbehavior of value and the discipline of the will. Neural Networks, 19(8), 1153-1160.
23. Dayan, P., & Sejnowski, T. (1996). Exploration bonuses and dual control. Machine Learning, 25(1), 5-22.
24. Dezfouli, A., & Balleine, B. W. (2013). Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized. PLoS Computational Biology, 9(12), e1003364.
25. Dickinson, A. (1980). Contemporary animal learning theory. Cambridge, England: Cambridge University Press.
26. Dickinson, A. (1985). Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 308(1135), 67-78.
27. Dickinson, A., & Balleine, B. (2002). The role of learning in motivation. In C. Gallistel (Ed.), Stevens' handbook of experimental psychology (pp. 497-533). New York: Wiley.
28. Doya, K., Samejima, K., Katagiri, K.-i., & Kawato, M. (2002). Multiple model-based reinforcement learning. Neural Computation, 14(6), 1347-1369.
29. Dragoi, G., & Buzsáki, G. (2006). Temporal encoding of place sequences by hippocampal cell assemblies. Neuron, 50(1), 145-157.
30. Dragoi, G., & Tonegawa, S. (2011). Preplay of future place cell sequences by hippocampal cellular assemblies. Nature, 469(7330), 397-401.
31. Evenden, J. L. (1999). Varieties of impulsivity. Psychopharmacology (Berl), 146(4), 348-361.
32. Everitt, B. J., & Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction: From actions to habits to compulsion. Nature Neuroscience, 8(11), 1481-1489.
33. Fear, C. F., & Healy, D. (1997). Probabilistic reasoning in obsessive-compulsive and delusional disorders. Psychological Medicine, 27(1), 199-208.
34. Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010). Evidence for model-based action planning in a sequential finger movement task. Journal of Motor Behavior, 42(6), 371-379.
35. Fine, C., Gardner, M., Craigie, J., & Gold, I. (2007). Hopping, skipping or jumping to conclusions? clarifying the role of the JTC bias in delusions. Cognitive Neuropsychiatry, 12(1), 46-77.
36. Fiser, J., Berkes, P., Orban, G., & Lengyel, M. (2010). Statistically optimal perception and learning: From behavior to neural representations. Trends in Cognitive Sciences, 14(3), 119-130.
37. Foster, D. J., & Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440(7084), 680-683.
38. Foster, D. J., & Wilson, M. A. (2007). Hippocampal theta sequences. Hippocampus, 17(11), 1093-1099.
39. Garety, P. A., Hemsley, D. R., & Wessely, S. (1991). Reasoning in deluded schizophrenic and paranoid patients. biases in performance on a probabilistic inference task. Journal of Nervous and Mental Disease, 179(4), 194-201.
40. Gelly, S., & Silver, D. (2011). Monte-Carlo tree search and rapid action value estimation in computer go. Artificial Intelligence.
41. Gittins, J. C. (1989). Multi-Armed Bandit Allocation Indices (Wiley Interscience Series in Systems and Optimization). New York: John Wiley & Sons Inc.
42. Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4), 585-595.
43. Gray, J. A., & McNaughton, N. (2003). The neuropsychology of anxiety (2nd ed). New York: Oxford University Press.
44. Graybiel, A. M. (2008). Habits, rituals, and the evaluative brain. Annual Review of Neuroscience, 31:359-387.
45. Guitart-Masip, M., Beierholm, U. R., Dolan, R., Duzel, E., & Dayan, P. (2011). Vigor in the face of fluctuating rates of reward: An experimental examination. Journal of Cognitive Neuroscience, 23(12), 3933-3938.
46. Guitart-Masip, M., Huys, Q. J. M., Fuentemilla, L., Dayan, P., Duzel, E., & Dolan, R. J. (2012). Go and no-go learning in reward and punishment: Interactions between affect and effect. Neuroimage, 62(1), 154-166.
47. Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20(6), 2369-2382.
48. Haruno, M., & Kawato, M. (2006). Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Networks, 19(8), 1242-1254.
49. Haruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M., Samejima, K., Imamizu, H., & Kawato, M. (2004). A neural correlate of reward-based behavioral learning in caudate nucleus: A functional magnetic resonance imaging study of a stochastic decision task. Journal of Neuroscience, 24(7), 1660-1665.
50. Hazy, T. E., Frank, M. J., & O'Reilly, R. C. (2006). Banishing the homunculus: Making working memory work. Neuroscience, 139(1), 105-118.
51. Hershberger, W. A. (1986). An approach through the looking-glass. Animal Learning and Behavior., 14, 443-51.
52. Horvitz, J. C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience, 96(4), 651-6.
53. Howes, A., Lewis, R., & Vera, A. (2009). Rational adaptation under task and processing constraints: Implications for testing theories of cognition and action. Psychological Review, 116(4), 717.
54. Huq, S. F., Garety, P. A., & Hemsley, D. R. (1988). Probabilistic judgements in deluded and non-deluded subjects. Quarterly Journal of Experimental Psychology A, 40(4), 801-812.
55. Huys, Q. J. M., & Dayan, P. (2009). A Bayesian formulation of behavioral control. Cognition, 113, 314-328.
56. Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012). Bonsai trees in your head: How the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput Biology, 8(3), e1002410.
57. Huys, Q. J. M., Moutoussis, M., & Williams, J. (2011). Are computational models of any use to psychiatry? Neural Networks, 24(6), 544-551.
58. Ikemoto, S., & Panksepp, J. (1999). The role of nucleus accumbens dopamine in motivated behavior: A unifying interpretation with special reference to reward-seeking. Brain Research. Brain Research Reviews, 31(1), 6-41.
59. Joel, D., & Weiner, I. (2000). The connections of the dopaminergic system with the striatum in rats and primates: An analysis with respect to the functional and compartmental organization of the striatum. Neuroscience, 96(3), 451-474.
60. Johnson, A., & Redish, A. D. (2005). Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Networks, 18(9), 1163-1171.
61. Johnson, A., & Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience, 27(45), 12176-12189.
62. Johnson, A., der Meer, M. A., & Redish, A. D. (2007). Integrating hippocampus and striatum in decision-making. Current Opinion Neurobiology, 17, 692-697.
63. Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar Straus & Giroux.
64. Kakade, S., & Dayan, P. (2002). Dopamine: Generalization and bonuses. Neural Networks, 15(4-6), 549-559.
65. Kappen, H., Gómez, V., & Opper, M. (2012). Optimal control as a graphical model inference problem. Machine Learning, 87, 159-182.
66. Koch, C. (1999). Biophysics of computation. Information processing in single neurons. New York: Oxford University Press.
67. Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139(4), 665-682.
68. Laming, D. (1968). Information theory of choice-reaction times. London: Academic Press.
69. Li, J., & Daw, N. D. (2011). Signals in human striatum are appropriate for policy update rather than value prediction. Journal of Neuroscience, 31(14), 5504-5511.
70. London, M., Roth, A., Beeren, L., Häusser, M., & Latham, P. E. (2010). Sensitivity to perturbations in vivo implies high noise and suggests rate coding in cortex. Nature, 466(7302), 123-127.
71. Louie, K., & Wilson, M. A. (2001). Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep. Neuron, 29(1), 145-156.
72. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390(6657), 279-281.
73. Mackintosh, N. (1983). Conditioning and associative learning. Oxford, England: Oxford University Press.
74. Maia, T. V., & Frank, M. J. (2011). From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience, 14(2), 154-162.
75. Marr, D. (1982). Vision. New York: Freeman.
76. McClure, S. M., Berns, G. S., & Montague, P. R. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron, 38(2), 339-346.
77. McNamara, J., & Houston, A. (1980). The application of statistical decision theory to animal behaviour. Journal of Theoretical Biology, 85(4), 673-690.
78. McNaughton, N., & Corr, P. J. (2004). A two-dimensional neuropsychology of defense: Fear/anxiety and defensive distance. Neuroscience and Biobehavioral Reviews, 28(3), 285-305.
79. Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81.
80. Montague, P. R., Dolan, R. J., Friston, K. J., & Dayan, P. (2012). Computational psychiatry. Trends in Cognitive Sciences, 16(1), 72-80.
81. Moutoussis, M., Bentall, R. P., El-Deredy, W., & Dayan, P. (2011). Bayesian modelling of jumping-to-conclusions bias in delusional patients. Cognitive Neuropsychiatry, 16(5), 422-447.
82. Ng, A., & Russell, S. (2000). Algorithms for inverse reinforcement learning. In Proceedings of the seventeenth international conference on machine learning (pp. 663-670). Stanford, CA: Morgan-Kaufmann.
83. Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berl), 191(3), 507-520.
84. Niv, Y., Edlund, J. A., Dayan, P., & O'Doherty, J. P. (2012). Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. Journal of Neuroscience, 32(2), 551-562.
85. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 38(2), 329-337.
86. Panksepp, J. (1998). Affective neuroscience. New York: Oxford University Press.
87. Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review, 87, 532-552.
88. Puterman, M. L. (2005). Markov decision processes: Discrete stochastic dynamic programming (Wiley series in probability and statistics). Hoboken, NH: Wiley.
89. Reynolds, S. M., & Berridge, K. C. (2001). Fear and feeding in the nucleus accumbens shell: Rostrocaudal segregation of GABA-elicited defensive behavior versus eating behavior. Journal of Neuroscience, 21(9), 3261-3270.
90. Reynolds, S. M., & Berridge, K. C. (2002). Positive and negative motivation in nucleus accumbens shell: Bivalent rostrocaudal gradients for GABA-elicited eating, taste "liking"/"disliking" reactions, place preference/avoidance, and fear. Journal of Neuroscience, 22(16), 7308-7320.
91. Rummery, G., & Niranjan, M. (1994). On-line Q-learning using connectionist systems, Cambridge University Engineering Department technical report F-INFENG/TR 166, Cambridge, England.
92. Russell, S., & Wefald, E. (1991). Do the right thing: Studies in limited rationality. Cambridge, MA: MIT Press.
93. Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010). Testing the reward prediction error hypothesis with an axiomatic model. Journal of Neuroscience, 30(40), 13525-13536.
94. Salamone, J. D., & Correa, M. (2002). Motivational views of reinforcement: Implications for understanding the behavioral functions of nucleus accumbens dopamine. Behavioural Brain Research, 137(1-2), 3-25.
95. Simon, H. (1972). Theories of bounded rationality. Decision and organization, 1, 161-176.
96. Singh, S., & Sutton, R. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22(1-3), 123-158.
97. Smith, P. L., & Ratcliff, R. (2004). Psychology and neurobiology of simple decisions. Trends in Neurosciences, 27(3), 161-168.
98. Solway, A., & Botvinick, M. M. (2012). Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates. Psychological Review, 119(1), 120-154.
99. Sorg, J. (2011). The optimal reward problem: Designing effective reward for bounded agents. PhD thesis, University of Michigan, Computer Science and Engineering.
100. Sorg, J., Singh, S., & Lewis, R. (2010). Internal rewards mitigate agent boundedness. In J. Fürnkranz & T. Joachims (Eds.), Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 1007-1014). Haifa, Israel: Omnipress.
101. Stanovich, K., & West, R. (2002). Individual differences in reasoning: Implications for the rationality debate? In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 421-440). Cambridge, UK: Cambridge University Press.
102. Sutton, R. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9-44.
103. Sutton, R. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning, 216, 224.
104. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Adaptive computation and machine learning). Cambridge, MA: MIT Press.
105. Sutton, R., & Pinette, B. (1985). The learning of world models by connectionist networks. In Proceedings of the Seventh Annual Conference of the Cognitive Science Society (pp. 54-64). Irvine, CA: Cognitive Science Society.
106. Sutton, R., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 181-211.
107. Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011). How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022), 1279-1285.
108. Todd, M. T., Niv, Y., & Cohen, J. D. (2009). Learning to use working memory in partially observable environments through dopaminergic reinforcement. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems 21 (pp. 1689-1696). Vancouver, Canada: NIPS Foundation.
109. Todorov, E. (2009). Efficient computation of optimal actions. Proceedings of the National Academy of Sciences of the United States of America, 106(28), 11478-11483.
110. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4), 189-208.
111. Vulkan, N. (2000). An economist's perspective on probability matching. Journal of Economic Surveys, 14(1), 101-118.
112. Wald, A. (1947). Sequential analysis. New York: Wiley.
113. Watkins, C. (1989). Learning from delayed rewards. PhD thesis, University of Cambridge.
114. Williams, D. R., & Williams, H. (1969). Auto-maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement. Journal of Experimental Analysis of Behavior, 12(4), 511-520.
115. Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience, 15(5), 786-791.