Learning the opportunity cost of time in a patch-foraging task

Cognitive, Affective and Behavioral Neuroscience

Volumn 15, Issue 4, 2015, Pages 837-853

  Constantino, Sara M ^a   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

Computational model; Decision making; Dopamine; Patch foraging; Reinforcement learning; Reward

Indexed keywords

ADULT; APPETITE; BAYES THEOREM; DECISION MAKING; FEMALE; HUMAN; LEARNING; MALE; PSYCHOLOGIC TEST; PSYCHOLOGICAL MODEL; REWARD; TIME FACTOR; YOUNG ADULT;

ADULT; APPETITIVE BEHAVIOR; BAYES THEOREM; CHOICE BEHAVIOR; FEMALE; HUMANS; LEARNING; MALE; MODELS, PSYCHOLOGICAL; PSYCHOLOGICAL TESTS; REWARD; TIME FACTORS; YOUNG ADULT;

EID: 84947040901     PISSN: 15307026     EISSN: None     Source Type: Journal    
DOI: 10.3758/s13415-015-0350-y     Document Type: Article

References (81)

1. Aston-Jones, G., & Cohen, J. D. (2005). An integrative theory of locus coeruleus–norepinephrine function: Adaptive gain and optimal performance. Annual Review of Neuroscience, 28, 403–450. doi:10.1146/annurev.neuro.28.061604.135709
2. Barraclough, D. J., Conroy, M. L., & Lee, D. (2004). Prefrontal cortex and decision making in a mixed-strategy game. Nature Neuroscience, 7, 404–410.
3. Baum, W. M. (1974). Choice in free-ranging wild pigeons. Science, 185, 78–79.
4. Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10, 1214–1221.
5. Beierholm, U., Guitart-Masip, M., Economides, M., Chowdhury, R., Düzel, E., Dolan, R., & Dayan, P. (2013). Dopamine modulates reward-related vigor. Neuropsychopharmacology, 38, 1495–1503.
6. Bernacchia, A., Seo, H., Lee, D., & Wang, X.-J. (2011). A reservoir of time constants for memory traces in cortical neurons. Nature Neuroscience, 14, 366–372.
7. Bernoulli, D. (1954). Exposition of a new theory on the measurement of risk. Econometrica, 22, 23–36.
8. Bernstein, C., Kacelnik, A., & Krebs, J. (1988). Individual decisions and the distribution of predators in a patchy environment. Journal of Animal Ecology, 57, 1007–1026.
9. Boorman, E. D., Rushworth, M. F., & Behrens, T. E. (2013). Ventromedial prefrontal and anterior cingulate cortex adopt choice and default reference frames during sequential multi-alternative choice. Journal of Neuroscience, 33, 2242–2253.
10. Brandstätter, E., Gigerenzer, G., & Hertwig, R. (2006). The priority heuristic: Making choices without trade-offs. Psychological Review, 113, 409–432. doi:10.1037/0033-295X.113.2.409
11. Cain, M. S., Vul, E., Clark, K., & Mitroff, S. R. (2012). A Bayesian optimal foraging model of human visual search. Psychological Science, 23, 1047–1054.
12. Charnov, E. L. (1976). Optimal foraging, the marginal value theorem. Theoretical Population Biology, 9, 129–136.
13. Cools, R. (2008). Role of dopamine in the motivational and cognitive control of behavior. The Neuroscientist, 14, 381–395.
14. Cools, R., Nakamura, K., & Daw, N. D. (2011). Serotonin and dopamine: Unifying affective, activational, and decision functions. Neuropsychopharmacology, 36, 98–113.
15. Curtis, C. E., & Lee, D. (2010). Beyond working memory: The role of persistent activity in decision making. Trends in Cognitive Sciences, 14, 216–222.
16. Cuthill, I. C., Kacelnik, A., Krebs, J. R., Haccou, P., & Iwasa, Y. (1990). Starlings exploiting patches: The effect of recent experience on foraging decisions. Animal Behaviour, 40, 625–640.
17. Daw, N. D., & Touretzky, D. S. (2002). Long-term reward prediction in TD models of the dopamine system. Neural Computation, 14, 2567–2583.
18. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704–1711.
19. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876–879.
20. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69, 1204–1215. doi:10.1016/j.neuron.2011.02.027
21. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Networks, 12, 961–974.
22. Frank, M. J., Seeberger, L. C., & O’Reilly, R. C. (2004). By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science, 306, 1940–1943.
23. Freidin, E., & Kacelnik, A. (2011). Rational choice, context dependence, and the value of information in european starlings (Sturnus vulgaris). Science, 334, 1000–1002.
24. Gallistel, C. R., & Gibbon, J. (2000). Time, rate, and conditioning. Psychological Review, 107, 289–344. doi:10.1037/0033-295X.107.2.289
25. Gan, J. O., Walton, M. E., & Phillips, P. E. M. (2009). Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nature Neuroscience, 13, 25–27.
26. Gilzenrat, M. S., Nieuwenhuis, S., Jepma, M., & Cohen, J. D. (2010). Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cognitive, Affective, & Behavioral Neuroscience, 10, 252–269. doi:10.3758/CABN.10.2.252
27. Guitart-Masip, M., Beierholm, U. R., Dolan, R., Duzel, E., & Dayan, P. (2011). Vigor in the face of fluctuating rates of reward: An experimental examination. Journal of Cognitive Neuroscience, 23, 3933–3938.
28. Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006). The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience, 26, 8360–8367.
29. Hare, T. A., Schultz, W., Camerer, C. F., O’Doherty, J. P., & Rangel, A. (2011). Transformation of stimulus value signals into motor commands during simple choice. Proceedings of the National Academy of Sciences, 108, 18120–18125.
30. Hayden, B. Y., Pearson, J. M., & Platt, M. (2011). Neuronal basis of sequential foraging decisions in a patchy environment. Nature Neuroscience, 14, 933–939.
31. Herrnstein, R. J. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behavior, 4, 267–272. doi:10.1901/jeab. 1961.4-267
32. Herrnstein, R. J. (1991). Experiments on stable suboptimality in individual behavior. American Economic Review, 81, 360–364.
33. Hills, T. T., & Hertwig, R. (2010). Information search in decisions from experience: Do our patterns of sampling foreshadow our decisions? Psychological Science, 21, 1787–1792.
34. Hills, T. T., Jones, M. N., & Todd, P. M. (2012). Optimal foraging in semantic memory. Psychological Review, 119, 431–440. doi:10.1037/a0027373
35. Hodges, C. M. (1985). Bumble bee foraging: Energetic consequences of using a threshold departure rule. Ecology, 66, 188–197.
36. Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249–270). Cambridge, MA: MIT Press.
37. Hutchinson, J. M. C., Wilke, A., & Todd, P. M. (2008). Patch leaving in humans: Can a generalist adapt its rules to dispersal of items across patches? Animal Behaviour, 75, 1131–1349.
38. Ito, M., & Doya, K. (2009). Validation of decision-making models and analysis of decision variables in the rat basal ganglia. Journal of Neuroscience, 29, 9861–9874. doi:10.1523/JNEUROSCI. 6157-08.2009
39. Jacobs, E. A., & Hackenberg, T. D. (1996). Humans’ choices in situations of time-based diminishing returns: Effects of fixed-interval duration and progressive-interval step size. Journal of the Experimental Analysis of Behavior, 65, 5–19.
40. Kacelnik, A. (1984). Central place foraging in starlings (Sturnus vulgaris): I. Patch residence time. Journal of Animal Ecology, 53, 283–299.
41. Kacelnik, A. (1997). Normative and descriptive models of decision making: Time discounting and risk sensitivity. Ciba Foundation Symposium, 208, 51–70.
42. Kacelnik, A., Vasconcelos, M., Monteiro, T., & Aw, J. (2010). Darwin’s “tug-of-war” vs. starlings’ “horse-racing”: How adaptations for sequential encounters drive simultaneous choice. Behavioral Ecology & Sociobiology, 65, 547–558.
43. Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7, e1002055. doi:10.1371/journal.pcbi.1002055
44. Kolling, N., Behrens, T. E. J., Mars, R. B., & Rushworth, M. F. (2012). Neural mechanisms of foraging. Science, 336, 95–98.
45. Krajbich, I., Armel, C., & Rangel, A. (2010). Visual fixations and the computation and comparison of value in simple choice. Nature Neuroscience, 13, 1292–1298.
46. Krebs, J. R., & Inman, A. J. (1992). The University of Chicago learning and foraging: Individuals, groups, and populations. American Naturalist, 140, S63–S84.
47. Krebs, J. R., Erichsen, J. T., Webber, M. I., & Charnov, E. L. (1977). Optimal prey selection in the great tit (Parus major). Animal Behaviour, 25, 30–38.
48. Kurzban, R., Duckworth, A., Kable, J. W., & Myers, J. (2012). An opportunity cost model of subjective effort and task performance. Behavioral and Brain Sciences, 36, 697–698.
49. Lex, A., & Hauber, W. (2008). Dopamine D1 and D2 receptors in the nucleus accumbens core and shell mediate Pavlovian-instrumental transfer. Learning and Memory, 15, 483–491.
50. Li, J., & Daw, N. D. (2011). Signals in human striatum are appropriate for policy update rather than value prediction. Journal of Neuroscience, 31, 5504–5511.
51. McCall, J. J. (1970). Economics of information and job search. Quarterly Journal of Economics, 84, 113–126.
52. McNamara, J. M., & Houston, A. I. (1985). Optimal foraging and learning. Journal of Theoretical Biology, 117, 231–249.
53. McNickle, G. G., & Cahill, J. F. (2009). Plant root growth and the marginal value theorem. Proceedings of the National Academy of Sciences, 106, 4747–4751. doi:10.1073/pnas.0807971106
54. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16, 1936–1947.
55. Niv, Y., Daw, N., & Dayan, P. (2006). How fast to work: Response vigor, motivation and tonic dopamine. In Y. Weiss, B. Scholkopf, & J. Platt (Eds.), Advances in neural information processing systems (Vol. 18, pp. 1019–1026). Cambridge, MA: MIT Press.
56. Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191, 507–520. doi:10.1007/s00213-006-0502-4
57. Niyogi, R. K., Breton, Y.-A., Solomon, R. B., Conover, K., Shizgal, P., & Dayan, P. (2014). Optimal indolence: A normative microscopic approach to work and leisure. Interface, 11, 91.
58. O’Doherthy, J. P. (2011). Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Annals of the New York Academy of Sciences, 1239, 118–129.
59. Ollason, J. G. (1980). Learning to forage-optimally? Theoretical Population Biology, 56, 44–56.
60. Puterman, M. L. (2009). Markov decision processes: Discrete stochastic dynamic programming. New York, NY: Wiley.
61. Rangel, A., Camerer, C., & Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience, 9, 545–556.
62. Robbins, T. W., & Everitt, B. J. (2007). A role for mesencephalic dopamine in activation: Commentary on Berridge (2006). Psychopharmacology, 191, 433–437.
63. Rushworth, M. F. S., Kolling, N., Sallet, J., & Mars, R. B. (2012). Valuation and decision-making in frontal cortex: One or many serial or parallel systems? Current Opinion in Neurobiology, 22, 946–955. doi:10.1016/j.conb.2012.04.011
64. Rustichini, A. (2009). Neuroeconomics: Formal models of decision making and cognitive neuroscience. In P. W. Glimcher, C. F. Camerer, E. Fehr, & R. A. Poldrack (Eds.), Neuroeconomics: Decision making and the brain (pp. 33–46). London, UK: Elsevier Academic Press.
65. Salamone, J. D. (1988). Dopaminergic involvement in activational aspects of motivation: Effects of haloperidol on schedule-induced activity, feeding, and foraging in rats. Psychobiology, 16, 196–206.
66. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593–1599.
67. Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards. In Proceedings of the Tenth International Conference on Machine Learning (ICML ’93) (pp. 298–305). Piscataway, NJ: IEEE Press.
68. Seo, H., Barraclough, D. J., & Lee, D. (2007). Dynamic signals related to choices and outcomes in the dorsolateral prefrontal cortex. Cerebral Cortex, 17, 110–117.
69. Shenhav, A., Botvinick, M. M., & Cohen, J. D. (2013). The expected value of control: An integrative theory of anterior cingulate cortex function. Neuron, 79, 217–240.
70. Shenhav, A., Straccia, M. A., Cohen, J. D., & Botvinick, M. M. (2014). Anterior cingulate engagement in a foraging context reflects choice difficulty, not foraging value. Nature Neuroscience, 17, 1249–1254. doi:10.1038/nn.3771
71. Smith, E. A., & Winterhalder, B. (1992). Evolutionary ecology and human behavior. New York, NY: Aldine De Gruyter.
72. Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009). Bayesian model selection for group studies. NeuroImage, 46, 1004–1017.
73. Stephens, D. W., & Krebs, J. R. (1986). Foraging theory. Princeton, NJ: Princeton University Press.
74. Stephens, D. W., Kerr, B., & Fernandez-Juricic, E. (2004). Impulsiveness without discounting: The ecological rationality hypothesis. Proceedings of the Royal Society B, 271, 2459–2465.
75. Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004). Matching behavior and the representation of value in the parietal cortex. Science, 304, 1782–1787.
76. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9–44.
77. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
78. Tom, S. M., Fox, C. R., Trepel, C., & Poldrack, R. A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315, 515–518.
79. Walton, M. E., Groves, J., Jennings, K. A., Croxson, P. L., Sharp, T., Rushworth, M. F. S., & Bannerman, D. M. (2009). Comparing the role of the anterior cingulate cortex and 6-hydroxydopamine nucleus accumbens lesions on operant effort-based decision making. European Journal of Neuroscience, 29, 1678–1691. doi:10.1111/j.1460-9568.2009.06726.x
80. Watkins, C. J. C. H. (1989). Learning from delayed rewards. PhD thesis, Cambridge University, Cambridge, UK.
81. Wikenheiser, A. M., Stephens, D. W., & Redish, A. D. (2013). Subjective costs drive overly patient foraging strategies in rats on an intertemporal foraging task. Proceedings of the National Academy of Sciences, 110, 8308–8313.