Humans use directed and random exploration to solve the explore-exploit dilemma

Journal of Experimental Psychology: General

Volumn 143, Issue 6, 2014, Pages 2074-2081

  Wilson, Robert C ^a   Geana, Andra ^a   White, John M ^a   Ludvig, Elliot A ^a,b   Cohen, Jonathan D ^a  

^a PRINCETON UNIVERSITY   (United States)

^b UNIVERSITY OF WARWICK   (United Kingdom)

Author keywords

Decision making; Decision noise; Explore exploit; Information bonus; Reinforcement learning

Indexed keywords

ADOLESCENT; DECISION MAKING; EXPLORATORY BEHAVIOR; FEMALE; HUMAN; MALE; NEUROPSYCHOLOGICAL TEST; PHYSIOLOGY; REWARD; YOUNG ADULT;

ADOLESCENT; CHOICE BEHAVIOR; DECISION MAKING; EXPLORATORY BEHAVIOR; FEMALE; HUMANS; MALE; NEUROPSYCHOLOGICAL TESTS; REWARD; YOUNG ADULT;

EID: 84925600345     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0038199     Document Type: Article

References (35)

1. Aston-Jones, G., & Cohen, J. D. (2005). An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annual Review of Neuroscience, 28, 403-450. http://dx.doi.org/ 10.1146/annurev.neuro.28.061604.135709
2. Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235-256. http://dx .doi.org/10.1023/A:1013689704352
3. Banks, J., Olson, M., & Porter, D. (1997). An experimental analysis of the bandit problem. Economic Theory, 10, 55-77. http://dx.doi.org/10.1007/ s001990050146
4. Beck, J. M., Ma, W. J., Pitkow, X., Latham, P. E., & Pouget, A. (2012). Not noisy, just wrong: The role of suboptimal inference in behavioral variability. Neuron, 74, 30-39. http://dx.doi.org/10.1016/j.neuron.2012 .03.016
5. Bier, V. M., & Connell, B. L. (1994). Ambiguity seeking in multi-attribute decisions: Effects of optimism and message framing. Journal of Behavioral Decision Making, 7, 169 -182. http://dx.doi.org/10.1002/bdm .3960070303
6. Bridle, J. S. (1990). Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimates of parameters. In D. S. Touretzky (Ed.), Advances in neural information processing systems (Vol. 2, pp. 211-217). Cambridge, MA: MIT Press.
7. Bubeck, S., & Cesa-Bianchi, N. (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5, 1-130.
8. Camerer, C., & Weber, M. (1992). Recent developments in modeling preferences: Uncertainty and ambiguity. Journal of Risk and Uncertainty, 5, 325-370. http://dx.doi.org/10.1007/BF00122575
9. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879. http://dx.doi.org/10.1038/nature04766
10. Doya, K., & Sejnowski, T. J. (1995). A novel reinforcement model of birdsong vocalization learning. In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems (Vol. 7, pp. 101-108). Cambridge, MA: MIT Press.
11. Ellsberg, D. (1961). Risk, ambiguity and the savage axioms. The Quarterly Journal of Economics, 75, 643. http://dx.doi.org/10.2307/1884324
12. Faisal, A. A., Selen, L. P., & Wolpert, D. M. (2008). Noise in the nervous system. Nature Reviews Neuroscience, 9, 292-303. http://dx.doi.org/ 10.1038/nrn2258
13. Frank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009). Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature Neuroscience, 12, 1062-1068. http://dx .doi.org/10.1038/nn.2342
14. Gittins, J. (1979). Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B. Methodological, 41, 148-177.
15. Gittins, J., & Jones, D. (1974). A dynamic allocation index for the sequential design of experiments. In J. Gans (Ed.), Progress in statistics (pp. 241-266). Amsterdam, the Netherlands: North-Holland.
16. Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004). Decisions from experience and the effect of rare events in risky choice. Psychological Science, 15, 534 -539. http://dx.doi.org/10.1111/j.0956-7976.2004 .00715.x
17. Hertwig, R., & Erev, I. (2009). The description-experience gap in risky choice. Trends in Cognitive Sciences, 13, 517-523. http://dx.doi.org/ 10.1016/j.tics.2009.09.004
18. Kaelbling, L., Littman, M., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237-285.
19. Kahn, B. E., & Sarin, R. K. (1988). Modeling ambiguity in decisions under uncertainty. The Journal of Consumer Research, 15, 265-272. http://dx .doi.org/10.1086/209163
20. Kao, M. H., Doupe, A. J., & Brainard, M. S. (2005). Contributions of an avian basal ganglia-forebrain circuit to real-time modulation of song. Nature, 433, 638-643. http://dx.doi.org/10.1038/nature03127
21. Lee, M. D., Zhang, S., Munro, M. N., & Steyvers, M. (2011). Psychological models of human and optimal performance on bandit problems. Cognitive Systems Research, 12, 164-174. http://dx.doi.org/10.1016/j .cogsys.2010.07.007
22. Luce, D. (1959). Individual choice behavior. New York, NY: Wiley.
23. McClure, S. M., Gilzenrat, M. S., & Cohen, J. D. (2006). An exploration- exploitation model based on norepinepherine and dopamine activity. In Y. Weiss, B. Schölkopf, & J. Platt (Eds.), Advances in neural information processing systems (Vol. 18, pp. 867-874). Cambridge, MA: MIT Press.
24. Meyer, R., & Shi, Y. (1995). Choice under ambiguity: Intuitive solutions to the armed-bandit problem. Management Science, 41, 817-834. http:// dx.doi.org/10.1287/mnsc.41.5.817
25. Ölveczky, B. P., Andalman, A. S., & Fee, M. S. (2005). Vocal experimentation in the juvenile songbird requires a basal ganglia circuit. PLoS Biology, 3, e153. http://dx.doi.org/10.1371/journal.pbio.0030153
26. Osborne, L. C., Lisberger, S. G., & Bialek, W. (2005). A sensory source for motor variation. Nature, 437, 412- 416. http://dx.doi.org/10.1038/ nature03961
27. Payzan-LeNestour, E., & Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, 7, e1001048. http://dx.doi.org/ 10.1371/journal.pcbi.1001048
28. Payzan-LeNestour, E., & Bossaerts, P. (2012). Do not bet on the unknown versus try to find out more: Estimation uncertainty and "unexpected uncertainty" both modulate exploration. Frontiers in Neuroscience, 6, 150. http://dx.doi.org/10.3389/fnins.2012.00150
29. Steyvers, M., Lee, M., & Wagenmakers, E. (2009). A Bayesian analysis of human decision-making on bandit problems. Journal of Mathematical Psychology, 53, 168-179. http://dx.doi.org/10.1016/j.jmp.2008.11.002
30. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
31. Thompson, W. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25, 285-294. http://dx.doi.org/10.1093/biomet/25.3-4.285
32. Tumer, E. C., & Brainard, M. S. (2007). Performance variability enables adaptive plasticity of 'crystallized' adult birdsong. Nature, 450, 1240- 1244. http://dx.doi.org/10.1038/nature06390
33. Watkins, C. J. C. H. (1989). Learning from delayed rewards (Unpublished doctoral dissertation). Cambridge University, Cambridge, England.
34. Wu, H. G., Miyamoto, Y. R., Gonzalez Castro, L. N., ölveczky, B. P., & Smith, M. A. (2014). Temporal structure of motor variability is dynamically regulated and predicts motor learning ability. Nature Neuroscience, 17, 312-321. http://dx.doi.org/10.1038/nn.3616
35. Zhang, S., & Yu, A. J. (2013). Forgetful Bayes and myopic planning: Human learning and decision making in a bandit setting. Advances in Neural Information Processing Systems, 26, 2607-2615.