Do not bet on the unknown versus try to find out more: Estimation uncertainty and "unexpected uncertainty" both modulate exploration

Frontiers in Neuroscience

Volumn , Issue OCT, 2012, Pages

  Payzan LeNestour, Élise ^a,b   Bossaerts, Peter ^b  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Bayesian learning; Estimation uncertainty; Exploration bonuses; Restless bandit problem; Unexpected uncertainty

Indexed keywords

ARTICLE; BEHAVIORAL SCIENCE; EXPLORATORY BEHAVIOR; GOAL ATTAINMENT; HUMAN; HYPOTHESIS; SIMULATION;

EID: 84870898601     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2012.00150     Document Type: Article

References (32)

1. Allais, M. (1953). Le comportement de l'homme rationnel devant le risque: critique des postulats et axiomes de l'ecole americaine. Econometrica 21, 503-546.
2. Badre, D., Doll, B. B., Long, N. M., and Frank, M. J. (2012). Rostro-lateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron 73, 595-607.
3. Behrens, T. E. J., Woolrich, M. W., Walton, M. E., and Rushworth, M. F. S. (2007). Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214-1221.
4. Cavanagh, J. F., Figueroa, C. M., Cohen, M. X., and Frank, M. J. (2011). Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cereb. Cortex. doi: 10.1093/cercor/bhr332
5. Cohen, J. D., McClure, S. M., and Yu, A. J. (2007). Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 933-942.
6. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., and Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature 441, 876-879.
7. Doya,K.(2008). Modulatorsof decision making. Nat. Neurosci. 11, 410-416.
8. Ellsberg,D. (1961). Risk, ambiguity, and the Savage axioms. Q. J. Econ. 75, 643-669.
9. Frank, M. J., Doll, B. B., Oas-Terpstra, J., and Moreno, F. (2009). Pre-frontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci. 12, 1062-1068.
10. Gittins, J., and Jones, D. (1974). Progress in Statistics. Amsterdam: North-Holland.
11. Hasselmo, M. E. (1999). Neuromod-ulation: acetylcholine and memory consolidation. Trends Cogn. Sci. (Regul. Ed.) 3, 351-359.
12. Hertwig, R., Barron, G., Weber, E. U., and Erev, I. (2003). Decisions from experience and the effect of rare events in risky choices. Psychol. Sci. 15, 534-539.
13. Hirayama, J., Yoshimoto, J., and Ishii, S. (2004). Bayesian representation learning in the cortex regulated by acetylcholine. Neural Netw. 17, 1391-1400.
14. Hirayama, J.,Yoshimotoa,J., and Ishii, S. (2006). Balancing plasticity and stability of on-line learning based on hierarchical Bayesian adaptation of forgetting factors. Neurocomputing 69, 1954-1961.
15. Hsu, M., Bhatt, M., Adolphs, R., Tranel, D.,and Camerer,C.F.(2005). Neural systems responding to degrees of uncertainty in human decision-making. Science 310, 1680-1683.
16. Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T., and Platt, M. L. (2006). Neural signatures of economic preferences forrisk andambi-guity. Neuron 49, 765-775.
17. Jepma, M., and Nieuwenhuis, S. (2011). Pupil diameter predicts changes in the exploration-exploitation tradeoff: evidence for the adaptive gain theory. J. Cogn. Neurosci. 23, 1587-1596.
18. Kakade, S., and Dayan, P. (2002). Dopamine: generalization and bonuses. Neural Netw. 15, 549-559.
19. Levy, I., Snell, J., Nelson, A. J., Rusti-chini, A., and Glimcher,P. W. (2010). Neural representation of subjective value under risk and ambiguity. J. Neurophysiol. 103, 1036-1047.
20. Payzan-LeNestour, E. (2012). Learning to Choose the Right Investment in an Unstable World: Experimental Evidence Based on the Bandit Problem.Swiss Finance Institute Research Paper No. 10-28. 1-51.
21. Payzan-LeNestour, E., and Bossaerts, P. (2011). Risk, estimation uncertainty, and unexpected uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 7, e1001048. doi:10.1371/journal.pcbi.1001048
22. Quinn, A., and Karny, M. (2007). Learning for non-stationary Dirich-let processes. Int. J. Adapt. Control Signal Process. 21, 827-855.
23. Rutishauser, U., Mamelak, A. N., and Schuman, E. M. (2006). Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron 49, 805-813.
24. Savage, L. J. (1954). The Foundations of Statistics. New York: Dover Publications, Inc.
25. Sugrue, L. P., Corrado, G. S., and New-some,W.T.(2004). Matching behavior and the representation of value in the parietal cortex. Science 304, 1782-1787.
26. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). Cambridge: The MIT Press.
27. Trommershäuser, J., Maloney, L. T., and Landy, M. S. (2008). Decision mak-ing,movementplanning andstatisti-cal decision theory. Trends Cogn. Sci. (Regul. Ed.) 12, 291-297.
28. van Dijk, E., and Zeelenberg, M. (2007). When curiosity killed regret: avoiding or seeking the unknown in decision-making under uncertainty. J. Exp. Soc. Psychol. 43, 656-662.
29. Wittle, P. (1988). Restless bandits: activity allocation in a changing world. J. Appl. Probab. 25, 287-298.
30. Yoshida, W., and Ishii, S. (2006). Resolution of uncertainty in prefrontal cortex. Neuron 50, 781-789.
31. Yu, A. J., and Cohen, J. D. (2009). "Sequential effects: superstition or rational behavior?" in Advances in Neural Information Processing Systems, Vol. 21, eds D. Koller, D. Schuurmans, Y. Bengio, and L. Bot-tou (Cambridge, MA: MIT Press), 1873-1880.
32. Yu, A. J., and Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron 46, 681-692.