Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings

PLoS Computational Biology

Volumn 7, Issue 1, 2011, Pages

  Payzan Lenestour, Elise ^a   Bossaerts, Peter ^b,c  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^c SWISS FINANCE INSTITUTE   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

RISK PERCEPTION; SURVEYS; UNCERTAINTY ANALYSIS;

BAYESIAN; BAYESIAN LEARNING; BAYESIAN UPDATING; ESTIMATION UNCERTAINTIES; HUMAN APPROACH; MODEL FREE; REINFORCEMENT ALGORITHMS; RESTLESS BANDIT; THREE-LEVEL; UNCERTAINTY;

REINFORCEMENT LEARNING;

ALGORITHM; ARTICLE; BAYESIAN LEARNING; CALCULATION; CONTROLLED STUDY; GAME; HUMAN; JUMPING; LEARNING; MENTAL TASK; PROBABILITY; TASK PERFORMANCE; BAYES THEOREM; RISK; UNCERTAINTY;

BAYES THEOREM; PROBABILITY; RISK; UNCERTAINTY;

EID: 79551573880     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1001048     Document Type: Article

References (47)

1. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-1711.
2. Dayan P, Long T (1997) Statistical Models of MIT Press Conditioning. In: Kearns MJ, et al. eds. Conf Proc Adv Neural Inf Process Syst Vol 10. pp 117-123.
3. Yoshida W, Ishii S (2006) Resolution of uncertainty in prefrontal cortex. Neuron 50: 781-789.
4. Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS (2007) Learning the value of information in an uncertain world. Nat Neurosci 10(9): 1214-21.
5. Yu AJ, Dayan P (2005) Uncertainty, neuromodulation, and attention. Neuron 46: 681-692.
6. Bossaerts P, Preuschoff K (2007) Adding prediction risk to the theory of reward learning. Ann N Y Acad Sci 1104: 135-146.
7. Wagner A, Rescorla R (1972) A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokasy WF, eds. Classical Conditioning II: Current Research and Theory. New York: Appleton Century Crofts. pp 64-99.
8. Pearce JM, Hall G (1980) A model for pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 87: 532-552.
9. Payzan-LeNestour E (2010) Bayesian learning in unstable settings: Experimental evidence based on the bandit problem. Swiss Finance Institute Research Paper No 10-28. pp 1-41.
10. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441: 876-879.
11. Paulus MP, Rogalsky C, Simmons A, Feinstein JS, Stein MB (2003) Increased activation in the right insula during risk-taking decision making is related to harm avoidance and neuroticism. Neuroimage 19: 1439-1448.
12. Huettel S, Song A, McCarthy G (2005) Decisions under uncertainty: Probabilistic context influences activation of prefrontal and parietal cortices. J Neurosci 25: 3304-3311.
13. Preuschoff K, Quartz SR, Bossaerts P (2008) Human insula activation reflects risk prediction errors as well as risk. J Neurosci 28: 2745-2752.
14. Ellsberg D (1961) Risk, ambiguity, and the savage axioms. Q J Econ 75: 643-669.
15. Hsu M, Bhatt M, Adolphs R, Tranel D, Camerer CF (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science 310: 1680-1683.
16. Huettel SA, Stowe CJ, Gordon EM, Warner BT, Platt ML (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron 49: 765-75.
17. Hampton AN, Bossaerts P, O'Doherty JP (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 26: 8360-8367.
18. Kakade S, Dayan P (2002) Dopamine: generalization and bonuses. Neural Netw 15: 549-559.
19. Allais M (1953) Le comportement de l'homme rationnel devant le risque: critique des postulats et axiomes de l'ecole americaine. Econometrica 21: 503-546.
20. Knight FH (1921) Risk, Uncertainty and Profit. University of Chicago Press.
21. Keynes JM (1921) A Treatise on Probability. London: Macmillan.
22. Basili M, Zappia C (2010) Ambiguity and uncertainty in Ellsberg and Shackle. Cambridge J Econ 34: 449-474.
23. Hansen LP, Sargent TJ (2001) Acknowledging misspecification in macroeconomic theory. Rev Econ Dyn 4: 519-535.
24. Draper D (1995) Assessment and propagation of model uncertainty. J R Stat Soc Series B Stat Methodol 57: 45-97.
25. Quinn A, Karny M (2007) Learning for non-stationary Dirichlet processes. Int J Adapt Control Signal Process 21: 827-855.
26. Ishii S, Yoshida W, Yoshimoto J (2002) Control of exploitation-exploration meta-parameter in reinforcement learning. Neural Netw 15: 665-687.
27. Aston-Jones G, Cohen JD (2005) An integrative theory of locus coeruleusnorepinephrine function: adaptive gain and optimal performance. Annu Rev Neurosci 28: 403-450.
28. Doya K (2008) Modulators of decision making. Nat Neurosci 11: 410-416.
29. Kass RE, Raftery AE (1995) Bayes factors. J Am Stat Assoc 90: 773-795.
30. Preuschoff K, Bossaerts P, Quartz S (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51: 381-390.
31. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898-1902.
32. Vanni-Mercier G, Mauguiere F, Isnard J, Dreher JC (2009) The hippocampus codes the uncertainty of cue-outcome associations: an intracranial electrophysiological study in humans. J of Neurosci 29: 5287-94.
33. d'Acremont M, Zhong-Lin L, Xiangrui L, Van der Linden M, Bechara A (2009) Neural correlates of risk prediction error during reinforcement learning in humans. Neuroimage 47: 1929-1939.
34. Tobler PN, O'Doherty JP, Dolan RJ, Schultz W (2007) Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J Neurophysiol 97: 1621-32.
35. Christopoulos GI, Tobler PN, Bossaerts P, Dolan RJ, Schultz W (2009) Neural correlates of value, risk, and risk aversion contributing to decision making under risk. J Neurosci 29: 12574-12583.
36. Cohen JD, McClure SM, Yu AJ (2007) Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos Trans R Soc Lond B Biol Sci 362: 933-942.
37. Strange BA, Duggins A, Penny W, Dolan RJ, Friston KJ (2005) Information theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Netw 18: 225-230.
38. Rutishauser U, Mamelak AN, Schuman EM (2006) Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complex. Neuron 49: 805-813.
39. Rushworth MFS, Behrens TEJ (2008) Choice, uncertainty and value in prefrontal and cingulate cortex. Nat Neurosci 11: 389-397.
40. Craik K (1943) The Nature of Explanation. Cambridge University Press.
41. Epstein W, Rock I (1960) Perceptual set as an artifact of recency. Am J Psychol 73: 214-228.
42. Thaler RH, Sunstein CR (2009) Nudge: Improving Decisions About Health, Wealth, and Happiness Penguin.
43. Dow J, Werlang S, da Costa R (1992) Uncertainty aversion, risk aversion, and the optimal choice of portfolio. Econometrica 60: 197-204.
44. Diaconis P, Freedman D (1986) On the consistency of Bayes estimates. Ann Stat 14: 1-26.
45. Sangjoon K, Shepherd N, Siddhartha C (1998) Stochastic volatility: Likelihood inference and comparison with Arch models. Rev Econ Stud 65: 361-393.
46. Courville AC, Daw ND, Touretzky DS (2006) Bayesian theories of conditioning in a changing world. Trends Cogn Sci 10: 294-300.
47. Berger JO (1980) Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics (Second Edition) Springer-Verlag.