Uncertainty in action-value estimation affects both action choice and learning rate of the choice behaviors of rats

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 1180-1189

  Funamizu, Akihiro ^a,b,c   Ito, Makoto ^b   Doya, Kenji ^b   Kanzaki, Ryohei ^c,d   Takahashi, Hirokazu ^c,d,e  

^a JAPAN SOCIETY FOR THE PROMOTION OF SCIENCE   (Japan)

^b OKINAWA INSTITUTE OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY   (Japan)

^c UNIVERSITY OF TOKYO   (Japan)

^d UNIVERSITY OF TOKYO   (Japan)

^e JAPAN SCIENCE AND TECHNOLOGY AGENCY   (Japan)

Author keywords

Ambiguity; Bayesian inference; Bayesian Q learning; Neuroeconomics; Reinforcement learning

Indexed keywords

ANIMAL EXPERIMENT; ARTICLE; BAYESIAN LEARNING; BEHAVIOR; CONTROLLED STUDY; DECISION MAKING; LEARNING; MALE; NEUROMODULATION; NONHUMAN; PRIORITY JOURNAL; PROBABILITY; RAT; REWARD; STATISTICAL MODEL; TASK PERFORMANCE; UNCERTAINTY;

ANIMALS; BAYES THEOREM; CHOICE BEHAVIOR; MALE; PSYCHOMOTOR PERFORMANCE; RATS; RATS, LONG-EVANS; REACTION TIME; REWARD; UNCERTAINTY;

EID: 84859353418     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2012.08025.x     Document Type: Article

References (55)

1. Aston-Jones, G. & Cohen, J.D. (2005) An integrative theory of locus coeruleus norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci., 28, 403-450.
2. Barraclough, D.J., Conroy, M.L. & Lee, D. (2004) Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci., 7, 404-410.
3. Behrens, T.E.J., Woolrich, M.W., Walton, M.E. & Rushworth, M.F.S. (2007) Learning the value of information in an uncertain world. Nat. Neurosci., 10, 1214-1221.
4. Bishop, C.M. (2006) Pattern Recognition and Machine Learning. Springer, New York.
5. Blundon, J.A., Bayazitov, I.T. & Zakharenko, S.S. (2011) Presynaptic gating of postsynaptically expressed plasticity at mature thalamocortical synapses. J. Neurosci., 31, 16012-16025.
6. Christopoulos, G.I., Tobler, P.N., Bossaerts, P., Dolan, R.J. & Schultz, W. (2009) Neural correlates of value, risk, and risk aversion contributing to decision making under risk. J. Neurosci., 29, 12574-12583.
7. Corrado, G. & Doya, K. (2007) Understanding neural coding through the model-based analysis of decision making. J. Neurosci., 27, 8178-8180.
8. Daw, N.D. & Doya, K. (2006) The computational neurobiology of learning and reward. Curr. Opin. Neurobiol., 16, 199-204.
9. Daw, N.D., Niv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci., 8, 1704-1711.
10. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, P. & Dolan, R.J. (2006) Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879.
11. Dayan, P. & Niv, Y. (2008) Reinforcement learning: the good, the bad and the ugly. Curr. Opin. Neurobiol., 18, 185-196.
12. Dayan, P. & Sejnowski, T.J. (1996) Exploration bonuses and dual control. Mach. Learn., 25, 5-22.
13. De Martino, B., Kumaran, D., Seymour, B. & Dolan, R.J. (2006) Frames, biases, and rational decision-making in the human brain. Science, 313, 684-687.
14. Dearden, R., Friedman, N. & Russell, S.J. (1998) Bayesian Q-learning. Proc. 15th AAAI, 761-768.
15. Downar, J., Crawley, A.P., Mikulis, D.J. & Davis, K.D. (2002) A cortical network sensitive to stimulus salience in a neutral behavioral context across multiple sensory modalities. J. Neurophysiol., 87, 615-620.
16. Doya, K. (2002) Metalearning and neuromodulation. Neural Netw., 15, 495-506.
17. Doya, K. (2008) Modulators of decision making. Nat. Neurosci., 11, 410-416.
18. Ellsberg, D. (1961) Risk, ambiguity, and the savage axioms. Quart. J. Econ., 75, 643-669.
19. Epstein, L.G. (1999) A definition of uncertainty aversion. Rev. Econ. Stud., 66, 579-608.
20. Esber, G.R. & Haselgrove, M. (2011) Reconciling the influence of predictiveness and uncertainty on stimulus salience: a model of attention in associative learning. Proc. Biol. Sci., 278, 2553-2561.
21. Fiorillo, C.D., Tobler, P.N. & Schultz, W. (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898-1902.
22. Froemke, R.C., Merzenich, M.M. & Schreiner, C.E. (2007) A synaptic memory trace for cortical receptive field plasticity. Nature, 450, 425-429.
23. Glascher, J., Daw, N., Dayan, P. & O'Doherty, J.P. (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66, 585-595.
24. Gompf, H.S., Mathai, C., Fuller, P.M., Wood, D.A., Pedersen, N.P., Saper, C.B. & Lu, J. (2010) Locus ceruleus and anterior cingulate cortex sustain wakefulness in a novel environment. J. Neurosci., 30, 14543-14551.
25. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci., 26, 8360-8367.
26. Hsu, M., Bhatt, M., Adolphs, R., Tranel, D. & Camerer, C.F. (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science, 310, 1680-1683.
27. Huettel, S.A., Stowe, C.J., Gordon, E.M., Warner, B.T. & Platt, M.L. (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron, 49, 765-775.
28. Ishii, S., Yoshida, W. & Yoshimoto, J. (2002) Control of exploitation-exploration meta-parameters in reinforcement learning. Neural Netw., 15, 665-687.
29. Ito, M. & Doya, K. (2009) Validation of decision-making models and analysis of decision variables in the rat basal ganglia. J. Neurosci., 29, 9861-9874.
30. Kahneman, D. & Tversky, A. (1979) Prospect theory: an analysis of decision under risk. Econometrica, 47, 263-291.
31. Kepecs, A., Uchida, N., Zariwala, H.A. & Mainen, Z.F. (2008) Neural correlates, computation and behavioural impact of decision confidence. Nature, 455, 227-231.
32. Kiani, R. & Shadlen, M.N. (2009) Representation of confidence associated with a decision by neurons in the parietal cortex. Science, 324, 759-764.
33. Knill, D.C. & Pouget, A. (2004) The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci., 27, 712-719.
34. Lau, B. & Glimcher, P.W. (2008) Value representations in the primate striatum during matching behavior. Neuron, 58, 451-463.
35. Letzkus, J.J., Wolff, S.B.E., Meyer, E.M.M., Tovote, P., Courtin, J., Herry, C. & Luthi, A. (2011) A disinhibitory microcircuit for associative fear learning in the auditory cortex. Nature, 480, 331-335.
36. O'Doherty, J.P., Hampton, A. & Kim, H. (2007) Model-based fMRI and its application to reward learning and decision making. Ann. NY Acad. Sci., 1104, 35-53.
37. O'Neill, M. & Schultz, W. (2010) Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron, 68, 789-800.
38. Ongür, D., An, X. & Price, J.L. (1998) Prefrontal cortical projections to the hypothalamus in macaque monkeys. J. Comp. Neurol., 401, 480-505.
39. Pasquereau, B., Nadjar, A., Arkadir, D., Bezard, E., Goillandeau, M., Bioulac, B., Gross, C.E. & Boraud, T. (2007) Shaping of motor responses by incentive values through the basal ganglia. J. Neurosci., 27, 1176-1183.
40. Pouget, A., Dayan, P. & Zemel, R.S. (2003) Inference and computation with population codes. Annu. Rev. Neurosci., 26, 381-410.
41. Preuschoff, K., Bossaerts, P. & Quartz, S.R. (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron, 51, 381-390.
42. Preuschoff, K., Quartz, S.R. & Bossaerts, P. (2008) Human insula activation reflects risk prediction errors as well as risk. J. Neurosci., 28, 2745-2752.
43. Rushworth, M.F.S. & Behrens, T.E.J. (2008) Choice, uncertainty and value in prefrontal and cingulate cortex. Nat. Neurosci., 11, 389-397.
44. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. (2005) Representation of action-specific reward values in the striatum. Science, 310, 1337-1340.
45. Schultz, W., Preuschoff, P., Camerer, C., Hsu, M., Fiorillo, C.D., Tobler, P.N. & Bossaerts, P. (2008) Explicit neural signals reflecting reward uncertainty. Philos. Trans. R. Soc. B, 363, 3801-3811.
46. Sutton, R.S. (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proc. 7th Int. Conf. Mach. Learn., 216-224.
47. Sutton, R.S. & Barto, A.G. (1998) Reinforcement Learning. MIT Press, Cambridge, MA, USA.
48. Symmonds, M., Bossaerts, P. & Dolan, R.J. (2010) A behavioral and neural evaluation of prospective decision-making under risk. J. Neurosci., 30, 14380-14389.
49. Symmonds, M., Wright, N.D., Bach, D.R. & Dolan, R.J. (2011) Deconstructing risk: separable encoding of variance and skewness in the brain. Neuroimage, 58, 1139-1149.
50. Tobler, P.N., Christopoulos, G.I., O'Doherty, J.P., Dolan, R.J. & Schultz, W. (2009) Risk-dependent reward value signal in human prefrontal cortex. Proc. Natl. Acad. Sci. USA, 106, 7185-7190.
51. Tversky, A. & Kahneman, D. (1981) The framing of decisions and the psychology of choice. Science, 211, 453-458.
52. Usher, M., Cohen, J.D., Servan-Schreiber, D., Rajkowski, J. & Aston-Jones, G. (1999) The role of locus coeruleus in the regulation of cognitive performance. Science, 283, 549-554.
53. Watkins, C.J.C.H. & Dayan, P. (1992) Q-learning. Mach. Learn., 8, 279-292.
54. Yu, A.J. & Dayan, P. (2002) Acetylcholine in cortical inference. Neural Netw., 15, 719-730.
55. Yu, A.J. & Dayan, P. (2005) Uncertainty, neuromodulation, and attention. Neuron, 46, 681-692.