A Bayesian foundation for individual learning under uncertainty

Frontiers in Human Neuroscience

Volumn , Issue MAY, 2011, Pages 9-

  Mathys, Christoph ^a,b   Daunizeau, Jean ^a,c   Friston, Karl J ^c   Stephan, Klaas E ^a,c  

^a UNIVERSITY OF ZURICH   (Switzerland)

^b ETH ZURICH   (Switzerland)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Acetylcholine; Decisionmaking; Dopamine; Hierarchical models; Neuromodulation; Serotonin; Variational bayes; Volatility

Indexed keywords

ADAPTIVE BEHAVIOR; ARTICLE; BAYES THEOREM; BRAIN FUNCTION; ERROR; LEARNING; NEUROMODULATION; PREDICTION; PROBABILITY; SIMULATION; THEORY; TIME SERIES ANALYSIS; UNCERTAINTY;

EID: 82955235655     PISSN: 16625161     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2011.00039     Document Type: Article

References (58)

1. Beal, M. J. (2003). Variational Algorithms for Approximate Bayesian Inference. Ph.D. thesis, University College London. Available at: http://www.cse.buffalo.edu/faculty/mbeal/papers/beal03.pdf
2. Beck, J., Ma, W., Kiani, R., Hanks, T., Churchland, A., Roitman, J., Shadlen, M., Latham, P. E., and Pouget, A. (2008). Probabilistic population codes for Bayesian decision making. Neuron 60, 1142-1152.
3. Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., and Rushworth, M. F. S. (2008). Associative learning of social value. Nature 456, 245-249.
4. Behrens, T. E. J., Woolrich, M. W., Walton, M. E., and Rushworth, M. F. S. (2007). Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214-1221.
5. Bresciani, J., Dammeier, F., and Ernst, M. O. (2006). Vision and touch are automatically integrated for the perception of sequences of events. J. Vis. 6, 554-564.
6. Brodersen, K. H., Penny, W. D., Harrison, L. M., Daunizeau, J., Ruff, C. C., Duzel, E., Friston, K. J., and Stephan, K. E. (2008). Integrated Bayesian models of learning and decision making for saccadic eye movements. Neural Netw. 21, 1247-1260.
7. Corrado, G. S., Sugrue, L. P., Sebastian Seung, H., and Newsome, W. T. (2005). Linear-nonlinear-poisson models of primate choice dynamics. J. Exp. Anal. Behav. 84, 581-617.
8. Cox, R. T. (1946). Probability, frequency and reasonable expectation. Am. J. Phys. 14, 1-13.
9. Daunizeau, J., den Ouden, H. E. M., Pessiglione, M., Kiebel, S. J., Stephan, K. E., and Friston, K. J. (2010a). Observing the observer (I): meta-Bayesian models of learning and decision-making. PLoS ONE 5, e15554. doi: 10.1371/journal.pone.0015555
10. Daunizeau, J., den Ouden, H. E. M., Pessiglione, M., Kiebel, S. J., Friston, K. J., and Stephan, K. E. (2010b). Observing the observer (II): deciding when to decide. PLoS ONE 5, e15555. doi: 10.1371/journal.pone.0015555
11. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., and Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature 441, 876-879.
12. Dayan, P., and Huys, Q. J. (2009). Serotonin in affective control. Annu. Rev. Neurosci. 32, 95-126.
13. Dayan, P., and Niv, Y. (2008). Reinforcement learning: the good, the bad and the ugly. Curr. Opin. Neurobiol. 18, 185-196.
14. Den Ouden, H. E. M., Daunizeau, J., Roiser, J., Friston, K. J., and Stephan, K. E. (2010). Striatal prediction error modulates cortical coupling. J. Neurosci. 30, 3210-3219.
15. Deneve, S. (2008). Bayesian spiking neurons II: learning. Neural. Comput. 20, 118-145.
16. Doya, K. (2008). Modulators of decision making. Nat. Neurosci. 11, 410-416.
17. Fearnhead, P., and Liu, Z. (2007). On-line inference for multiple changepoint problems. J. R. Stat. Soc. Series B Stat. Methodol. 69, 589-605.
18. Frank, M. J. (2008). Schizophrenia: a computational reinforcement learning perspective. Schizophr. Bull. 34, 1008-1011.
19. Frank, M. J., Doll, B. B., Oas-Terpstra, J., and Moreno, F. (2009). Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci. 12, 1062-1068.
20. Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T., and Hutchison, K. E. (2007). Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. U.S.A. 104, 16311-16316.
21. Friston, K. (2008). Hierarchical models in the brain. PLoS Comput. Biol. 4, e1000211. doi: 10.1371/journal. pcbi.1000211
22. Friston, K. (2009). The free-energy principle: a rough guide to the brain? Trends Cogn. Sci. (Regul. Ed.) 13, 293-301.
23. Friston, K., Mattout, J., Trujillo-Barreto, N., Ashburner, J., and Penny, W. (2007). Variational free energy and the Laplace approximation. Neuroimage 34, 220-234.
24. Friston, K. J., and Stephan, K. E. (2007). Free-energy and the brain. Synthese 159, 417-458.
25. Geisler, W. S., and Diehl, R. L. (2002). Bayesian natural selection and the evolution of perceptual systems. Philos. Trans. R. Soc. B Biol. Sci. 357, 419-448.
26. Gershman, S. J., and Niv, Y. (2010). Learning latent structure: carving nature at its joints. Curr. Opin. Neurobiol. 20, 251-256.
27. Gluck, M. A., Shohamy, D., and Myers, C. (2002). How do people solve the weather prediction task? individual variability in strategies for probabilistic category learning. Learn. Mem. 9, 408-418.
28. Gu, Q. (2002). Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience 111, 815-835.
29. Häfner, H., Maurer, K., Löffler, W., Van der Heiden, W., Munk-Jørgensen, P., Hambrecht, M., and Riecher-Rössler, A. (1998). The ABC schizophrenia study: a preliminary overview of the results. Soc. Psychiatry Psychiatr. Epidemiol. 33, 380-386.
30. Herz, A. V. M., Gollisch, T., Machens, C. K., and Jaeger, D. (2006). Modeling single-neuron dynamics and computations: a balance of detail and abstraction. Science 314, 80-85.
31. Jaynes, E. T. (1957). Information theory and statistical mechanics. Phys. Rev. 106, 620.
32. Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge, UK: Cambridge University Press.
33. Kalman, R.E. (1960). A new approach to linear fltering and prediction problems. J.Basic Eng. 82, 35-45.
34. Kording, K. P., and Wolpert, D. M. (2004). Bayesian integration in sensorimotor learning. Nature 427, 244-247.
35. Krugel, L. K., Biele, G., Mohr, P. N. C., Li, S., and Heekeren, H. R. (2009). Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and fexibly adapt decisions. Proc. Natl. Acad. Sci. U.S.A. 106, 17951-17956.
36. Laplace, P. (1774). Mémoire sur la prob-abilité des causes par les évènemens. Mém. Acad. Roy. Sci. 6, 621-656.
37. Laplace, P. (1812). Théorie Analytique des Probabilités. Paris: Courcier Imprimeur.
38. London, M., and Hausser, M. (2005). Dendritic computation. Annu. Rev. Neurosci. 28, 503-532.
39. Montague, P. R., Hyman, S. E., and Cohen, J. D. (2004). Computational roles for dopamine in behavioural control. Nature 431, 760-767.
40. Murray, G. K., Corlett, P. R., Clark, L., Pessiglione, M., Blackwell, A. D., Honey, G., Jones, P. B., Bullmore, E. T., Robbins, T. W., and Fletcher, P. C. (2007). Substantia nigra/ventral teg-mental reward prediction error disruption in psychosis. Mol. Psychiatry 13, 267-276.
41. Nassar, M. R., Wilson, R. C., Heasly, B., and Gold, J. I. (2010). An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366-12378.
42. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., and Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452-454.
43. Orbán, G., Fiser, J., Aslin, R. N., and Lengyel, M. (2008). Bayesian learning of visual chunks by human observers. Proc. Natl. Acad. Sci. U.S.A. 105, 2745-2750.
44. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., and Frith, C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behavior in humans. Nature 442, 1042-1045.
45. Preuschoff, K., and Bossaerts, P. (2007). Adding prediction risk to the theory of reward learning. Ann. N. Y. Acad. Sci. 1104, 135-146.
46. Rescorla, R. A., and Wagner, A. R. (1972). "A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement," in Classical Conditioning II: Current Research and Theory, eds A. H. Black and W. F. Prokasy (New York: Appleton-Century-Crofts), 64-99.
47. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599.
48. Smith, A., Li, M., Becker, S., and Kapur, S. (2006). Dopamine, prediction error and associative learning: a model-based account. Network 17, 61-84.
49. Stephan, K. E., Friston, K. J., and Frith, C. D. (2009). Dysconnection in schizophrenia: from abnormal synaptic plasticity to failures of self-monitoring. Schizophr. Bull. 35, 509-527.
50. Steyvers, M., and Brown, S. (2006). Prediction and change detection. Adv. Neural Inf. Process Syst. 18, 1281-1288.
51. Steyvers, M., Lee, M. D., and Wagenmakers, E. (2009). A Bayesian analysis of human decision-making on bandit problems. J. Math. Psychol. 53, 168-179.
52. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
53. Thiel, C. M., Huston, J. P., and Schwarting, R. K. (1998). Hippocampal acetyl-choline and habituation learning. Neuroscience 85, 1253-1262.
54. Wilson, R. C., Nassar, M. R., and Gold, J. I. (2010). Bayesian online learning of the hazard rate in change-point problems. Neural. Comput. 22, 2452-2476.
55. Xu, F., and Tenenbaum, J. B. (2007). Sensitivity to sampling in Bayesian word learning. Dev. Sci. 10, 288-297.
56. Yang, T., and Shadlen, M. N. (2007). Probabilistic reasoning by neurons. Nature 447, 1075-1080.
57. Yu, A. J., and Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron 46, 681-692.
58. Yuille, A., and Kersten, D. (2006). Vision as Bayesian inference: analysis by synthesis? Trends Cogn. Sci. (Regul. Ed.),