Making predictions in a changing world-inference, uncertainty, and learning

Frontiers in Neuroscience

Volumn , Issue 7 JUN, 2013, Pages

  O'Reilly, Jill X ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Bayes theorem; Change detection; Exploratory behavior; Learning; Modeling; Uncertainty

Indexed keywords

ARTICLE; BAYES THEOREM; ENVIRONMENTAL CHANGE; HUMAN; LEARNING; LEARNING ALGORITHM; NERVE CELL; NONHUMAN; PREDICTION; PROBABILITY; UNCERTAINTY;

EID: 84888786687     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2013.00105     Document Type: Article

References (37)

1. Aston-Jones, G., and Cohen, J. D. (2005). An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403-450. doi: 10.1146/annurev.neuro.28.061604.135709
2. Behrens, T. E., Woolrich, M. W., Walton, M. E., and Rushworth, M. F. (2007). Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214-1221. doi: 10.1038/nn1954
3. Bogacz, R., Brown, E., Moehlis, J., Holmes, P., and Cohen, J. D. (2006). The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700-765. doi: 10.1037/0033-295X.113.4.700
4. Bouret, S., and Sara, S. J. (2005). Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci. 28, 574-582. doi: 10.1016/j.tins.2005.09.002
5. Cohen, M. X., and Ranganath, C. (2007). Reinforcement learning signals predict future decisions. J. Neurosci. 27, 371-378. doi: 10.1523/JNEUROSCI.4421-06.2007
6. Courville, A. C., Daw, N. D., and Touretzky, D. S. (2006). Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294-300. doi: 10.1016/j.tics.2006.05.004
7. Dayan, P. (2012). Twenty-five lessons from computational neuromodulation. Neuron 76, 240-256. doi: 10.1016/j.neuron.2012.09.027
8. Dayan, P., Kakade, S., and Montague, P. R. (2000). Learning and selective attention. Nat. Neurosci. 3, 1218-1223. doi: 10.1038/81504
9. Dayan, P., and Long, T. (1998). "Statistical models of conditioning, " in Advances in Neural Information Processing Systems 10, eds M. I. Jordan, M. J. Kearns, and S. A. Solla (Cambridge, MA: MIT Press), 117-123.
10. Dayan, P., and Yu, A. J. (2006). Phasic norepinephrine: a neural interrupt signal for unexpected events. Network 17, 335-350. doi: 10.1080/09548980601004024
11. Debener, S., Ullsperger, M., Siegel, M., Fiehler, K., Von Cramon, D. Y., and Engel, A. K. (2005). Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J. Neurosci. 25, 11730-11737. doi: 10.1523/JNEUROSCI.3286-05.2005
12. Doya, K. (2002). Metalearning and neuromodulation. Neural Netw. 15, 495-506. doi: 10.1016/S0893-6080(02)00044-8
13. Friston, K. (2010). The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127-138. doi: 10.1038/nrn2787
14. Friston, K., and Kiebel, S. (2009). Predictive coding under the free-energy principle. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364, 1211-1221. doi: 10.1098/rstb.2008.0300
15. Hall, G., and Channell, S. (1985). Differential-effects of contextual change on latent inhibition and on the habituation of an orienting response. J. Exp. Psychol. Anim. Behav. Process. 11, 470-481. doi: 10.1037/0097-7403.11.3.470
16. Jodo, E., Chiang, C., and Aston-Jones, G. (1998). Potent excitatory influence of prefrontal cortex activity on noradrenergic locus coeruleus neurons. Neuroscience 83, 63-79. doi: 10.1016/S0306-4522(97)00372-2
17. Karlsson, M. P., Tervo, D. G., and Karpova, A. Y. (2012). Network resets in medial prefrontal cortex mark the onset of behavioral uncertainty. Science 338, 135-139. doi: 10.1126/science.1226518
18. Knight, F. H. (1921). Risk, Uncertainty and Profit. Boston MA: Hart, Schaffner and Marx.
19. Kolling, N., Behrens, T. E., Mars, R. B., and Rushworth, M. F. (2012). Neural mechanisms of foraging. Science 336, 95-98. doi: 10.1126/science.1216930
20. Lovibond, P. F., Preston, G. C., and Mackintosh, N. J. (1984). Context specificity of conditioning, extinction, and latent inhibition. J. Exp. Psychol. Anim. Behav. Process. 10, 360-375. doi: 10.1037/0097-7403.10.3.360
21. Matsumoto, M., Matsumoto, K., Abe, H., and Tanaka, K. (2007). Medial prefrontal cell activity signaling prediction errors of action values. Nat. Neurosci. 10, 647-656. doi: 10.1038/nn1890
22. McGrayne, S. B. (2011). The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy. New Haven, CT: Yale University Press.
23. McLaren, I. P. L., Bennett, C., Plaisted, K., Aitken, M., and Mackintosh, N. J. (1994). Latent inhibition, context specificity, and context familiarity. Q. J. Exp. Psychol. B 47, 387-400. doi: 10.1080/14640749408401366
24. Nassar, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., and Gold, J. I. (2012). Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040-1046. doi: 10.1038/nn.3130
25. Nassar, M. R., Wilson, R. C., Heasly, B., and Gold, J. I. (2010). An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366-12378. doi: 10.1523/JNEUROSCI.0822-10.2010
26. Nieuwenhuis, S., Aston-Jones, G., and Cohen, J. D. (2005). Decision making, the P3, and the locus coeruleus-norepinephrine system. Psychol. Bull. 131, 510-532. doi: 10.1037/0033-2909.131.4.510
27. Payzan-Lenestour, E., and Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 7:e1001048. doi: 10.1371/journal.pcbi.1001048
28. Pearce, J. M., and Hall, G. (1980). A model for pavlovian learning-variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532-552. doi: 10.1037/0033-295X.87.6.532
29. Preuschoff, K., and Bossaerts, P. (2007). Adding prediction risk to the theory of reward learning. Ann. N.Y. Acad. Sci. 1104, 135-146. doi: 10.1196/annals.1390.005
30. Preuschoff, K., T, Hart, B. M., and Einhauser, W. (2011). Pupil dilation signals surprise: evidence for noradrenaline's role in decision making. Front. Neurosci. 5:115. doi: 10.3389/fnins.2011.00115
31. Quilodran, R., Rothe, M., and Procyk, E. (2008). Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 57, 314-325. doi: 10.1016/j.neuron.2007.11.031
32. Rescorla, R. A., and Wagner, A. R. (1972). "A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement, " in Classical Conditioning II: Current Research and Theory, eds A. H. Black and W. F. Prokasy (New York, NY: Appleton-Century Crofts), 64-99. doi: 10.1037/a0030892
33. Sara, S. J. (2009). The locus coeruleus and noradrenergic modulation of cognition. Nat. Rev. Neurosci. 10, 211-223. doi: 10.1038/nrn2573
34. Sara, S. J., and Herve-Minvielle, A. (1995). Inhibitory influence of frontal cortex on locus coeruleus neurons. Proc. Natl. Acad. Sci. U.S.A. 92, 6032-6036. doi: 10.1073/pnas.92.13.6032
35. Usher, M., and McClelland, J. L. (2001). The time course of perceptual choice: the leaky, competing accumulator model. Psychol. Rev. 108, 550-592. doi: 10.1037/0033-295X.108.3.550
36. Wilson, R. C., and Niv, Y. (2011). Inferring relevance in a changing world. Front. Hum. Neurosci. 5:189. doi: 10.3389/fnhum.2011.00189
37. Wilson, R. C., Nassar, M. R., and Gold, J. I. (2010). Bayesian online learning of the hazard rate in change-point problems. Neural Comput. 22, 2452-2476. doi: 10.1162/NECO_a_00007