The anatomy of choice: Active inference and agency

Frontiers in Human Neuroscience

Volumn , Issue SEP, 2013, Pages

  Friston, Karl ^a   Schwartenbeck, Philipp ^a   FitzGerald, Thomas ^a   Moutoussis, Michael ^a   Behrens, Timothy ^a,b   Dolan, Raymond J ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b JOHN RADCLIFFE HOSPITAL   (United Kingdom)

Author keywords

Active inference; Agency; Bayesian; Bounded rationality; Embodied cognition; Free energy; Inference; Utility theory

Indexed keywords

ACCURACY; ACTIVE INFERENCE; ARTICLE; BAYES THEOREM; BEHAVIOR; BEHAVIOR CONTROL; BEHAVIORAL SCIENCE; BRAIN FUNCTION; DECISION MAKING; DECISION THEORY; ENTROPY; FUNCTIONAL ANATOMY; MATHEMATICAL MODEL; NEUROANATOMY; PROBABILITY; SENSE OF AGENCY; UNCERTAINTY;

EID: 84884687911     PISSN: 16625161     EISSN: None     Source Type: Journal    
DOI: 10.3389/fnhum.2013.00598     Document Type: Article

References (81)

1. Ashby, W. R. (1947). Principles of the self-organizing dynamic system. J. Gen. Psychol. 37, 125-128. doi: 10.1080/00221309.1947.9918144
2. Bastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., and Friston, K. J. (2012). Canonical microcircuits for predictive coding. Neuron 76, 695-711. doi: 10.1016/j.neuron.2012.10.038
3. Beal, M. J. (2003). Variational Algorithms for Approximate Bayesian Inference. Ph.D. thesis, University College London.
4. Birkhoff, G. D. (1931). Proof of the ergodic theorem. Proc. Natl. Acad. Sci. U.S.A. 17, 656-660. doi: 10.1073/pnas.17.12.656
5. Braun, D. A., Ortega, P. A., Theodorou, E., and Schaal, S. (2011). "Path integral control and bounded rationality," in IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) (Paris: IEEE), 202-209
6. Camerer, C. F. (2003). Behavioural studies of strategic thinking in games. Trends Cogn. Sci. 7, 225-231. doi:10.1016/S1364-6613(03)00094-9
7. Canolty, R. T., Edwards, E., Dalal, S. S., Soltani, M., Nagarajan, S. S., Kirsch, H. E., et al. (2006). High gamma power is phase-locked to theta oscillations in human neocortex. Science 313, 1626-1628. doi: 10.1126/sci-ence.1128115
8. Cisek, P. (2007). Cortical mechanisms of action selection: the affordance competition hypothesis. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 1585-1599. doi: 10.1098/rstb.2007.2054
9. Cohen, J. D., McClure, S. M., and Yu, A. J. (2007). Should I stay or should I go. How the human brain manages the trade-off between exploitation and exploration. Philos. Trans R. Soc. Lond. B Biol. Sci. 362, 933-942. doi: 10.1098/rstb.2007.2098
10. Conant, R. C., and Ashby, R. W. (1970). Every good regulator of a system must be a model of that system. Int. J. Systems. Sci. 1, 89-97. doi: 10.1080/00207727008920220
11. Coull, J. T., Cheng, R. K., and Meck, W. (2011). Neuroanatomical and neurochemical substrates of timing. Neuropsychopharmacology 36, 3-25. doi: 10.1038/npp.2010.113
12. Daw, N. D., and Doya, K. (2006). The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 16, 199-204. doi: 10.1016/j.conb.2006.03.006
13. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711. doi: 10.1038/nn1560
14. Daw, N. D., and Touretzky, D. S. (2002). Long-term reward prediction in TD models of the dopamine system. Neural Comput. 14, 2567-2583. doi: 10.1162/08997660 2760407973
15. Dayan, P., and Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 8, 429-453. doi: 10.3758/CABN.8.4.429
16. Dayan, P., and Hinton, G. E. (1997). Using expectation maximization for reinforcement learning. Neural Comput. 9, 271-278. doi: 10.1162/neco.1997.9.2.271
17. Dayan, P., Hinton, G. E., and Neal, R. (1995). The helmholtz machine. Neural Comput. 7, 889-904. doi: 10.1162/neco.1995.7.5.889
18. De Martino, B., Fleming, S. M., Garrett, N., and Dolan, R. J. (2012). Confidence in value-based choice. Nat. Neurosci. 16, 105-110. doi: 10.1038/nn.3279
19. Ellison, C. J., Mahoney, J. R., James, R. G., and Crutchfield, J. P. (2011). Information symmetries in irreversible processes. Chaos 21, 037107. doi: 10.1063/1.3637490
20. Feynman, R. P. (1972). Statistical Mechanics. Reading, MA: Benjamin.
21. Fiorillo, C. D., Newsome, W. T., and Schultz, W. (2008). The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci. 11, 966-973. doi: 10.1038/ nn.2159
22. Fiorillo, C. D., Tobler, P. N., and Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898-1902. doi: 10.1126/sci-ence.1077349
23. Fletcher, P. C., and Frith, C. D. (2009). Perceiving is believing: a Bayesian approach to explaining the positive symptoms of schizophrenia. Nat. Rev. Neurosci. 10, 48-58. doi: 10.1038/nrn2536
24. Fox, C., and Roberts, S. (2012). A tutorial on variational Bayes. Ar tif. Intel l. Rev. 38, 85-95. doi: 10.1007/s10462-011-9236-8
25. Frank, M. J., Scheres, A., and Sherman, S. J. (2007). Understanding decision-making deficits in neurological conditions: insights from models of natural action selection. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 1641-1654. doi: 10.1098/rstb.2007.2058
26. Friston, K. (2010). The free-energy principle: a unified brain theory. Nat. Rev. Neurosci. 11, 127-138. doi: 10.1038/nrn2787
27. Friston, K. (2012). A free energy principle for biological systems. Entropy 14, 2100-2121. doi: 10.3390/e14112100
28. Friston, K., Kilner, J., and Harrison, L. (2006). A free energy principle for the brain. J. Physiol. Paris. 100, 70-87. doi: 10.1016/j.jphysparis.2006.10.001
29. Friston, K., Mattout, J., Trujillo-Barreto, N., Ashburner, J., and Penny, W. (2007). Variational free energy and the Laplace approximation. Neuroimage 34, 220-234. doi: 10.1016/j.neuroimage.2006.08.035
30. Friston, K. J., Daunizeau, J., Kilner, J., and Kiebel, S. J. (2010). Action and behavior: a free-energy formulation. Biol. Cybern. 102, 227-260. doi: 10.1007/s00422-010-0364-z
31. Friston, K., Samothrakis, S., and Montague, R. (2012a). Active inference and agency: optimal control without cost functions. Biol. Cybern. 106, 523-541. doi: 10.1007/s00422-012-0512-8
32. Friston, K. J., Fitzgerald, T., Galea, J. M., Adams, R., Brown, H., Dolan, R. J., et al. (2012b). Dopamine, affordance and active inference. PLoS Comput. Biol. 8:e1002327. doi: 10.1371/journal.pcbi.1002327
33. Fudenberg, D., and Kreps, D. (1993). Learning mixed equilibria. Game. Econ. Behav. 5, 320-367. doi: 10.1006/game.1993.1021
34. Galea, J. M., Bestmann, S., Beigi, M., Jahanshahi, M., and Rothwell, J. C. (2012). Action reprogram-ming in Parkinson's disease: response to prediction error is modulated by levels of dopamine. J. Neurosci. 32, 542-550. doi: 10.1523/JNEUROSCI.3621-11.2012
35. Gläscher, J., Daw, N., Dayan, P., and O'Doherty, J. P. (2010). States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585-595. doi: 10.1016/j.neuron.2010.04.016
36. Goldman-Rakic, P. S. (1997). The cortical dopamine system: role in memory and cognition. Adv. Pharmacol. 42, 707-711. doi: 10.1016/S1054-3589(08)60846-7
37. Goldman-Rakic, P. S., Lidow, M. S., Smiley, J. F., and Williams, M. S. (1992). The anatomy of dopamine in monkey and human prefrontal cortex. J. Neural Transm. Suppl. 36, 163-177.
38. Gregory, R. L. (1968). Perceptual illusions and brain models. Proc. R. Soc. Lond. B Biol. Sci. 171, 179-196. doi: 10.1098/rspb.1968.0071
39. Gurney, K., Prescott, T. J., and Redgrave, P. (2001). A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84, 401-410. doi: 10.1007/PL000 07984
40. Haile, P. A., Hortaçsu, A., and Kosenok, G. (2008). On the empirical content of quantal response equilibrium. Am. Econ. Rev. 98, 180-200. doi: 10.1257/aer.98.1.180
41. Helmholtz, H. (1866/1962). "Concerning the perceptions in general," in Treatise on Physiological Optics, 3rd Edn. Vol. 3. Transl. J. Southall (New York, NY: Dover).
42. Hinton, G. E., and van Camp, D. (1993). "Keeping neural networks simple by minimizing the description length of weights," in Proceedings of COLT-93, 5-13. doi: 10.1145/168304.168306
43. Jaynes, E. T. (1957). Information theory and statistical mechanics. Phys. Rev. Ser. II 106, 620-630. doi: 10.1103/PhysRev.106.620
44. Jocham, G., Hunt, L. T., Near, J., and Behrens, T. E. (2012). A mechanism for value-guided choice based on the excitation-inhibition balance in prefrontal cortex. Nat. Neurosci. 15, 960-961. doi: 10.1038/ nn.3140
45. Kahneman, D., and Tversky, A. (1979). Prospect theory: an analysis of decision under risk. Econometrica 47, 263-291. doi: 10.2307/1914185
46. Kakade, S., and Dayan, P. (2002). Dopamine: generalization and bonuses. Neural Netw. 15, 549-559. doi: 10.1016/S0893-6080(02)00048-5
47. Kappen, H. J., Gomez, Y., and Opper, M. (2012). Optimal control as a graphical model inference problem. Mach. Learn. 87, 159-182. doi: 10.1007/s10994-012-5278-7
48. Kauder, E. (1953). Genesis of the marginal utility theory: from aristotle to the end of the eighteenth century. Econ. J. 63, 638-650. doi: 10.2307/2226451
49. Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005). "Empowerment: a universal agent-centric measure of control," in The 2005 IEEE Congress on Evolutionary Computation. Vol. 1, 128-135.
50. Krebs, R. M., Schott, B. H., Schütze, H., and Düzel, E. (2009). The novelty explo ration bonus and its attentional modulation. Neuropsychologia 47, 2272-2281. doi: 10.1016/ j.neuropsychologia.2009.01.015
51. Luce, R. D. (1959). Individual Choice Behavior. Oxford, UK: Wiley.
52. MacKay, D. J. (2003). Information Theory, Inference and Learning Algorithms. Cambridge, UK: Cambridge University Press.
53. Mathys, C., Daunizeau, J., Friston, K. J., and Stephan, K. E. (2011). A Bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci. 5:39. doi: 10.3389/fnhum.2011.00039
54. McKelvey, R., and Palfrey, T. (1995). Quantal response equilibria for normal form games. Game. Econ. Behav. 10, 6-38. doi: 10.1006/game.1995.1023
55. Moran, R. J., Symmonds, M., Stephan, K. E., Friston, K. J., and Dolan, R. J. (2011). An in vivo assay of synaptic function mediating human cognition. Curr. Biol. 21, 1320-1325. doi: 10.1016/j.cub.2011.06.053
56. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., and Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452-454.
57. Ortega, P. A., and Braun, D. A. (2011). Information, utility and bounded rationality. Lect. Notes Artif. Int. 6830, 269-274. doi: 10.1007/978-3-642-22887-2_28
58. Ortega, P. A., and Braun, D. A. (2013). Thermodynamics as a theory of decision-making with information-processing costs. Proc. R. Soc. A 469, 2153. doi: 10.1098/rspa. 2012.0683
59. Pellicano, E., and Burr, D. (2012). When the world becomes 'too real': a Bayesian explanation of autistic perception. Trends Cogn. Sci. 16, 504-510. doi: 10.1016/j.tics.2012.08.009
60. Penny, W. D., Stephan, K. E., Mechelli, A., and Friston, K. J. (2004). Comparing dynamic causal models. Neuroimage 22, 1157-1172. doi: 10.1016/j.neuroimage.2004.03.026
61. Pezzulo, G., and Castelfranchi, C. (2009). Thinking as the control of imagination: a conceptual framework for goal-directed system. Psychol. Res. 73, 559-577. doi: 10.1007/s00426-009-0237-z
62. Rao, R. P., and Ballard, D. H. (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79-87. doi: 10.1038/4580
63. Redgrave, P., and Gurney, K. (2006). The short-latency dopamine signal: a role in discovering novel actions. Nat. Rev. Neurosci. 7, 967-975. doi: 10.1038/nrn2022
64. Schmidhuber, J. (1991). "Curious model-building control systems," in Proceedings in IEEE International Joint Conference on Neural Networks, Vol. 2, (Singapore), 1458-1463.
65. Schultz, W. (2007). Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259-288. doi: 10.1146/ annurev.neuro.28.061604.135722
66. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599. doi: 10.1126/sci-ence.275.5306.1593
67. Schultz, W., Preuschoff, K., Camerer, C., Hsu, M., Fiorillo, C. D., Tobler, P. N., et al. (2008). Explicit neural signals reflecting reward uncertainty. Philos. Trans. R. Soc. Lond. B Biol. Sci. 363, 3801-3811. doi: 10.1098/rstb.2008.0152
68. Sharot, T., Guitart-Masip, M., Korn, C. W., Chowdhury, R., and Dolan, R. J. (2012). How dopamine enhances an optimism bias in humans. Curr. Biol. 22, 1477-1481. doi: 10.1016/j.cub.2012.05.053
69. Simon, H. A. (1956). Rational choice and the structure of the environment. Psychol. Rev. 63, 129-138. doi: 10.1037/h0042769
70. Sozou, P. D. (1998). A Discounting framework for choice with delayed and probabilistic rewards. Proc. R. Soc. Lond. B 265, 2015-2020. doi: 10.1098/rspb.1998.0534
71. Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and van der Linde, A. (2002). Bayesian measures of model complexity and fit. J. R. Statist. Soc. B 64, 583-639. doi: 10.1111/1467-9868.00353
72. Surmeier, D., Plotkin, J., and Shen, W. (2009). Dopamine and synaptic plasticity in dorsal striatal circuits controlling action selection. Curr. Opin. Neurobiol. 19, 621-628. doi: 10.1016/j.conb.2009.10.003
73. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
74. Tishby, N., and Polani, D. (2011). "Information theory of decisions and actions," in Perception-Action Cycle Springer Series in Cognitive and Neural Systems, eds V. Cutsuridis, A. Hussain, and J. G. Taylor (Berlin: Springer), 601-636.
75. Tobler, P. N., Fiorillo, C. D., and Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science 307, 1642-1645. doi: 10.1126/science.1105370
76. Toussaint, M., and Storkey, A. (2006). "Probabilistic inference for solving discrete and continuous state markov decision processes," in Proceedings of the 23rd International Conference on Machine Learning, 945-952.
77. Van den Broek, J. L., Wiegerinck, W. A., and Kappen, H. J. (2010). Risk-sensitive path integral control. UAI 6, 1-8.
78. Von Neumann, J., and Morgenstern, O. (1944). Theory of Games and Economic Behavior. Princeton, NJ: Princeton University Press.
79. Watkins, C. J., and Dayan, P. (1992). Q-learning. Mach. Learn. 8, 279-292. doi: 10.1007/BF00992698
80. Wissner-Gross, A. D., and Freer, C. E. (2013). Causal Entropic Forces. Phys. Rev. Lett. 110:168702. doi: 10.1103/PhysRevLett.110.168702
81. Zokaei, N., Gorgoraptis, N., and Husain, M. (2012). Dopamine modulates visual working memory precision. J. Vis. 12, 350. doi: 10.1167/12.9.350