Active inference and learning

Neuroscience and Biobehavioral Reviews

Volumn 68, Issue , 2016, Pages 862-879

  Friston, Karl ^a   FitzGerald, Thomas ^a,b   Rigoli, Francesco ^a   Schwartenbeck, Philipp ^a,b,c,d   O'Doherty, John ^e   Pezzulo, Giovanni ^f  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b MAX PLANCK UCL CENTRE FOR COMPUTATIONAL PSYCHIATRY AND AGEING RESEARCH   (United Kingdom)

^c UNIVERSITY OF SALZBURG   (Austria)

^d PARACELSUS MEDICAL UNIVERSITY   (Austria)

^e CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^f INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

Author keywords

Active inference; Bayesian inference; Bayesian surprise; Epistemic value; Exploitation; Exploration; Free energy; Goal directed; Habit learning; Information gain

Indexed keywords

DOPAMINE;

ACTIVE INFERENCE; AMBIGUITY; ASSOCIATION; COGNITION; CONDITIONING; DOPAMINE RELEASE; EVENT RELATED POTENTIAL; FORAGING BEHAVIOR; HABIT; HIPPOCAMPUS; LEARNING; MOTIVATION; MOTOR CONTROL; NONHUMAN; PERCEPTION; PERSEVERATION; PRIORITY JOURNAL; REINFORCEMENT; REVIEW; REWARD; DECISION MAKING;

CHOICE BEHAVIOR; DOPAMINE; HABITS; LEARNING; REWARD;

EID: 84979255321     PISSN: 01497634     EISSN: 18737528     Source Type: Journal    
DOI: 10.1016/j.neubiorev.2016.06.022     Document Type: Review

References (94)

1. Abbott, L.F., Nelson, S., Synaptic plasticity: taming the beast. Nat. Neurosci. 3 (2000), 1178–1183.
2. Alagoz, O., Hsu, H., Schaefer, A.J., Roberts, M.S., Markov decision processes: a tool for sequential decision making under uncertainty. Med. Decis. Making 30:4 (2010), 474–483.
3. Attias, H., Planning by probabilistic inference. Proceedings of the 9th International Workshop on Artificial Intelligence and Statistics, 2003.
4. Averbeck, B.B., Theory of choice in bandit: information sampling and foraging tasks. PLoS Comput. Biol., 11(3), 2015, e1004164.
5. Balleine, B.W., Dickinson, A., Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37:4-5 (1998), 407–419.
6. Balleine, B.W., Ostlund, S.B., Still at the choice-point: action selection and initiation in instrumental conditioning. Ann. N. Y. Acad. Sci. 1104 (2007), 147–171.
7. Barlow, H., Possible principles underlying the transformations of sensory messages. Rosenblith, W., (eds.) Sensory Communication, 1961, MIT Press, Cambridge, MA, 217–234.
8. Barto, A., Singh, S., Chentanez, N., Intrinsically motivated learning of hierarchical collections of skills. Proceedings of the 3rd International Conference on Development and Learning (ICDL 2004), Salk Institute, San Diego, 2004.
9. Barto, A., Mirolli, M., Baldassarre, G., Novelty or surprise?. Front. Psychol., 2013, 4.
10. Baxter, J., Bartlett, P.L., Weaver, L., Experiments with infinite-horizon: policy-gradient estimation. J. Artif. Intell. Res. 15 (2001), 351–381.
11. Beal, M.J., Variational Algorithms for Approximate Bayesian Inference. PhD. Thesis. 2003, University College London.
12. Bellman, R., On the theory of dynamic programming. Proc. Natl. Acad. Sci. U. S. A. 38 (1952), 716–719.
13. Bonet, B., Geffner, H., Belief tracking for planning with sensing: width, complexity and approximations. J. Artif. Intell. Res. 50 (2014), 923–970.
14. Botvinick, M., An, J., Goal-directed decision making in prefrontal cortex: a computational framework. Adv. Neural Inf. Process. Syst. (NIPS), 2008.
15. Botvinick, M., Toussaint, M., Planning as inference. Trends Cogn. Sci. 16:10 (2012), 485–488.
16. Braun, D.A., Ortega, P.A., Theodorou, E., Schaal, S., Path integral control and bounded rationality. Adaptive Dynamic Programming And Reinforcement Learning (ADPRL), 2011 IEEE Symposium On, Paris, IEEE, 2011.
17. Brown, L.D., A complete class theorem for statistical problems with finite-sample spaces. Ann. Stat. 9:6 (1981), 1289–1300.
18. Buzsaki, G., Memory consolidation during sleep: a neurophysiological perspective. J. Sleep Res. 7:Suppl. 1 (1998), 17–23.
19. Cooper, G., A method for using belief networks as influence diagrams. Proceedings of the Conference on Uncertainty in Artificial Intelligence, 1988.
20. Daw, N.D., Niv, Y., Dayan, P., Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8:12 (2005), 1704–1711.
21. Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P., Dolan, R.J., Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:6 (2011), 1204–1215.
22. Dayan, P., Niv, Y., Seymour, B., Daw, N.D., The misbehavior of value and the discipline of the will. Neural Netw. 19:8 (2006), 1153–1160.
23. Delamater, A.R., Westbrook, R.F., Psychological and neural mechanisms of experimental extinction: a selective review. Neurobiol. Learn Mem. 108 (2014), 38–51.
24. Dezfouli, A., Balleine, B.W., Actions, action sequences and habits: evidence that goal-directed and habitual action control are hierarchically organized. PLoS Comput. Biol., 9(12), 2013, e1003364.
25. Dolan, R.J., Dayan, P., Goals and habits in the brain. Neuron 80:2 (2013), 312–325.
26. Duff, M., Optimal Learning: Computational Procedure for Bayes-Adaptive Markov Decision Processes. 2002, Amherst.
27. Everitt, B.J., Robbins, T.W., From the ventral to the dorsal striatum: devolving views of their roles in drug addiction. Neurosci. Biobehav. Rev. 37:9, Part A (2013), 1946–1954.
28. Fiorillo, C.D., Tobler, P.N., Schultz, W., Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:5614 (2003), 1898–1902.
29. FitzGerald, T., Dolan, R., Friston, K., Model averaging, optimal inference, and habit formation. Front. Hum. Neurosci., 2014, 10.3389/fnhum.2014.00457.
30. FitzGerald, T.H., Dolan, R.J., Friston, K., Dopamine, reward learning, and active inference. Front. Comput. Neurosci., 9, 2015, 136.
31. FitzGerald, T.H., Schwartenbeck, P., Moutoussis, M., Dolan, R.J., Friston, K., Active inference, evidence accumulation, and the urn task. Neural Comput. 27:2 (2015), 306–328.
32. Frank, M.J., Scheres, A., Sherman, S.J., Understanding decision-making deficits in neurological conditions: insights from models of natural action selection. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362:1485 (2007), 1641–1654.
33. Friston, K., Buzsaki, G., The functional anatomy of time: what and when in the brain. Trends Cogn. Sci. 20:July 7 (2016), 500–511.
34. Friston, K., Adams, R., Montague, R., What is value—accumulated reward or evidence?. Front. Neurorob., 6, 2012, 11.
35. Friston, K., Adams, R.A., Perrinet, L., Breakspear, M., Perceptions as hypotheses: saccades as experiments. Front. Psychol., 3, 2012, 151.
36. Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T., Raymond, R.J., Dolan, J., The anatomy of choice: active inference and agency. Front. Hum. Neurosci., 7, 2013, 598.
37. Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T., Dolan, R.J., The anatomy of choice: dopamine and decision-making. Philos. Trans. R. Soc. Lond. B Biol. Sci., 369(1655), 2014.
38. Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., Pezzulo, G., Active inference and epistemic value. Cogn. Neurosci., 2015, 1–28.
39. Friston, K., Life as we know it. J. R. Soc. Interface, 10(86), 2013, 20130475.
40. Gläscher, J., Daw, N., Dayan, P., O'Doherty, J.P., States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:4 (2010), 585–595.
41. Howard, R.A., Dynamic Programming and Markov Processes. 1960, MIT Press, Cambridge, MA.
42. Howard, R., Information value theory. IEEE Trans. Syst. Sci. Cybern. SSC-2:1 (1966), 22–26.
43. Humphries, M.D., Wood, R., Gurney, K., Dopamine-modulated dynamic cell assemblies generated by the GABAergic striatal microcircuit. Neural Netw. 22:8 (2009), 1174–1188.
44. Humphries, M.D., Khamassi, M., Gurney, K., Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Front. Neurosci., 6(9), 2012.
45. Huys, Q.J., Moutoussis, M., Williams, J., Are computational models of any use to psychiatry?. Neural Netw. 24:6 (2011), 544–551.
46. Itti, L., Baldi, P., Bayesian surprise attracts human attention. Vision Res. 49:10 (2009), 1295–1306.
47. Jaynes, E.T., Information theory and statistical mechanics. Phys. Rev. Ser. II 106:4 (1957), 620–630.
48. Kahneman, D., Tversky, A., Prospect theory: an analysis of decision under risk. Econometrica 47:2 (1979), 263–291.
49. Keramati, M., Dezfouli, A., Piray, P., Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol., 7(5), 2011, e1002055.
50. Kesner, R.P., Behavioral analysis of the contribution of the hippocampus and parietal cortex to the processing of information: interactions and dissociations. Hippocampus 10:4 (2000), 483–490.
51. Klyubin, A.S., Polani, D., Nehaniv, C.L., Empowerment: a universal agent-centric measure of control. Proceedings of CEC 2005. IEEE 1 (2005), 128–135.
52. Knutson, B., Bossaerts, P., Neural antecedents of financial decisions. J. Neurosci. 27:31 (2007), 8174–8177.
53. Krebs, R.M., Schott, B.H., Schütze, H., Düzel, E., The novelty exploration bonus and its attentional modulation. Neuropsychologia 47 (2009), 2272–2281.
54. Laughlin, S.B., Efficiency and complexity in neural coding. Novartis Found. Symp. 239 (2001), 177–187.
55. Lee, S.W., Shimojo, S., O'Doherty, J.P., Neural computations underlying arbitration between model-based and model-free learning. Neuron 81:3 (2014), 687–699.
56. Linsker, R., Perceptual neural organization: some approaches based on network models and information theory. Annu. Rev. Neurosci. 13 (1990), 257–281.
57. Mannella, F., Baldassarre, G., Selection of cortical dynamics for motor behaviour by the basal ganglia. Biol. Cybern. 109:6 (2015), 575–595.
58. Mirza, M.B., Adams, R.A., Mathys, C.D., Friston, K.J., Scene construction, visual foraging and active inference. Front. Comput. Neurosci., 10, 2016.
59. Montague, P.R., Dolan, R.J., Friston, K.J., Dayan, P., Computational psychiatry. Trends Cogn. Sci. 16:1 (2012), 72–80.
60. Moser, E.I., Kropff, E., Moser, M.B., Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci. 31 (2008), 69–89.
61. Moutoussis, M., Trujillo-Barreto, N.J., El-Deredy, W., Dolan, R.J., Friston, K.J., A formal model of interpersonal inference. Front. Hum. Neurosci., 8, 2014, 160.
62. Mushiake, H., Saito, N., Sakamoto, K., Itoyama, Y., Tanji, J., Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 50 (2006), 631–641.
63. Oliehoek, F., Spaan, M.T.J., Vlassis, N., Best-response play in partially observable card games. Proceedings of the 14th Annual Machine Learning Conference of Belgium and the Netherlands, 2005.
64. Olshausen, B.A., Field, D.J., Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381 (1996), 607–609.
65. Ortega, P.A., Braun, D.A., Thermodynamics as a theory of decision-making with information-processing costs. Proc. R. Soc. A, 469(2153), 2013.
66. Pearson, J.M., Watson, K.K., Platt, M.L., Decision making: the neuroethological turn. Neuron 82:5 (2014), 950–965.
67. Penny, W., Zeidman, P., Burgess, N., Forward and backward inference in spatial cognition. PLoS Comput. Biol., 9(12), 2013, e1003383.
68. Pennartz, C., Ito, R., Verschure, P., Battaglia, F., Robbins, T., The hippocampal–striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci. 34 (2011), 548–559.
69. Pezzulo, G., Rigoli, F., Chersi, F., The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front. Psychol., 4(92), 2013.
70. Pezzulo, G., van der Meer, M., Lansink, C., Pennartz, C., Internally generated sequences in learning and executing goal-directed behavior. Trends Cogn Sci. 18:December 12 (2014), 647–657, 10.1016/j.tics.2014.06.011.
71. Pezzulo, G., Rigoli, F., Friston, K., Active Inference, homeostatic regulation and adaptive behavioural control. Prog. Neurobiol. 134 (2015), 17–35.
72. Pezzulo, G., Cartoni, E., Rigoli, F., Pio-Lopez, L., Friston, K., Active Inference, epistemic value, and vicarious trial and error. Learn. Mem. 23 (2016), 322–338.
73. Pezzulo, G., Cisek, P., Navigating the affordance landscape: feedback control as a process model of behavior and cognition. Trends Cogn. Sci. 20:6 (2016), 414–424.
74. Preuschoff, K., Quartz, S.R., Bossaerts, P., Human insula activation reflects risk prediction errors as well as risk. J. Neurosci. 28:11 (2008), 2745–2752.
75. Ravindran, B., Relativized hierarchical decomposition of Markov decision processes. Prog. Brain Res. 202 (2013), 465–488.
76. Schmidhuber, J., Curious model-building control systems. Proceedings of International Joint Conference on Neural Networks, Singapore. IEEE 2 (1991), 1458–1463.
77. Schultz, W., Dickinson, A., Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23 (2000), 473–500.
78. Schwartenbeck, P., Fitzgerald, T., Dolan, R.J., Friston, K., Exploration, novelty, surprise, and free energy minimization. Front. Psychol., 4, 2013, 710.
79. Schwartenbeck, P., FitzGerald, T.H., Mathys, C., Dolan, R., Friston, K., The dopaminergic midbrain encodes the expected certainty about desired outcomes. Cereb. Cortex 25:10 (2015), 3434–3445.
80. Schwartenbeck, P., FitzGerald, T.H., Mathys, C., Dolan, R., Kronbichler, M., Friston, K., Evidence for surprise minimization over value maximization in choice behavior. Sci. Rep., 5, 2015, 16575.
81. Schwartenbeck, P., FitzGerald, T.H., Mathys, C., Dolan, R., Wurst, F., Kronbichler, M., Friston, K., Optimal inference with suboptimal models: addiction and active Bayesian inference. Med. Hypotheses 84:2 (2015), 109–117.
82. Sella, G., Hirsh, A.E., The application of statistical physics to evolutionary biology. Proc. Natl. Acad. Sci. U. S. A. 102 (2005), 9541–9546.
83. Solway, A., Botvinick, M., Goal-directed decision making as probabilistic inference: a com-putational framework and potential neural correlates. Psychol. Rev. 119 (2012), 120–154.
84. Still, S., Precup, D., An information-theoretic approach to curiosity-driven reinforcement learning. Theory Biosci. 131:3 (2012), 139–148.
85. Stoianov, I., Genovesio, A., Pezzulo, G., Prefrontal goal-codes emerge as latent states in probabilistic value learning. J. Cogn. Neurosci. 28:1 (2016), 140–157.
86. Thrailkill, E.A., Bouton, M.E., Contextual control of instrumental actions and habits. J. Exp. Psychol. Anim. Learn. Cogn. 41:1 (2015), 69–80.
87. van den Broek, J.L., Wiegerinck, W.A.J.J., Kappen, H.J., Risk-sensitive path integral control. Uai 6 (2010), 1–8.
88. van der Meer, M., Kurth-Nelson, Z., Redish, A.D., Information processing in decision-making systems. Neuroscientist 18:4 (2012), 342–359.
89. Verschure, P.F.M.J., Pennartz, C.M.A., Pezzulo, G., The why, what, where, when and how of goal-directed choice: neuronal and computational principles. Philos. Trans. R. Soc. B: Biol. Sci., 369(1655), 2014, 20130483.
90. Wang, X.J., Krystal, J.H., Computational psychiatry. Neuron 84:3 (2014), 638–654.
91. Williams, R.J., Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8 (1992), 229–256.
92. Wittmann, B.C., Daw, N.D., Seymour, B., Dolan, R.J., Striatal activity underlies novelty-based choice in humans. Neuron 58:6 (2008), 967–973.
93. Yin, H.H., Knowlton, B.J., The role of the basal ganglia in habit formation. Nat. Rev. Neurosci. 7:6 (2006), 464–476.
94. Zak, P.J., Neuroeconomics. Philos. Trans. R. Soc. Lond. B Biol. Sci. 359:1451 (2004), 1737–1748.