Active inference and agency: Optimal control without cost functions

Biological Cybernetics

Volumn 106, Issue 8-9, 2012, Pages 523-541

  Friston, Karl ^a   Samothrakis, Spyridon ^b   Montague, Read ^c  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF ESSEX   (United Kingdom)

^c VIRGINIA TECH CARILION SCHOOL OF MEDICINE   (United States)

Author keywords

Action; Agency; Bayesian; Free energy; Inference; Optimal control; Partially observable Markov decision processes

Indexed keywords

ACTION; AGENCY; BAYESIAN; INFERENCE; OPTIMAL CONTROLS; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS;

COST FUNCTIONS; FREE ENERGY;

CONTROL;

ALGORITHM; ARTICLE; BAYES THEOREM; BIOLOGICAL MODEL; DECISION MAKING; PHYSIOLOGY; PROBABILITY;

ALGORITHMS; BAYES THEOREM; DECISION MAKING; MARKOV CHAINS; MODELS, NEUROLOGICAL;

EID: 84867578045     PISSN: 03401200     EISSN: 14320770     Source Type: Journal    
DOI: 10.1007/s00422-012-0512-8     Document Type: Article

References (80)

1. Ashby WR (1947) Principles of the self-organizing dynamic system. J Gen Psychol 37:125-128
2. Axmacher N, Henseler MM, Jensen O, Weinreich I, Elger CE, Fell J (2010) Cross-frequency coupling supports multi-itemworking memory in the human hippocampus. Proc Natl Acad Sci 107(7):3228-3233
3. Baxter J, Bartlett PL, Weaver L (2001) Experiments with Infinite- Horizon, Policy-Gradient Estimation. J Artif Intell Res 15:351-381
4. Beal MJ (2003) Variational algorithms for approximate bayesian inference'. PhD. Thesis, University College London, London
5. Bellman R (1952) On the theory of dynamic programming. Proc Natl Acad Sci USA 38:716-719
6. Berridge KC (2004) Motivation concepts in behavioral neuroscience. Physiol Behav 81(2):179-209
7. Birkhoff GD (1931) Proof of the ergodic theorem. Proc Natl Acad Sci USA 17:656-660
8. BotvinickMM,AnJ (2008) Goal-directed decision making in prefrontal cortex: a computational framework. Adv Neural Inf Process Syst (NIPS) 21
9. BraunDA, Ortega P, Theodorou E, Schaal S (2011) Path integral control and bounded rationality. In: ADPRL 2011, Paris
10. Brown LD (1981) A complete class theorem for statistical problems with finite sample spaces. Ann Stat 9(6):1289-1300
11. Camerer CF (2003) Behavioural studies of strategic thinking in games. Trends Cogn Sci 7(5):225-231
12. Canolty RT, Edwards E, Dalal SS, SoltaniM,Nagarajan SS,KirschHE, Berger MS, Barbaro NM, Knight R (2006) High gamma power is phase-locked to theta oscillations in human neocortex. Science 313(5793):1626-1628
13. Cooper G (1988) A method for using belief networks as influence diagrams. In: Proceedings of the Conference on uncertainty in artificial intelligence
14. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16(2):199-204
15. Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8(4):429-453
16. Dayan P, Hinton GE (1997) Using expectation maximization for reinforcement learning. Neural Comput 9:271-278
17. Dayan P, Hinton GE, Neal R (1995) The Helmholtz machine. Neural Comput 7:889-904
18. Duff M, (2002) Optimal learning: computational procedure for bayes- adaptive markov decision processes. PhD thesis. University of Massachusetts, Amherst
19. Evans DJ (2003) A non-equilibrium free energy theorem for deterministic systems. Mol Phys 101:15551-15554
20. Feldbaum AA (1961) Dual control theory, Part I. Autom Remote Control 21(9):874-880
21. Feldman H, Friston KJ (2010) Attention, uncertainty, and free-energy. Front Hum Neurosci 4:215
22. Feynman RP (1972) Statistical mechanics. Benjamin, Reading M.A.
23. Filatov N, Unbehauen H (2004) Adaptive dual control: theory and applications (lecture notes in control and information sciences. Springer, Berlin
24. Fox C, Roberts S (2011) A tutorial on variational Bayes. In: Artificial intelligence review. Spinger, Berlin
25. Friston K (2008) Hierarchical models in the brain. PLoS Comput Biol 4(11):e1000211
26. FristonK (2010) The free-energy principle: a unified brain theory?. Nat Rev Neurosci 11(2):127-138
27. Friston K (2011) What is optimal about motor control?. Neuron 72(3):488-498
28. Friston K, Ao P (2012) Free-energy, value and attractors. In: Computational and mathematical methods in medicine, vol 2012
29. Friston K, Kiebel S (2009) Cortical circuits for perceptual inference. Neural Netw 22(8):1093-1104
30. Friston K, Kiebel S (2009) Predictive coding under the free-energy principle. Philos Trans R Soc Lond B Biol Sci 364(1521):1211-1221
31. Friston KJ, Daunizeau J, Kiebel SJ (2009) Active inference or reinforcement learning?. PLoS One 4(7):e6421
32. Friston KJ, Daunizeau J, Kilner J, Kiebel SJ (2010) Action and behavior: a free-energy formulation. Biol Cybern 102(3): 227-260
33. Friston KJST, Fitzgerald T, Galea JM, Adams R, Brown H, Dolan RJ, Moran R, Stephan KE, Bestmann S (2012) Dopamine, affordance and active inference. PLoS Comput Biol 8(1):e1002327
34. Friston K, Kilner J, Harrison L (2006) A free energy principle for the brain. J Physiol Paris 100(1-3):70-87
35. Friston K, Mattout J, Kilner J (2011) Action understanding and active inference. Biol Cybern 104:137-160
36. Friston KJ, Tononi G, Reeke GNJ, Sporns O, Edelman GM (1994) Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience 59(2):229-243
37. Gigerenzer G, Gaissmaier W (2011) Heuristic decision making. Annu Rev Psychol 62:451-482
38. Gläscher J, Daw N, Dayan P, O'Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4):585-595
39. Gomez F, Miikkulainen R (2001) Learning robust nonlinear control with neuroevolution. Technical Report AI01-292, Department of Computer Sciences, The University of Texas at Austin
40. Gomez F, Schmidhuber J, Miikkulainen R (2009) Accelerated neural evolution through cooperatively coevolved synapses. J Mach Learn Res 9:937-965
41. Helmholtz H (1866/1962), Concerning the perceptions in general. In: Treatise on physiological optics, 3rd edn. Dover, New York
42. Hinton GE, van Camp D (1993) Keeping neural networks simple by minimizing the description length of weights. In: Proceedings of COLT-93,pp 5-13
43. Hoffman, M, de Freitas, N, Doucet, A, Peters J (2009) An expectation maximization algorithm for continuous markov decision processes with arbitrary rewards. In: Twelfth Int. Conf. on artificial intelligence and statistics (AISTATS 2009)
44. HowardRA(1960) Dynamic programming andMarkov processes. MIT Press Cambridge, M.A.
45. Jaeger H (2000) Observable operator models for discrete stochastic time series. Neural Comput 12:1371-1398
46. Jensen F, Jensen V, Dittmer SL (1994) From influence diagrams to junction trees. In: Proc. of the Tenth Conference on uncertainty in artificial intelligence. Morgan Kaufmann, San Fransisco
47. Kaelbling LP, Littman ML, Cassandra AR (1998) Planning and acting in partially observable stochastic domains. Artif Intell 101 (1-2):99-134
48. KappenHJ (2005) Linear theory for control of nonlinear stochastic systems. Phys Rev Lett 95(20):200201
49. Kappen HJ (2005) Path integrals and symmetry breaking for optimal control theory. J Stat Mech: Theory Exp 11:P11011
50. Kappen HJ, Gomez Y, Opper M (2009) Optimal control as a graphical model inference problem. arXiv:0901.0633v2
51. Kiebel SJ, Daunizeau J, Friston KJ (2009a) Perception and hierarchical dynamics. Front Neuroinf 3:20
52. Kiebel SJ, von Kriegstein K, Daunizeau J, Friston KJ (2009b) Recognizing sequences of sequences. PLoS Comput Biol 5(8):e1000464
53. Kishida KT, King-Casas B, Montague PR (2010) Neuroeconomic approaches to mental disorders. Neuron 67(4):543-554
54. LittmanML, Majercik SM, Pitassi T (2001) Stochastic boolean satisfiability. J Autom Reason 27(3):251-296
55. Littman ML, Sutton RS, Singh S (2002) Predictive Representations of State. Adv Neural Inf Process Syst 14
56. MacKay DJ (1995) Free-energy minimisation algorithm for decoding and cryptoanalysis. Electron Lett 31:445-447
57. Montague PR,Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377(6551):725-728
58. Moutoussis M, Bentall RP, El-Deredy W, Dayan P (2011) Bayesian modelling of Jumping-to-conclusions bias in delusional patients. Cogn Neuropsychiatry 7:1-26
59. Namikawa J, Nishimoto R, Tani J (2011) A neurodynamic account of spontaneous behaviour. PLoS Comput Biol. 7(10):e1002221
60. Neal RM, Hinton GE (1998) A view of the EM algorithm that justifies incremental sparse and other variants. In: JordanM(ed.) Learning in graphical models. Kluwer Academic, Dordrecht
61. Oliehoek F, Spaan MTJ, Vlassis N (2005) Best-response play in partially observable card games. In: Proceedings of the 14th Annual Machine Learning Conference of Belgium and the Netherlands
62. Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann, San Fransisco
63. Rao RP (2010) Decision making under uncertainty: a neural model based on partially observable markov decision processes. Front Comput Neurosci 4:146
64. Rao RP, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2(1):79-87
65. Rawlik K, Toussaint M, Vijayakumar S (2010) Approximate inference and stochastic optimal control. arXiv:1009.3958
66. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: A Black, W Prokasy (eds.) Classical conditioning II: current research and theory. Appleton Century Crofts, New York
67. RobertC(1992)L'analyse statistique Bayesienne. In: Economica. Paris, France
68. Shachter RD (1988) Probabilistic inference and influence diagrams. Operat Res 36:589-605
69. Silver D, Veness J (2010) Monte-Carlo planning in large POMDPs. In: Proceedings of the Conference on neural information processing systems
70. Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88(2):135-170
71. Tani J (2003) Learning to generate articulated behavior through the bottom-up and the top-down interaction processes. Neural Netw 16(1):11-23
72. Theodorou E, Buchli J, Schaal S (2010) A generalized path integral control approach to reinforcement learning. J Mach Learn Res 11:3137-3181
73. Todorov E (2006) Linearly-solvable Markov decision problems. In: Advances in neural information processing systems. MIT Press, Boston
74. Todorov E (2008) General duality between optimal control and estimation. In: IEEE Conference on decision and control
75. Toussaint M, Charlin L, Poupart P (2008) Hierarchical POMDP controller optimization by likelihood maximization. In: Uncertainty in artificial intelligence (UAI 2008), AUAI Press, Menlo Park
76. Toussaint M, Storkey A (2006) Probabilistic inference for solving discrete and continuous stateMarkov decision processes. In: Proceedings of the 23nd International Conference on machine learning
77. Van den Broek B, Wiegerinck W, Kappen B (2008) Graphical model inference in optimal control of stochastic multi-agent systems. J Artif Int Res 32(1):95-122
78. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8:279-292
79. WilliamsRJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8:229-256
80. Zhang NL (1998) Probabilistic inference in influence diagrams. Comput Intell 14(4):475-497