Optimal control as a graphical model inference problem

Machine Learning

Volumn 87, Issue 2, 2012, Pages 159-182

  Kappen, Hilbert J ^a   Gómez, Vicenç ^a   Opper, Manfred ^b  

^a RADBOUD UNIVERSITY   (Netherlands)

^b TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

Author keywords

Approximate inference; Belief propagation; Cluster variation method; Graphical model; Kullback leibler divergence; Optimal control; Uncontrolled dynamics

Indexed keywords

APPROXIMATE INFERENCE; BELIEF PROPAGATION; CLUSTER VARIATION METHODS; GRAPHICAL MODEL; KULLBACK LEIBLER DIVERGENCE; OPTIMAL CONTROLS; UNCONTROLLED DYNAMICS;

GRAPHIC METHODS; OPTIMAL CONTROL SYSTEMS;

CONTROL;

EID: 84862024986     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-012-5278-7     Document Type: Article

References (40)

1. Albers, C. A., Heskes, T., & Kappen, H. J. (2007). Haplotype inference in general pedigrees using the cluster variation method. Genetics, 177(2), 1101-1118.
2. Albers, C. A., Leisink, M. A. R., & Kappen, H. J. (2006). The cluster variation method for efficient linkage analysis on extended pedigrees. BMC Bioinformatics, 7 (S-1).
3. Bagnell, J. A., & Schneider, J. (2003). Covariant policy search. In IJCAI'03: Proceedings of the 18th international joint conference on artificial intelligence (pp. 1019-1024). San Francisco: Morgan Kaufmann.
4. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Belmont: Athena Scientific.
5. Bierkens, J., & Kappen, B. (2012). Kl-learning: Online solution of Kullback-Leibler control problems. http://arxiv.org/abs/1112.1996.
6. Boutilier, C., Dearden, R., & Goldszmidt, M. (1995). Exploiting structure in policy construction. In IJCAI'95: Proceedings of the 14th international joint conference on artificial intelligence (pp. 1104-1111). San Francisco: Morgan Kaufmann.
7. Cooper, G. (1988). A method for using belief networks as influence diagrams. In Proceedings of the workshop on uncertainty in artificial intelligence (UAI'88) (pp. 55-63).
8. da Silva, M., Durand, F., & Popović, J. (2009). Linear Bellman combination for control of character animation. ACM Transactions on Graphics, 28(3), 82:1-82:10.
9. Dayan, P., & Hinton, G. E. (1997). Using expectation-maximization for reinforcement learning. Neural Computation, 9(2), 271-278. (Pubitemid 127635391)
10. Friston, K. J., Daunizeau, J., & Kiebel, S. J. (2009). Reinforcement learning or active inference? PLoS ONE, 4(7), e6421.
11. Heskes, T., Albers, K., & Kappen, H. J. (2003). Approximate inference and constrained optimization. In Proceedings of the 19th conference on uncertainty in artificial intelligence (UAI'03), Acapulco, Mexico, (pp. 313-320). San Francisco: Morgan Kaufmann.
12. Jordan, M. I. (Ed.) (1999). Learning in graphical models. Cambridge: MIT Press.
13. Kappen, H. J. (2005). Linear theory for control of nonlinear stochastic systems. Physical Review Letters, 95(20), 200201.
14. Kappen, H. J., & Wiegerinck, W. (2002). Novel iteration schemes for the cluster variation method. In Advances in neural information processing systems (Vol. 14, pp. 415-422). Cambridge: MIT Press.
15. Kober, J., & Peters, J. (2011). Policy search for motor primitives in robotics. Machine Learning, 84(1-2), 171-203.
16. Koller, D., & Parr, R. (1999). Computing factored value functions for policies in structured mdps. In IJCAI'99: Proceedings of the 16th international joint conference on artificial intelligence (pp. 1332-1339). San Francisco: Morgan Kaufmann.
17. Lauritzen, S. L., & Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems. Journal of the Royal Statistical Society. Series B. Methodological, 50(2), 154-227.
18. Mooij, J. M. (2010). libDAI: A free and open source C++ library for discrete approximate inference in graphical models. Journal of Machine Learning Research, 11, 2169-2173.
19. Murphy, K., Weiss, Y., & Jordan, M. (1999). Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the 15th conference on uncertainty in artificial intelligence (UAI'99) (pp. 467-475). San Francisco: Morgan Kaufmann.
20. Peters, J., Mülling, K., & Altün, Y. (2010). Relative entropy policy search. In Proceedings of the 24th AAAI conference on artificial intelligence (AAAI 2010) (pp. 1607-1612). Menlo Park: AAAI Press.
21. Russell, S. J., Norvig, P., Candy, J. F., Malik, J. M., & Edwards, D. D. (1996). Artificial intelligence: a modern approach. Upper Saddle River: Prentice-Hall, Inc.
22. Shachter, R. D., & Peot, M. A. (1992). Decision making using probabilistic inference methods. In Proceedings of the 8th conference on uncertainty in artificial intelligence (UAI'92) (pp. 276-283). San Francisco: Morgan Kaufmann.
23. Skyrms, B. (1996). Evolution of the social contract. Cambridge: Cambridge University Press.
24. Skyrms, B. (Ed.) (2004). The stag hunt and evolution of social structure. Cambridge: Cambridge University Press.
25. Stengel, R. F. (1994). Optimal control and estimation. New York: Dover Publications, Inc.
26. Tatman, J., & Shachter, R. (1990). Dynamic programming and influence diagrams. IEEE Transactions on Systems, Man, and Cybernetics, 20(2), 365-379. (Pubitemid 20702819)
27. Theodorou, E. A., Buchli, J., & Schaal, S. (2009). Path integral-based stochastic optimal control for rigid body dynamics. In Adaptive dynamic programming and reinforcement learning, 2009. ADPRL'09. IEEE symposium on (pp. 219-225).
28. Theodorou, E. A., Buchli, J., & Schaal, S. (2010a). Learning policy improvements with path integrals. In International conference on artificial intelligence and statistics (AISTATS 2010).
29. Theodorou, E. A., Buchli, J., & Schaal, S. (2010b). Reinforcement learning of motor skills in high dimensions: A path integral approach. In Proceedings of the international conference on robotics and automation (ICRA 2010) (pp. 2397-2403). New York: IEEE Press.
30. Todorov, E. (2007). Linearly-solvable Markov decision problems. In Advances in neural information processing systems (Vol. 19, pp. 1369-1376). Cambridge: MIT Press.
31. Todorov, E. (2008). General duality between optimal control and estimation. In 47th IEEE conference on decision and control (pp. 4286-4292).
32. Todorov, E. (2009). Efficient computation of optimal actions. Proceedings of the National Academy of Sciences of the United States of America, 106(28), 11478-11483.
33. Toussaint, M., & Storkey, A. (2006). Probabilistic inference for solving discrete and continuous state Markov decision processes. In ICML'06: Proceedings of the 23rd international conference on machine learning (pp. 945-952). New York: ACM.
34. van den Broek, B., Wiegerinck, W., & Kappen, H. J. (2008a). Graphical model inference in optimal control of stochastic multi-agent systems. Journal of Artificial Intelligence Research, 32(1), 95-122.
35. van den Broek, B., Wiegerinck, W., & Kappen, H. J. (2008b). Optimal control in large stochastic multi-agent systems. Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning, 4865, 15-26.
36. Wiegerinck, W., van den Broek, B., & Kappen, H. J. (2006). Stochastic optimal control in continuous spacetime multi-agent systems. In Proceedings of the 22nd conference on uncertainty in artificial intelligence (UAI'06), Arlington, Virginia (pp. 528-535). Corvallis: AUAI Press.
37. Wiegerinck, W., van den Broek, B., & Kappen, H. J. (2007). Optimal on-line scheduling in stochastic multiagent systems in continuous space and time. In Proceedings of the 6th international joint conference on autonomous agents and multiagent systems AAMAS 07 (pp. 749-756).
38. Yedidia, J., Freeman, W., & Weiss, Y. (2001). Generalized belief propagation. In T. K. Leen, T. G. Dieterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 13, pp. 689-995). Cambridge: MIT Press.
39. Yedidia, J., Freeman, W., & Weiss, Y. (2005). Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51(7), 2282-2312. (Pubitemid 41136394)
40. Yoshida, W., Dolan, R. J., & Friston, K. J. (2008). Game theory of mind. PLoS Computational Biology, 4(12), e1000254.