Convexity arguments for efficient minimization of the Bethe and Kikuchi free energies

Journal of Artificial Intelligence Research

Volumn 26, Issue , 2006, Pages 153-190

  Heskes, Tom ^a  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; MARKOV PROCESSES; OPTIMIZATION; RANDOM PROCESSES;

BAYESIAN NETWORKS; BELIEF PROPAGATION; CONVEX BOUNDS; KIKUCHI FREE ENERGY;

FREE ENERGY;

EID: 33749029684     PISSN: 10769757     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.1933     Document Type: Article

References (38)

1. Aji, S., & McEliece, R. (2001). The generalized distributive law and free energy minimization. In Proceedings of the Allerton Conference on Communication, Control, and Computing.
2. Besag, J. (1974). Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society Series B, 36, 192-236.
3. Chiang, M., & Forney, G. (2001). Statistical physics, convex optimization and the sum product algorithm. Tech. rep., Stanford University.
4. Darroch, J., & Ratcliff, D. (1972). Generalized iterative scaling. Annals of Mathematical Statistics, 43, 1470-1480.
5. Dechter, R., Kask, K., & Mateescu, R. (2002). Iterative join-graph propagation. In Darwiche, A., & Friedman, N. (Eds.), Proceedings UAI-2002, pp. 128-136.
6. Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39, 1-38.
7. Hall, P. (1935). On representatives of subsets. Journal of the London Mathematical Society, 10, 26-30.
8. Heckerman, D. (1989). A tractable inference algorithm for diagnosing multiple diseases. In Kanal, L., Henrion, M., Shachter, R., & Lemmer, J. (Eds.), Proceedings of the Fifth Workshop on Uncertainty in Artificial Intelligence, pp. 163-171, Amsterdam. Elsevier.
9. Heskes, T. (2003). Stable fixed points of loopy belief propagation are minima of the Bethe free energy. In Becker, S., Thrun, S., & Obermayer, K. (Eds.), Advances in Neural Information Processing Systems 15, pp. 359-366, Cambridge. MIT Press.
10. Heskes, T. (2004). On the uniqueness of loopy belief propagation fixed points. Neural Computation, 16, 2379-2413.
11. Heskes, T., Albers, K., & Kappen, B. (2003). Approximate inference and constrained optimization. In Uncertainty in Artificial Intelligence: Proceedings of the Nineteenth Conference (UAI-2003), pp. 313-320, San Francisco, CA. Morgan Kaufmann Publishers.
12. Heskes, T., Zoeter, O., & Wiegerinck, W. (2004). Approximate Expectation Maximization. In Thrun, S., Saul, L., & Schölkopf, B. (Eds.), Advances in Neural Information Processing Systems 16, pp. 353-360, Cambridge. MIT Press.
13. Ihler, A., Fisher, J., & Willsky, A. (2005). Loopy belief propagation: Convergence and effects of message errors. Journal of Machine Learning Research, 6, 905-936.
14. Jaakkola, T., & Jordan, M. (1999). Variational probabilistic inference and the QMR-DT network. Journal of Artificial Intelligence Research, 10, 291-299.
15. Jiroušek, R., & Přeučil, S. (1995). On the effective implementation of the iterative proportional fitting procedure. Computational Statistics and Data Analysis, 19, 177-189.
16. Jordan, M., Ghahramani, Z., Jaakkola, T., fe Saul, L. (1998). An introduction to variational methods for graphical models. In Jordan, M. (Ed.), Learning in Graphical Models, pp. 183-233. Kluwer Academic Publishers, Dordrecht.
17. Kikuchi, R. (1951). The theory of cooperative phenomena. Physical Review, 81, 988-1003.
18. Kschischang, F., Frey, B., & Loeliger, H. (2001). Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory, 47(2), 498-519.
19. Lauritzen, S. (1996). Graphical models. Oxford University Press, Oxford.
20. Luenberger, D. (1984). Linear and Nonlinear Programming. Addison-Wesley, Reading, Massachusetts.
21. McEliece, R., MacKay, D., & Cheng, J. (1998). Turbo decoding as as an instance of Pearl's 'belief propagation' algorithm. IEEE Journal on Selected Areas in Communication, 16(2), 140-152.
22. McEliece, R., & Yildirim, M. (2003). Belief propagation on partially ordered sets. In Gilliam, D., & Rosenthal, J. (Eds.), Mathematical Systems Theory in Biology, Communications, Computation, and Finance, pp. 275-300. Springer, New York.
23. Murphy, K. (2001). The Bayes Net toolbox for Matlab. Computing Science and Statistics, 33, 331-350.
24. Murphy, K., Weiss, Y., & Jordan, M. (1999). Loopy belief propagation for approximate inference: An empirical study. In Laskey, K., & Prade, H. (Eds.), Proceedings of the Fifteenth Conference on Uncertainty in Articial Intelligence, pp. 467-475, San Francisco, CA. Morgan Kaufmann Publishers.
25. Neal, R., & Hinton, G. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. In Jordan, M. (Ed.), Learning in Graphical Models, pp. 355-368. Kluwer Academic Publishers, Dordrecht.
26. Pakzad, P., & Anantharam, V. (2002). Belief propagation and statistical physics. In 2002 Conference on Information Sciences and Systems, Princeton University.
27. Pakzad, P., & Anantharam, V. (2005). Estimation and marginalization using Kikuchi approximation methods. Neural Computation, 17, 1836-1873.
28. Pearl, J. (1988). Probabilistic Reasoning in Intelligent systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco, CA.
29. Pelizzola, A. (2005). Cluster variation method in statistical physics and graphical models. Journal of Physics A, 38, R309-R339.
30. Seung, S., Richardson, T., Lagarias, J., &: Hopfield, J. (1998). Minimax and Hamiltonian dynamics of excitatory-inhibitory networks. In Jordan, M., Kearns, M., & Solla, S. (Eds.), Advances in Neural Information Processing Systems 10, pp. 329-335. MIT Press.
31. Tatikonda, S., & Jordan, M. (2002). Loopy belief propagation and Gibbs measures. In Darwiche, A., & Friedman, N. (Eds.), Uncertainty in Artificial Intelligence: Proceedings of the Eighteenth Conference (UAI-2002), pp. 493-500, San Francisco, CA. Morgan Kaufmann Publishers.
32. Teh, Y., &: Welling, M. (2002). The unified propagation and scaling algorithm. In Dietterich, T., Becker, S., & Ghahramani, Z. (Eds.), Advances in Neural Information Processing Systems 14, pp. 953-960, Cambridge. MIT Press.
33. Wainwright, M., Jaakkola, T., &: Willsky, A. (2002a). A new class of upper bounds on the log partition function. In Darwiche, A., &: Friedman, N. (Eds.), Uncertainty in Artificial Intelligence: Proceedings of the Eighteenth Conference (UAI-2002), pp. 536-543, San Francisco, CA. Morgan Kaufmann Publishers.
34. Wainwright, M., Jaakkola, T., & Willsky, A. (2002b). Tree-based reparameterization for approximate estimation on loopy graphs. In Dietterich, T., Becker, S., & Ghahramani, Z. (Eds.), Advances in Neural Information Processing Systems 14, pp. 1001-1008, Cambridge. MIT Press.
35. Wainwright, M., Jaakkola, T., & Willsky, A. (2003). Tree-reweighted belief propagation algorithms and approximate ML estimation via pseudo-moment matching. In Bishop, C., & Frey, B. (Eds.), Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics. Society for Artificial Intelligence and Statistics.
36. Yedidia, J., Freeman, W., & Weiss, Y. (2001). Generalized belief propagation. In Leen, T., Dietterich, T., & Tresp, V. (Eds.), Advances in Neural Information Processing Systems 13, pp. 689-695, Cambridge. MIT Press.
37. Yedidia, J., Freeman, W., & Weiss, Y. (2005). Constructing free energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51, 2282-2312.
38. Yuille, A. (2002). CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation. Neural Computation, 14, 1691-1722.