Constructing free-energy approximations and generalized belief propagation algorithms

IEEE Transactions on Information Theory

Volumn 51, Issue 7, 2005, Pages 2282-2312

  Yedidia, Jonathan S ^a   Freeman, William T ^b   Weiss, Yair ^c  

^a MITSUBISHI ELECTRIC RESEARCH LABORATORIES   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

Belief propagation (BP); Bethe free energy; Cluster variation method; Generalized belief propagation (GBP); Kikuchi free energy; Message passing; Sum product algorithm

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; GRAPH THEORY; LAGRANGE MULTIPLIERS; MATHEMATICAL MODELS; MATRIX ALGEBRA; PROBABILITY DISTRIBUTIONS; RANDOM PROCESSES;

BETHE FREE ENERGY; CLUSTER VARIATION METHOD; GENERALIZED BELIEF PROPAGATION (GBP); KIKUCHI FREE ENERGY; MESSAGE PASSING ALGORITHMS; SUM-PRODUCT ALGORITHM;

INFORMATION THEORY;

EID: 23744513375     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.850085     Document Type: Article

References (59)

1. B. J. Frey, Graphical Models for Machine Learning and Digital Communication. Cambridge, MA: MIT Press, 1998.
2. Learning in Graphical Models, 1998.
3. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb, 1989.
4. A. J. Viterbi, "Error bounds for convolutional codes and an asymptotically optimum decoding algorithm," IEEE Trans. Inf. Theory, vol. IT-13, no. 2, pp. 260-269, Apr. 1967.
5. G. D. Forney Jr., "The Viterbi algorithm," Proc. IEEE, vol. 61, no. 3, pp. 268-278, Mar. 1973.
6. R. G. Gallager, Low-Density Parity Check Codes. Cambridge, MA: MIT Press, 1963.
7. C. Berrou, A. Glavieux, and P. Thitimajshima, "Near Shannon limit error-correcting coding and decoding: Turbo-codes," in Proc. 1993 IEEE Int. Conf. Communications, Geneva, Switzerland, 1993, pp. 1064-1070.
8. R. J. McEliece, D. J. C. MacKay, and J. F. Cheng, "Turbo decoding as an instance of Pearl's 'belief propagation' algorithm," IEEE J. Sel. Areas Commun., vol. 16, no. 2, pp. 140-152, Feb. 1998.
9. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco, CA: Morgan Kaufmann, 1988.
10. R. E. Kalman, "A new approach to linear filtering and prediction problems," J. Basic Eng., vol. 82, pp. 34-45, 1960.
11. A. Gelb, Ed., Applied Optimal Estimation. Cambridge, MA: MIT Press, 1974.
12. R. J. Baxter, Exactly Solved Models in Statistical Mechanics. New York: Academic, 1982.
13. R. Cowell, "Advanced inference in Bayesian networks" in Learning in Graphical Models, M. I. Jordan, Ed. Cambridge, MA: MIT Press, 1998.
14. K. P. Murphy, Y. Weiss, and M. I. Jordan, "Loopy belief propagation for approximate inference: An empirical study," in Proc. 15th Conf. Uncertainty in Artificial Intelligence, K. B. Laskey and H. Prade, Eds., Stockholm, Sweden, Jul. 1999, pp. 467-475.
15. H. A. Bethe, "Statistical theory of superlattices," in Proc. Roy. Soc. London A, 1935, p. 552.
16. R. Kikuchi, "A theory of cooperative phenomena," Phys. Rev., vol. 81, no. 6, pp. 988-1003, 1951.
17. J. S. Yedidia, W. T. Freeman, and Y. Weiss, "Generalized belief propagation," in Advances in Neural Information Processing Systems, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds. Cambridge, MA: MIT Press, 2001, vol. 13, pp. 689-695.
18. J. Hijmans and J. De Boer, "An approximation method for order-disorder problems I, II, III," Physica, vol. XXI, pp. 471-516, 1955.
19. T. Morita, "Cluster variation method for nonuniform Ising and Heisenberg models and spin-pair correlation function," Progr. Theor. Phys., vol. 85, no. 2, pp. 243-255, 1991.
20. T. Morita, M. Suzuki, K. Wada, and M. Kaburagi, Eds., "Foundations and applications of cluster variation method and path probability method, in Progr. Theor. Phys. Suppl., 1994, vol. 115.
21. F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, "Factor graphs and the sum-product algorithm," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 498-519, Feb. 2001.
22. G. D. Forney Jr., "Codes on graphs: Normal realizations," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 520-548, Feb. 2001.
23. T. P. Minka, "A Family of algorithms for approximate Bayesian inference.," Ph.D. dissertation, MIT, Cambridge, MA, 2001.
24. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky, "Tree-based reparameterization framework for analysis of sum-product and related algorithms," IEEE Trans. Inf. Theory, vol. 49, no. 5, pp. 1120-1146, May 2003.
25. S. M. Aji and R. J. McEliece, "The generalized distributive law, IEEE Trans. Inf. Theory, vol. 46, no. 2, pp. 325-343, Mar. 2000.
26. E. Offer and E. Soljanin, "An algebraic description of iterative decoding schemes," in Codes, Systems, and Graphical Models, Volumes in Mathematics, and Its Applications, B. Marcus and J. Rosenthal, Eds. New York: Springer Verlag, 2000.
27. A. Braunstein, M. Mezard, and R. Zecchina. (2002) Survey Propagation: An Algorithm for Satisfi Ability. [Online]. Available: http://xxx.lanl.gov/ pdf/cs/0212002
28. J. Dauwels, H.-A. Loeliger, P. Merkli, and M. Ostojic, "Structured-summary propagation, LFSR synchronization, and low-complexity trellis decoding," in Proc. 41st Annu. Allerton Conf. Communication, Control, and Computing, Monticello, IL, Oct. 2003, pp. 459-467.
29. S. M. Aji and R. J. McEliece, "The generalized distributive law and free energy minimization," in Proc. 39th Annu. Allerton Conf. Communication, Control, and Computing, Monticello, IL, Oct. 2001, pp. 672-681.
30. R. J. McEliece and M. Yildirim, "Belief propagation on partially ordered sets," in Mathematical Systems Theory in Biology, Communications, Computation, and Finance, D. Gilliam and J. Rosenthal, Eds. New York: Springer Verlag, 2002, pp. 275-299.
31. P. Pakzad and V. Anantharam, "Minimal graphical representation of Kikuchi regions," in Proc. 40th Annu. Allerton Conf. Communication, Control, and Computing, Monticello, IL, Oct. 2002, pp. 1585-1594.
32. P. Pakzad and V. Anantharam, (2003) Estimation and Marginalization Using Kikuchi Approximation Methods. [Online]. Available: http://www-inst.eecs. berkeley.edu/~payamp/papers/kikuchi.pdf
33. J. S. Yedidia, W. T. Freeman, and Y. Weiss. (2001) Characterizing Belief Propagation and Its Generalizations. [Online]. Available: http://www. merl.com/reports/TR2001-15/index.html
34. J. S. Yedidia, W. T. Freeman, and Y. Weiss, (2001) Bethe Free Energies, Kikuchi Approximations, and Belief Propagation Algorithms. [Online]. Available: http://www.merl.com/reports/TR2001-16/index.html
35. J. S. Yedidia, W. T. Freeman, and Y. Weiss, (2002) Constructing Free Energy Approximations and Generalized Belief Propagation Algorithms. [Online]. Available: http://www.merl.com/reports/TR2002-35/
36. J. S. Yedidia, W. T. Freeman, and Y. Weiss, "Understanding belief propagation and its generalizations" in Exploring Artificial Intelligence in the New Millenium, G. Lakemeyer and B. Nebel, Eds. San Francisco, CA: Morgan Kaufmann, 2003, ch. 8, pp. 239-269.
37. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
38. M. I. Jordan, Z. Ghahramani, T. Jaakkola, and L. Saul, "An introduction to variational methods for graphical models," in Learning in Graphical Models, M. I. Jordan, Ed. Cambridge, MA: MIT Press, 1998, pp. 105-161.
39. D. J. C. Mackay, J. S. Yedidia, W. T. Freeman, and Y. Weiss. (2001) A Conversation About the Bethe Free Energy and Sum-Product (Discussion Document). [Online]. Available: http://www.merl.com/reports/TR2001-18/ index.html
40. W. Wiegerinck and T. Heskes, "Fractional belief propagation," in Advances in Neural Information Processing Systems 15, S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, pp. 438-445.
41. M. J. Wainwright, T Jaakkola, and A. S. Willsky, "A new class of upper bounds on the log partition function," in Proc. Conf. Uncertainty in Artificial Intelligence, Edmonton, AB, Canada, Aug. 2002, pp. 536-543.
42. G. Toulouse, "Theory of the frustration effect in spin glasses: I," Commun. Phys., vol. 2, pp. 115-119, 1977.
43. M. J. Wainwright and M. I. Jordan, "Graphical Models, Exponential Families, and Variational Inference," Dept. Statistics, Univ. Calif., Berkeley, Tech. Rep. 649, 2003.
44. D. Bertsekas, Nonlinear Programming. Belmont, MA: Athena Scientific, 1999.
45. Y. Weiss, "Correctness of local probability propagation in graphical models with loops," Neural Comput., vol. 12, pp. 1-41, 2000.
46. K. Huang, Statistical Mechanics. New York: Wiley, 1963.
47. S. Tatikonda and M. I. Jordan, "Loopy belief propagation and Gibbs measures," in Proc. 40th Annu. Allerton Conf. Communication, Control, and Computing, Monticello, IL, Oct. 2002.
48. T. Heskes, "On the uniqueness of belief propagation fixed points," Neural Comput., vol. 16, no. 11, pp. 2379-2413, 2004.
49. A. L. Yuille, "CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation," Neural Comput., vol. 14, no. 7, pp. 1691-1722, 2002.
50. M. Welling and Y. W. Teh, "Belief optimization for binary networks: A stable alternative to belief propagation," in Proc. Conf. Uncertainty in Artificial Intelligence, Seattle, WA, Aug. 2001, pp. 554-561.
51. H. J. Kappen and W. Wiegerinck, "Novel iteration schemes for the cluster variation method," in Advances in Neural Information Processing Systems, T. Dietterich, S. Becker, and Z. Ghahramani, Eds. Cambridge, MA: MIT Press, 2002, vol. 14, pp. 415-422.
52. T. Heskes, "Stable fixed points of loopy belief propagation are local minima of the Bethe free energy," in Advances in Neural Information Processing Systems, S. Thrun, S. Becker, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, vol. 15, pp. 343-350.
53. R. P. Stanley, Enumerative Combinatorics. Cambridge, U.K.: Cambridge Univ. Press, 1997, vol. 1.
54. D. Sherrington and S. Kirpatrick, "Infinite-ranged models of spin-glasses," Phys. Rev. B, vol. 17, pp. 4384-4403, 1978.
55. T. Minka and Y. Qi, "Tree-structured approximations by expectation propagation," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2004, vol. 16.
56. M. Welling, "On the choice of regions for generalized belief propagation," in Proc. Conf. Uncertainty in Artificial Intelligence, Banff, Canada, Jul. 2004, pp. 585-594.
57. S. Fujiki and S. Katsura, "Frustration effect on the d-dimensional ising spin glass: II. Existence of the spin glass phase," J. Phys. C: Solid State Phys., vol. 13, pp. 4723-4734, 1980.
58. J. Harel, R. J. McEliece, and R. Palanki, "Poset belief propagation - Experimental results," in Proc. IEEE Int. Symp. Information Theory, Yokohama, Japan, Jun./Jul. 2003, p. 177.
59. J. Harel, "Poset belief propagation: Experimental results," Caltech senior thesis, Calif. Inst. Technol., Pasadena, 2003.