Tree-based reparameterization framework for analysis of sum-product and related algorithms

IEEE Transactions on Information Theory

Volumn 49, Issue 5, 2003, Pages 1120-1146

  Wainwright, Martin J ^a   Jaakkola, Tommi S ^b   Willsky, Alan S ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b Massachusetts Institute of Technology   (United States)

Author keywords

Approximate inference; Belief propagation (BP); Bethe Kikuchi free energies; Convex duality; Factor graphs; Graphical models; Hypergraphs; Information geometry; Iterative decoding; Junction tree; Markov random fields; Sum product algorithm

Indexed keywords

ALGORITHMS; DECODING; GRAPH THEORY; MARKOV PROCESSES; MATHEMATICAL MODELS; OPTIMIZATION; PROBABILITY DISTRIBUTIONS;

APPROXIMATE INFERENCE; BELIEF PROPAGATION; BETHE-KIKUCHI FREE ENERGIES; CONVEX DUALITY; FACTOR GRAPHS; HYPERGRAPHS; INFORMATION GEOMETRY; SUM-PRODUCT ALGORITHM; TREE-BASED REPARAMETERIZATION FRAMEWORK;

INFORMATION THEORY;

EID: 0037516827     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2003.810642     Document Type: Article

References (58)

1. J. Pearl, Probabilistic Reasoning in Intelligent Systems. San Mateo, CA: Morgan Kaufman, 1988.
2. R. J. McEliece, D. J. C. McKay, and J. F. Cheng, "Turbo decoding as an instance of Pearl's belief propagation algorithm," IEEE J. Select. Areas Commun., vol. 16, pp. 140-152, Feb. 1998.
3. J. S. Yedidia, W. T. Freeman, and Y. Weiss, "Generalized belief propagation," in NIPS 13. Cambridge, MA: MIT Press, 2001, pp. 686-695.
4. R. J. McEliece and M. Yildirim, "Belief propagation on partially ordered sets," in Mathematical Theory of Systems and Networks. Minneapolis, MN: Inst. Math. and Applic., Aug. 2002.
5. R. G. Gallager, Low-Density Parity Check Codes. Cambridge, MA: MIT Press, 1963.
6. F. Kschischang and B. Frey, "Iterative decoding of compound codes by probability propagation in graphical models," IEEE J. Select. Areas Commun., vol. 16, pp. 219-230, Feb. 1998.
7. M. Jordan, Learning in Graphical Models. Cambridge, MA: MIT Press, 1999.
8. R. J. Baxter, Exactly Solved Models in Statistical Mechanics. New York: Academic, 1982.
9. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, "Learning low-level vision," Intl. J. Computer Vision, vol. 40, no. 1, pp. 25-40, 2000.
10. G. Cooper, "The computational complexity of probabilistic inference using Bayesian belief networks," Artificial Intell., vol. 42, pp. 393-405, 1990.
11. Y. Weiss, "Correctness of local probability propagation in graphical models with loops," Neural Comput., vol. 12, pp. 1-4, 2000.
12. S. M. Aji and R. J. McEliece, "The generalized distributive law," IEEE Trans. Inform. Theory, vol. 46, pp. 325-343, Mar. 2000.
13. T. Richardson, "The geometry of turbo-decoding dynamics," IEEE Trans. Inform. Theory, vol. 46, pp. 9-23, Jan. 2000.
14. K. Murphy, Y. Weiss, and M. Jordan, "Loopy-belief propagation for approximate inference: An empirical study," Uncertainty in Artificial Intell., vol. 15, pp. 467-475, 1999.
15. J. B. Anderson and S. M. Hladnik, "Tailbiting map decoders," IEEE J. Select. Areas Commun., vol. 16, pp. 297-302, Feb. 1998.
16. S. M. Aji, G. Horn, and R. McEliece, "On the convergence of iterative decoding on graphs with a single cycle," in Proc. IEEE Intl. Symp. Information Theory, Cambridge, MA, Aug. 1998, p. 276.
17. G. D. Forney, Jr., F. R. Kschischang, B. Marcus, and S. Tuncel, "Iterative decoding of tail-biting trellises and connections with symbolic dynamics," in Codes, Systems and Graphical Models. New York: Springer, 2001, pp. 239-264
18. Y. Weiss and W. T. Freeman, "Correctness of belief propagation in Gaussian graphical models of arbitrary topology," in NIPS 12. Cambridge, MA: MIT Press, 2000, pp. 673-679.
19. P. Rusmevichientong and B. Van Roy, "An analysis of turbo decoding with Gaussian densities," in NIPS 12. Cambridge, MA: MIT Press, 2000, pp. 575-581.
20. K. Tanaka and T. Morita, "Cluster variation method and image restoration problem," Phys. Lett. A, vol. 203, pp. 122-128, 1995.
21. C. H. Wu and P. C. Doerschuk, "Tree approximations to Markov random fields," IEEE Trans. Pattern Anal. Machine Intell., vol. 17, pp. 391-402, Apr. 1995.
22. M. Welling and Y. Teh, "Belief optimization: A stable alternative to loopy belief propagation," Uncertainty in Artificial Intelligence, vol. 17, pp. 554-561, July 2001.
23. A. Yuille, "A double-loop algorithm to minimize the Bethe and Kikuchi free energies," Neural Comput., 2001, to be published.
24. R. Kikuchi, "The theory of cooperative phenomena," Phys. Rev., pp. 988-1003, 1951.
25. P. Pakzad and V. Anantharam, "Belief propagation and statistical physics," in Proc. Conf. Information Sciences and Systems, Princeton, NJ, Mar. 2002.
26. S. Tatikonda and M. I. Jordan, "Loopy belief propagation and Gibbs measures," Proc. Uncertainty in Artificial Intell., vol. 18, pp. 493-500, Aug. 2002.
27. S. L. Lauritzen and D. J. Spiegelhalter, "Local computations with probabilities on graphical structures and their application to expert systems (with discussion)," J. Roy. Statist. Soc. B, vol. 50, pp. 155-224, Jan. 1988.
28. F. V. Jensen, An Introduction to Bayesian Networks. London, U.K.: UCL, 1996.
29. T. P. Minka, "A family of algorithms for approximate Bayesian inference," Ph.D. dissertation, MIT, Cambridge, MA, Jan. 2001.
30. S. Amari, "Differential geometry of curved exponential families-Curvatures and information loss," Ann. Statist., vol. 10, no. 2, pp. 357-385, 1982.
31. N. N. Chentsov, "Statistical decision rules and optimal inference," in Translations of Mathematical Monographs. Providence, RI: Amer. Math. Soc., 1982, vol. 53.
32. Y. Censor and S. A. Zenios, "Parallel optimization: Theory, algorithms, and applications," in Numerical Mathematics and Scientific Computation. Oxford, U.K.: Oxford Univ. Press, 1988.
33. J. S. Yedidia, W. T. Freeman, and Y. Weiss, "Understanding belief propagation and its generalizations," Mitsubishi Elec. Res. Labs, Tech. Rep. TR2001-22, Jan. 2002.
34. N. Biggs, Algebraic Graph Theory. Cambridge, U.K.: Cambridge Univ. Press, 1993.
35. B. Bollobás, Modern Graph Theory. New York: Springer-Verlag, 1998.
36. G. R. Grimmett, "A theorem about random fields," Bull. London Math. Soc., vol. 5, pp. 81-84, 1973.
37. R. G. Cowell, A. P. Dawid, S. L. Lauritzen, and D. J. Spiegelhalter, "Probablistic networks and expert systems," Statistics for Engineering and Information Science, 1999.
38. J. Besag, "Spatial interaction and the statistical analysis of lattice systems," J. Roy. Statist. Soc. Ser. B, vol. 36, pp. 192-236, 1974.
39. M. Basseville, A. Benveniste, K. Chou, S. Golden, R. Nikoukhah, and A. Willsky, "Modeling and estimation of multiresolution stochastic processes," IEEE Trans. Inform. Theory, vol. 38, pp. 766-784, Mar. 1992.
40. O. E. Barndorff-Nielson, Information and Exponential Families. Chichester, U.K.: Wiley, 1978.
41. S. Amari, K. Kurata, and H. Nagaoka, "Information geometry of Boltzmann machines," IEEE Trans. Neural Networks, vol. 3, pp. 260-271, Mar. 1992.
42. S. S. Gupta, Ed., "Differential geometry in statistical inference," in Lecture Notes-Monograph Series. Hayward, CA: Inst. Math. Statist., 1987, vol. 10.
43. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
44. G. Rockafellar, Convex Analysis. Princeton, NJ: Princeton Univ. Press, 1970.
45. M. J. Wainwright, "Stochastic processes on graphs with cycles: Geometric and variational approaches," Ph.D. dissertation, MIT, Cambridge, MA, Jan. 2002.
46. I. Csiszár, "I-divergence geometry of probability distributions and minimization problems," Ann. Probab., vol. 3, no. 1, pp. 146-158, Feb. 1975.
47. D. P. Bertsekas, Nonlinear Programming. Belmont, MA: Athena Scientific, 1995.
48. J. N. Darroch and D. Ratcliff, "Generalized iterative scaling for log-linear models," Ann. Math. Statist., vol. 43, pp. 1470-1480, 1972.
49. M. J. Wainwright, E. B. Sudderth, and A. S. Willsky, "Tree-based modeling and estimation of Gaussian processes on graphs with cycles," in NIPS 13. Cambridge, MA: MIT Press, 2001, pp. 661-667.
50. E. Sudderth, "Embedded trees: Estimation of Gaussian processes on graphs with cycles," M.S. thesis, MIT, Cambridge, MA, Jan. 2002.
51. M. A. R. Leisink and H. J. Kappen, "A tighter bound for graphical models," in NIPS 13. Cambridge, MA: MIT Press, 2001, pp. 266-272.
52. T. S. Jaakkola and M. Jordan, "Recursive algorithms for approximating probabilities in graphical models," in NIPS 9. Cambridge, MA: MIT Press, 1996, pp. 487-493.
53. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky, "A new class of upper bounds on the log partition function," in Proc. Uncertainty in Artificial Intell., vol. 18, Aug. 2002, pp. 536-543.
54. D. Jungnickel, Graphs, Networks, and Algorithms. New York: Springer-Verlag, 1999.
55. C. Berge, Hypergraphs. Amsterdam. The Netherlands: North-Holland, 1989.
56. R. P. Stanley, Enumerative Combinatorics. Cambridge, U.K.: Cambridge Univ. Press, 1997, vol. 1.
57. S. L. Lauritzen, Graphical Models. Oxford, U.K.; Oxford Univ. Press, 1996.
58. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky. (2002, July) Tree consistency and bounds on the max-product algorithm and its generalizations. MIT LIDS Tech. Rep. P-2554. [Online]. Available: http://www.eecs.berkeley.edu/-martinw