MAP estimation via agreement on trees: Message-passing and linear programming

IEEE Transactions on Information Theory

Volumn 51, Issue 11, 2005, Pages 3697-3717

  Wainwright, Martin J ^a   Jaakkola, Tommi S ^b   Willsky, Alan S ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b USA   (United States)

Author keywords

Approximate inference; Integer programming; Iterative decoding; Linear programming (LP) relaxation; Marginal polytope; Markov random fields; Max product algorithm; Maximum a posteriori probability (MAP) estimation; Message passing algorithms; Min sum algorithm

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; DECODING; DYNAMIC PROGRAMMING; INTEGER PROGRAMMING; ITERATIVE METHODS; LINEAR PROGRAMMING; MARKOV PROCESSES; PROBABILITY DISTRIBUTIONS; TREES (MATHEMATICS);

APPROXIMATION INFERENCE; LINEAR PROGRAMMING (LP) RELAXATION; MARGINAL POLYTOPE; MARKOV RANDOM FIELD; MAX-PRODUCT ALGORITHM; MAXIMUM A POSTERIORI PROBABILITY (MAP); MESSAGE-PASSING ALGORITHMS; MIN-SUM ALGORITHM;

INFORMATION THEORY;

EID: 27744456278     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.856938     Document Type: Article

References (47)

1. S. M. Aji, G. Horn, R. J. McEliece, and M. Xu, "Iterative min-sum decoding of tail-biting codes," in Proc. IEEE Information Theory Workshop, Killarney, Ireland, Jun. 1998, pp. 68-69.
2. S. M. Aji and R. J. McEliece, "The generalized distributive law," IEEE Trans. Inf. Theory, vol. 46, no. 2, pp. 325-343, Mar. 2000.
3. S. Amari, "Information geometry on a hierarchy of probability distributions," IEEE Trans. Inf. Theory, vol. 47, no. Jul., pp. 1701-1711, 2001.
4. O. E. Barndorff-Nielson, Information and Exponential Families. Chichester, U.K.: Wiley, 1978.
5. R. J. Baxter, Exactly Solved Models in Statistical Mechanics. New York: Academic, 1982.
6. D. P. Bertsekas, Nonlinear Programming. Belmont, MA: Athena Scientific, 1995.
7. D. Bertsimas and J. Tsitsikilis, Introduction to Linear Optimization. Belmont, MA: Athena Scientific, 1997.
8. J. Besag, "On the statistical analysis of dirty pictures," J. Roy. Statist. Soc., ser. B, vol. 48, no. 3, pp. 259-279, 1986.
9. N. Biggs, Algebraic Graph Theory. Cambridge, MA: Cambridge Univ. Press, 1993.
10. B. Bollobás, Modern Graph Theory. New York: Springer-Verlag, 1998.
11. E. Boros, Y. Crama, and P. L. Hammer, "Upper bounds for quadratic 0-1 maximization," Oper. Res. Lett., vol. 9, pp. 73-79, 1990.
12. P. Brémaud, Markov Chains, Gibbs Fields, Monte Carlo Simulation, and Queues. Berlin, Germany: Springer-Verlag, 1991.
13. L. Chen, M. J. Wainwright, M. Cetin, and A. Willsky, "Multitarget-multisensor data association using the tree-reweighted max-product algorithm," in Proc. SPIE Aerosense Conf., Orlando, FL, Apr. 2003.
14. R. G. Cowell, A. P. Dawid, S. L. Lauritzen, and D. J. Spiegelhalter, Probabilistic Networks and Expert Systems, ser. Statistics for Engineering and Information Science. Berlin, Germany: Springer-Verlag, 1999.
15. A. P. Dawid, "Applications of a general propagation algorithm for probabilistic expert systems," Statist. and Comput., vol. 2, pp. 25-36, 1992.
16. M. Deza and M. Laurent, Geometry of Cuts and Metric Embeddings. New York: Springer-Verlag, 1997.
17. J. Edmonds, "Matroids and the greedy algorithm," Math. Programming, vol. 1, pp. 127-136, 1971.
18. J. Feldman, D. R. Karger, and M. J. Wainwright, "Using linear programming to decode LDPC codes," in Proc. Conf. Information Science and Systems, Baltimore, MD, Mar. 2003.
19. J. Feldman, M. J. Wainwright, and D. R. Karger, "Linear programming-based decoding of turbo-like codes and its relation to iterative approaches," in Proc. Allerton Conf. Communication, Control and Computing, Monticello, IL, Oct. 2002.
20. J. Feldman, M. J. Wainwright, and D. R. Karger, "Using linear programming to decode binary linear codes," IEEE Trans. Inf. Theory, vol. 51, no. 3, pp. 954-972, Mar. 2005.
21. G. D. Forney, Jr., F. R. Kschischang, B. Marcus, and S. Tuncel, "Iterative decoding of tail-biting trellises and connections with symbolic dynamics," in Codes, Systems and Graphical Models. New York: Springer-Verlag, 2001, pp. 239-264.
22. G. D. Forney Jr., R. Koetter, F. R. Kschischang, and A. Reznick, "On the effective weights of pseudocodewords for codes defined on graphs with cycles," in Codes, Systems and Graphical Models. New York: Springer-Verlag, 2001, pp. 101-112.
23. W. T. Freeman and Y. Weiss, "On the optimality of solutions of the max-product belief propagation algorithm in arbitrary graphs," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 736-744, Feb. 2001.
24. B. J. Frey and R. Koetter, "Exact inference using the attenuated max-product algorithm," in Advanced Mean Field Methods: Theory and Practice. Cambridge, MA: MIT Press, 2000.
25. B. J. Frey, R. Koetter, and A. Vardy, "Signal-space characterization of iterative decoding," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 766-781, Feb. 2001.
26. D. M. Greig, B. T. Porteous, and A. H. Seheuly, "Exact maximum a posteriori estimation for binary images," J. Roy. Statist. Soc., ser. B, vol. 51, pp. 271-279, 1989.
27. P. L. Hammer, P. Hansen, and B. Simeone, "Roof duality, complementation, and persistency in quadratic 0-1 optimization," Math. Programming, vol. 28, pp. 121-155, 1984.
28. J. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms. New York: Springer-Verlag, 1993, vol. 1.
29. G. Horn, "Iterative decoding and pseudocodewords," Ph.D. dissertation, Calif. Inst. Technol., Pasadena, CA, 1999.
30. R. Koetter and P. O. Vontobel, "Graph-covers and iterative decoding of finite length codes," in Proc. 3rd Int. Symp. Turbo Codes, Brest, France, Sep. 2003.
31. V. Kolmogorov, "Convergent tree-reweighted message-passing for energy minimization," in Proc. Int. Workshop on Artificial Intelligence and Statistics, Barbados, Jan. 2005.
32. V. Kolmogorov, "Convergent Tree-Reweighted Message-Passing for Energy Minimization," Microsoft Research Tech. Rep., Jan. 2005. Submitted to IEEE Trans. Pattern Anal. Machine Intel..
33. V. Kolmogorov and M. J. Wainwright, "On optimality properties of tree-reweighted message-passing," in Uncertainty in Artificial Intelligence. Edinburgh, Scotland, U.K.: Morgan-Kaufmann, 2005.
34. S. L. Lauritzen, Graphical Models. Oxford, U.K.: Oxford Univ. Press, 1996.
35. R. J. McEliece, D. J. C. McKay, and J. F. Cheng, "Turbo decoding as an instance of Pearl's belief propagation algorithm," IEEE J. Sel. Areas Commun., vol. 16, no. 2, pp. 140-152, Feb. 1998.
36. T. Minka, "Power EP," Dep. Statistics, Carnegie Mellon University, Pittsburgh, PA, Tech. Rep., 2004.
37. G. Rockafellar, Convex Analysis. Princeton, NJ: Princeton Univ. Press, 1970.
38. A. Schrijver, Combinatorial Optimization: Polyhedra and Efficiency. New York: Springer-Verlag, 2003.
39. S. Verdú, Multiuser Detection. Cambridge, U.K.: Cambridge Univ. Press, 1998.
40. A. Viterbi, "Error bounds for convolutional codes and an asymptotically optimal decoding algorithm," IEEE Trans. Inf. Theory, vol. IT-13, no. 2, pp. 260-269, Apr. 1967.
41. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky, "A new class of upper bounds on the log partition function," Uncertainty in Artificial Intelligence, vol. 18, pp. 536-543, Aug. 2002.
42. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky, "Tree-based reparameterization framework for analysis of sumproduct and related algorithms," IEEE Trans. Inf. Theory, vol. 49, no. 5, pp. 1120-1146, May 2003.
43. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky, "Tree consistency and bounds on the max-product algorithm and its generalizations," Statist. and Comput., vol. 14, pp. 143-166, Apr. 2004.
44. M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky, "A new class of upper bounds on the log partition function," IEEE Trans. Inf. Theory, vol. 51, no. 7, pp. 2313-2335, Jul. 2005.
45. N. Wiberg, "Codes and decoding on general graphs," Ph.D. dissertation, Univ. Linköping, Linköping, Sweden, 1996.
46. W. Wiegerinck and T. Heskes, "Fractional belief propagation," Neural Inf. Process. Syst., vol. 12, pp. 438-445, 2002.
47. J. S. Yedidia, W. T. Freeman, and Y. Weiss, "Understanding belief propagation and its generalizations," Mitsubishi Electric Research Labs, Tech. Rep. TR2001-22, Jan. 2002.