Message-Passing Algorithms for Inference and Optimization

Journal of Statistical Physics

Volumn 145, Issue 4, 2011, Pages 860-890

  Yedidia, Jonathan S ^a  

^a DISNEY RESEARCH   (United States)

Author keywords

Belief propagation; Constraint satisfaction; Difference map; Divide and concur; Factor graphs; Inference; Message passing algorithms; Optimization

Indexed keywords

EID: 81755174182     PISSN: 00224715     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10955-011-0384-7     Document Type: Article

References (78)

1. Bahl, L., Cocke, J., Jelinek, F., Raviv, J.: Optimal decoding of linear codes for minimizing symbol error rate. IEEE Trans. Inf. Theory 20, 284-287 (1974).
2. Barber, D.: Bayesian Reasoning and Machine Learning. Cambridge University Press, Cambridge (2011).
3. Bauschke, H. H., Combettes, P. L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011).
4. Bauschke, H. H., Combettes P. L., Luke D. R.: Phase retrieval, error reduction algorithm, and Fienup variants: a view from convex optimization. J. Opt. Soc. Am. A 19, 1334-1345 (2002).
5. Baxter, R. J.: Exactly Solved Models in Statistical Mechanics. Academic Press, San Diego (1982).
6. Berrou, C., Glavieux, A., Thitimajshima, P.: Near Shannon limit error-correcting coding and decoding: turbo-codes. In: Proc. 1993 IEEE Int. Conf. on Comm., pp. 1064-1070 (1993).
7. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004).
8. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimisation via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 29, 1222-1239 (2001).
9. Braunstein, A., Mézard, M., Parisi, G.: Survey propagation: an algorithm for satisfiability. Random Struct. Algorithms 27, 201-226 (2005).
10. Bryant, R. E.: Graph-based algorithms for Boolean function manipulation. IEEE Trans. Comput. 35, 677-691 (1986).
11. Chertkov, M., Chernyak, M.: Loop series for discrete statistical models on graphs. J. Stat. Mech. (2006) doi: 10. 1088/1742-5468/2006/06/P06009.
12. Cormen, C. H., Leiserson, C. E., Rivest, R. L., Stein, C.: An Introduction to Algorithms, 3rd edn. MIT Press, Cambridge (2009). Chap. 15.
13. Douglas, J., Rachford, H. H.: On the numerical solution of heat conduction problems in two or three space variables. Trans. Am. Math. Soc. 82, 421-439 (1956).
14. Durbin, R., Eddy, S. R., Krogh, A., Mitchison, G.: Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, Cambridge (1998).
15. Elser, V., Rankenburg, I.: Deconstructing the energy landscape: constraint-based algorithms for folding heteropolymers. Phys. Rev. E 73, 026702 (2006).
16. Elser, V., Rankenburg, I., Thibault, P.: Searching with iterated maps. Proc. Natl. Acad. Sci. USA 104, 418-423 (2007).
17. Felzenszwalb, P. F., Huttnelocher, D. P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70, 41-54 (2006).
18. Fienup, J. R.: Phase retrieval algorithms: a comparison. Appl. Opt. 21, 2758-2769 (1982).
19. Forney, G. D.: Codes on graphs: normal realizations. IEEE Trans. Inf. Theory 47, 520-548 (2001).
20. Forney, G. D.: The Viterbi algorithm. Proc. IEEE 61, 268-278 (1973).
21. Freeman, W. T., Pasztor, E. C., Charmichael, O. T.: Learning low-level vision. Int. J. Comput. Vis. 40, 25-47 (2000).
22. Gallager, R. G.: Information Theory and Reliable Communication. Wiley, New York (1968).
23. Gallager, R. G.: Low-Density Parity-Check Codes. MIT Press, Cambridge (1963).
24. Gamarnik, D., Shah, D., Wei, Y.: Belief propagation for min-cost network flow: convergence and correctness. In: Proc. of the 2010 ACM-SIAM Symp. on Discrete Algorithms (2010).
25. Geman, S., Geman, D.: Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Trans. Pattern Anal. Mach. Intell. 6, 721-741 (1984).
26. Gravel, S.: Using symmetries to solve asymmetric problems. Ph. D. dissertation, Cornell University, Ithica, NY (2009).
27. Gravel, S., Elser, V.: Divide and concur: a general approach to constraint satisfaction. Phys. Rev. E 78, 036706 (2008).
28. Hershey, J. R., Rennie, S. J., Olsen, P. A., Kristjansson, T. T.: Super-human multi-talker speech recognition: a graphical modeling approach. Comput. Speech Lang. 24, 45-66 (2010).
29. Ihler, A., McAllester, D.: Particle belief propagation. In: Proc. of the 12th Int. Conf. on Artificial Intelligence and Statistics (2009).
30. Ising, E.: A contribution to the theory of ferromagnetism. Z. Phys. 31, 253 (1925).
31. Jelinek, F.: Statistical Methods for Speech Recognition. MIT Press, Cambridge (1997).
32. Johnson, J. K., Bickson, D., Dolev, D.: Fixing convergence of Gaussian belief propagation. In: Proc. Int. Symposium Inform. Theory (2009).
33. Jurafsky, D.: Martin, J. H: Speech and Language Processing. Prentice Hall, Upper Saddle River (2000).
34. Kallus, Y., Elser, V., Gravel, S.: Method for dense packing discovery. Phys. Rev. E 82, 056707 (2010).
35. Kalman, R. E.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82, 35-45 (1960).
36. Kikuchi, R.: A theory of cooperative phenomena. Phys. Rev. 81, 988-1003 (1951).
37. Knuth, D. E.: The Art of Computer Programming, vol. 4A. Combinatorial Algorithms. Addison-Wesley, New York (2011). Sect. 7. 1. 4.
38. Koller, D., Friedman, N.: Probabilistic Graphical Models. MIT Press, Cambridge (2009).
39. Krauth, W.: Statistical Mechanics: Algorithms and Computations. Oxford University Press, Oxford (2006).
40. Kschischang, F. R., Frey, B. J., Loeliger, H.-A.: Factor graphs and the sum-product algorithm. IEEE Trans. Inf. Theory 47, 498-519 (2001).
41. Lauritzen, S. L.: Spiegelhalter, D. J.: Local computations with probabilities on graphical structures and their application to expert systems. J. R. Stat. Soc., Ser. B 50, 157-194 (1988).
42. Lions, P.-L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964-979 (1979).
43. Loeliger, H.-A.: An introduction to factor graphs. IEEE Signal Proc. Mag., 28-41 (2004).
44. Loeliger, H.-A.: Least squares and Kalman filtering on Forney graphs. In: Blahut, R. E., Koetter, R. (eds.) Codes, Graphs, and Systems, pp. 113-135. Kluwer Academic, Norwell (2002).
45. Loeliger, H.-A., Dauwels, J., Hu, J., Korl, S., Ping, L., Kschischang, F.: The factor graph approach to model-based signal processing. Proc. IEEE 95, 1295-1322 (2007).
46. MacKay, D. J. C.: Information Theory, Inference, and Learning Algorithms. Cambridge University Press, Cambridge (2003).
47. Malioutov, D. M., Johnson, J. K., Willsky, A. S.: Walk-sums and belief propagation in Gaussian graphical models. J. Mach. Learn. Res. 7, 2031-2064 (2006).
48. Manning, C. D., Schutze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Cambridge (1999).
49. McEliece, R. J., MacKay, D. J. C., Cheng, J. F.: Turbo decoding as an instance of Pearl's "belief propagation" algorithm. IEEE J. Sel. Areas Commun. 16, 140-152 (1998).
50. Meltzer, T., Yanover, C., Weiss, Y.: Globally optimal solutions for energy minimization in stereo vision using reweighted belief propagation. In: Int. Conference on Computer Vision (2005).
51. Mézard, M., Montanari, A.: Information, Physics, and Computation. Oxford University Press, Oxford (2009).
52. Mézard, M., Parisi, G., Virasaro, M. A.: Spin Glass Theory and Beyond. World Scientific, Singapore (1987).
53. Mézard, M., Parisi, G., Zecchina, R.: Analytic and algorithmic solution of random satisfiability problems. Science 297, 812-815 (2002).
54. Minato, S.-E.: Zero-suppressed BDDs for set manipulation in combinatorial problems. In: Proc. 30th ACM/IEEE Design Automation Conference, pp. 272-277 (1993).
55. Minka, T. P.: Expectation propagation for approximate Bayesian inference. In: Proc. of the 17th Conf. on Uncertainty in Artificial Intelligence, pp. 362-369 (2001).
56. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco (1988).
57. Papadimitriou, C. H.: Computational Complexity. Addison-Wesley, Reading (1993).
58. Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 257-286 (1989).
59. Richardson, T.: Error floors of LDPC codes. In: Proc. 41st Allerton Conf. Commun. Contr., Comput., Monticello, IL (2003).
60. Richardson, T., Urbanke, R.: Modern Coding Theory. Cambridge University Press, Cambridge (2008).
61. Ruozzi, N., Tatikonda, S.: Convergent and correct message passing schemes for optimization problems over graphical models. Available at http://arxiv. org/abs/1002. 3239 (2010).
62. Rusmevichientong, P., van Roy, B.: An analysis of belief propagation on the turbo decoding graph with Gaussian densities. IEEE Trans. Inf. Theory 47, 745-765 (2001).
63. Russell, S., Norvig, P.: Artificial Intelligence, A Modern Approach, 3rd edn. Prentice Hall, Upper Saddle River (2009).
64. Ryan, W. E., Lin, S.: Channel Codes: Classical and Modern. Cambridge University Press, Cambridge (2009).
65. Sontag, D., Meltzer, T., Globerson, A., Jaakkola, T., Weiss, Y.: Tightening LP relaxations for MAP using message-passing. In: Uncertainty in Artificial Intelligence (2008).
66. Sudderth, E. B., Freeman, W. T.: Signal and Image processing with belief propagation. IEEE Signal Process. Mag. 25, 114-141 (2008).
67. Sudderth, E. B., Ihler, A., Isard, M., Freeman, W. T., Willsky, A. S.: Non-parametric belief propagation. Commun. ACM 53, 95-103 (2010).
68. Tanner, R. M.: A recursive approach to low complexity codes. IEEE Trans. Inf. Theory 27, 533-547 (1981).
69. Vardy, A.: Trellis structure of codes. In: Pless, V. S., Huffman, W. C. (eds.) Handbook of Coding Theory, vol. 2, pp. 1989-2118. Elsevier, Amsterdam (1998).
70. Viterbi, A. J.: Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory 13, 260-269 (1967).
71. Wainwright, M. J., Jaakkola, T., Willsky, A. S.: A new class of upper bounds on the log partition function. IEEE Trans. Inf. Theory 51, 2313-2335 (2005).
72. Wainwright, M. J., Jaakkola, T., Willsky, A. S.: Tree-based reparametrization framework for analysis of sum-product and related algorithms. IEEE Trans. Inf. Theory 45, 1120-1146 (2003).
73. Wainwright, M. J., Jaakkola, T., Willsky, A. S.: MAP estimation via agreement on (hyper)trees: message-passing and linear programming approaches. IEEE Trans. Inf. Theory 51, 3697-3717 (2005).
74. Wainwright, M. J., Jordan, M. I.: Graphical models, exponential families, and variational inference. Faund. Trends Mach. Learn. 1, 1-305 (2008).
75. Weiss, Y., Freeman, W. T.: On the optimality of the max-product belief propagation algorithm on arbitrary graphs. IEEE Trans. Inf. Theory 47, 736-744 (2001).
76. Yedidia, J. S., Freeman, W. T., Weiss, Y.: Understanding belief propagation and its generalizations. In: Lakemeyer, G., Nebel, B. (eds.) Exploring Artificial Intelligence in the New Millenium, pp. 239-270. Morgan Kaufmann, San Francisco (2003).
77. Yedidia, J. S., Freeman, W. T., Weiss, Y.: Constructing free energy approximations and generalized belief propagation algorithms. IEEE Trans. Inf. Theory 51, 2282-2312 (2005).
78. Yedidia, J. S., Wang, Y., Draper, S. C.: Divide and concur and difference-map BP decoders for LDPC codes. IEEE Trans. Inf. Theory 57, 786-802 (2011).