Approximate inference in Boltzmann machines

Artificial Intelligence

Volumn 143, Issue 1, 2003, Pages 19-50

  Welling, Max ^a   Teh, Yee Whye ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Advanced mean field methods; Boltzmann machines; Inference; Loopy belief propagation

Indexed keywords

ALGORITHMS; APPROXIMATION THEORY; GIBBS FREE ENERGY; LEARNING SYSTEMS; OPTIMIZATION; THEOREM PROVING;

BOLTZMANN MACHINES;

ARTIFICIAL INTELLIGENCE;

EID: 0037216891     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0004-3702(02)00361-2     Document Type: Article

References (33)

1. Amari S. Information geometry of the EM and em algorithms for neural networks. Neural Networks. 8:1995;1379-1408.
2. Frey B.J., MacKay D.J.C. A revolution: Belief propagation in graphs with cycles. Advances in Neural Information Processing Systems. 10:1997;MIT Press, Cambridge, MA.
3. Georges A., Yedidia J.S. How to expand around mean-field theory using high-temperature expansions. J. Phys A: Math. Gen. 24:1991;2173-2192.
4. Hinton G.E., Sejnowski T.J. Optimal perceptual inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC. 1983;448-453.
5. Hinton G.E., Sejnowski T.J. Learning and relearning in Boltzmann machines. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. 1986;MIT Press, Cambridge, MA.
6. Hopfield J.J. Neural networks and physical systems with emergent collective computational abilities. Proc. Nat. Acad. Sci. 79:1992;2554-2558.
7. Jordan M.I., Ghahramani Z., Jaakkola T., Saul L.K. An introduction to variational methods for graphical models. Machine Learning. 37:(2):1999;183-233.
8. Kabashima Y., Saad D. Belief propagation vs. TAP for decoding corrupted messages. Europhys. Lett. 44:1998;668.
9. Kappen H.J., Rodriguez F.B. Efficient learning in Boltzmann machines using linear response theory. Neural Comput. 10:1998;1137-1156.
10. Leisink M.A.R., Kappen H.J. Validity of TAP equations in neural networks. Proceedings of the International Conference on Artificial Neural Networks. 1999;425-430.
11. D.J.C. MacKay, J.S. Yedidia, W.T. Freeman, Y. Weiss, A conversation about the Bethe free energy and sum-product, http://www.inference.phy.cam.ac.uk/mackay/.
12. McEliece R., MacKay D., Cheng J. Turbo decoding as an instance of Pearl's belief propagation algorithm. IEEE J. Selected Areas in Communication. 16:1998;140-152.
13. Mezard M., Parisi G., Virasoro M.A. Spin Glass Theory and Beyond. 1987;World Scientific, Singapore.
14. Morita T. Variational principle for the distribution function of the effective field for the random Ising model in the Bethe approximation. Physica A. 98:1979;566-572.
15. Murphy K., Weiss Y., Jordan M. Loopy belief propagation for approximate inference: An empirical study. Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, Stockholm, Sweden. 1999;467-475.
16. Nishimori H. Statistical Physics of Spin Glasses and Information Processing, Oxford. 2001.
17. Onsager L. Electric moments of molecules in liquids. J. Amer. Chem. Soc. 58:1936;1486-1493.
18. Opper M., Saad D. Advanced Mean Field Methods - Theory and Practice. 2001;MIT Press, Cambridge, MA.
19. Opper M., Winther O. A mean field approach to Bayes learning in large feed-forward neural networks. Advances in Neural Information Processing Systems. 9:1997;225-331 MIT Press, Cambridge, MA.
20. Opper M., Winther O. From naive mean field theory to the TAP equations. Advanced Mean Field Methods - Theory and Practice. 2001;MIT Press, Cambridge, MA.
21. Parisi G. Statistical Field Theory. 1988;Perseus Books.
22. Pearl J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. 1988;Morgan Kaufmann, San Mateo, CA.
23. Peterson C., Anderson J. A mean field theory learning algorithm for neural networks. Complex Systems. 1:1987;995-1019.
24. Plefka T. Convergence condition of the TAP equations for infinite ranged Ising spin glass model. J. Phys. A. 15:1982;1971.
25. Teh Y.W., Welling M. The unified propagation and scaling algorithm. Advances in Neural Information Processing Systems. 2002;MIT Press, Cambridge, MA.
26. Weiss Y. Correctness of local probability propagation in graphical models with loops. Neural Comput. 12:2000;1-41.
27. Weiss Y. Comparing the mean field method and belief propagation for approximate inference in MRFs. Advanced Mean Field Methods - Theory and Practice. 2001;MIT Press, Cambridge, MA.
28. Weiss Y., Freeman W. Correctness of belief propagation in Gaussian graphical models of arbitrary topology. Advances in Neural Information Processing Systems. 12:1999;MIT Press, Cambridge, MA.
29. Welling M., Teh Y.W. Belief optimization for binary networks: A stable alternative to loopy belief propagation. Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence, Seattle, WA. 2001;554-561.
30. Wiegerinck W. Variational approximations between mean field theory and the junction tree algorithm. Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, Stanford, CA. 2000;626-636.
31. Yedidia J.S. An idiosyncratic journey beyond mean field theory. Advanced Mean Field Methods - Theory and Practice. 2001;MIT Press, Cambridge, MA.
32. Yedidia J.S., Freeman W., Weiss Y. Generalized belief propagation. Advances in Neural Information Processing Systems. 13:2000;MIT Press, Cambridge, MA.
33. Yuille A.L. CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation. Neural Comput. 14:(7):2002;1691-1722.