Introduction to variational methods for graphical models

Machine Learning

Volumn 37, Issue 2, 1999, Pages 183-233

  Jordan, Michael I ^a   Ghahramani, Zoubin ^d   Jaakkola, Tommi S ^b   Saul, Lawrence K ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c AT AND T LABS RESEARCH   (United States)

^d NONE

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; DATABASE SYSTEMS; GRAPHIC METHODS; LEARNING SYSTEMS; MARKOV PROCESSES; MATHEMATICAL MODELS; MATHEMATICAL TRANSFORMATIONS; NEURAL NETWORKS; PROBABILITY;

BAYESIAN NETWORKS; BOLTZMANN MACHINE; GRAPHICAL MODELS; HIDDEN MARKOV MODELS; INFERENCE ALGORITHMS; SIGMOID NETWORK;

VARIATIONAL TECHNIQUES;

EID: 0033225865     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1007665907178     Document Type: Article

References (61)

1. Bathe, K. J. (1996). Finite element procedures. Englewood Cliffs, NJ: Prentice-Hall.
2. Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Mathematical Statistics, 41, 164-171.
3. Bishop, C. M., Lawrence, N., Jaakkola, T. S., & Jordan, M. I. (1998). Approximating posterior distributions in belief networks using mixtures. In M. Jordan, M. Kearns, & S. Solla (Eds.), Advances in neural information processing systems 10, Cambridge MA: MIT Press.
4. Cover, T., & Thomas, J. (1991). Elements of information theory. New York: John Wiley.
5. Dagum, P., & Luby, M. (1993). Approximating probabilistic inference in Bayesian belief networks is NP-hard. Artificial Intelligence, 60, 141-153.
6. Dayan, P., Hinton, G. E., Neal, R., & Zemel, R. S. (1995). The Helmholtz Machine. Neural Computation, 7, 889-904.
7. Dean, T., & Kanazawa, K. (1989). A model for reasoning about causality and persistence. Computational Intelligence, 5, 142-150.
8. Dechter, R. (1999). Bucket elimination: A unifying framework for probabilistic inference. In M. I. Jordan (Ed.), Learning in graphical models. Cambridge, MA: MIT Press.
9. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum-likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, B39, 1-38.
10. Draper, D. L., & Hanks, S. (1994). Localized partial evaluation of belief networks. Uncertainty and Artificial Intelligence: Proceedings of the Tenth Conference. San Mateo, CA: Morgan Kaufmann.
11. Frey, B., Hinton, G. E., & Dayan, P. (1996). Does the wake-sleep algorithm learn good density estimators? In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems 8. Cambridge, MA: MIT Press.
12. Fung, R. & Favero, B. D. (1994). Backward simulation in Bayesian networks. Uncertainty and Artificial Intelligence: Proceedings of the Tenth Conference. San Mateo, CA: Morgan Kaufmann.
13. Galland, C. (1993). The limitations of deterministic Boltzmann machine learning. Network, 4, 355-379.
14. Ghahramani, Z., & Hinton, G. E. (1996). Switching state-space models. (Technical Report CRG-TR-96-3). Toronto: Department of Computer Science, University of Toronto.
15. Ghahramani, Z., & Jordan, M. I. (1997). Factorial Hidden Markov models. Machine Learning, 29, 245-273.
16. Gilks, W., Thomas, A., & Spiegelhalter, D. (1994). A language and a program for complex Bayesian modelling. The Statistician, 43, 169-178.
17. Heckerman, D. (1999). A tutorial on learning with Bayesian networks. In M. I. Jordan (Ed.), Learning in graphical models. Cambridge, MA: MIT Press.
18. Henrion, M. (1991). Search-based methods to bound diagnostic probabilities in very large belief nets. Uncertainty and Artificial Intelligence: Proceedings of the Seventh Conference. San Mateo, CA: Morgan Kaufmann.
19. Hinton, G. E., & Sejnowski, T. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing (Vol. 1). Cambridge, MA: MIT Press.
20. Hinton, G. E., & van Camp, D. (1993). Keeping neural networks simple by minimizing the description length of the weights. Proceedings of the 6th Annual Workshop on Computational Learning Theory. New York, NY: ACM Press.
21. Hinton, G. E., Dayan, P., Frey, B., & Neal, R. M. (1995). The wake-sleep algorithm for unsupervised neural networks. Science, 268, 1158-1161.
22. Hinton, G. E., Sallans, B., & Ghahramani, Z. (1999). A hierarchical community of experts. In M. I. Jordan (Ed.), Learning in graphical models. Cambridge, MA: MIT Press.
23. Horvitz, E. J., Suermondt, H. J., & Cooper, G. F. (1989). Bounded conditioning: Flexible inference for decisions under scarce resources. Conference on Uncertainty in Artificial Intelligence: Proceedings of the Fifth Conference. Mountain View, CA: Association for UAI.
24. Jaakkola, T. S., & Jordan, M. I. (1996). Computing upper and lower bounds on likelihoods in intractable networks. Uncertainty and Artificial Intelligence: Proceedings of the Twelth Conference. San Mateo, CA: Morgan Kaufmann.
25. Jaakkola, T. S. (1997). Variational methods for inference and estimation in graphical models. Unpublished doctoral dissertation, Massachusetts Institute of Technology, Cambridge, MA.
26. Jaakkola, T. S., & Jordan, M. I. (1997a). Recursive algorithms for approximating probabilities in graphical models. In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems 9. Cambridge, MA: MIT Press.
27. Jaakkola, T. S., & Jordan, M. I. (1997b). Bayesian logistic regression: A variational approach. In D. Madigan & P. Smyth (Eds.), Proceedings of the 1997 Conference on Artificial Intelligence and Statistics. Ft. Lauderdale, FL.
28. Jaakkola, T. S., & Jordan, M. I. (1999a). Improving the mean field approximation via the use of mixture distributions. In M. I. Jordan (Ed.), Learning in graphical models. Cambridge, MA: MIT Press.
29. Jaakkola, T. S., & Jordan, M. I. (1999b). Variational methods and the QMR-DT database. Journal of Artificial Intelligence Research, 10, 291-322.
30. Jensen, C. S., Kong, A., & Kjærulff, U. (1995). Blocking-Gibbs sampling in very large probabilistic expert systems. International Journal of Human-Computer Studies, 42, 647-666.
31. Jensen, F. V., & Jensen, F. (1994). Optimal junction trees. Uncertainty and Artificial Intelligence: Proceedings of the Tenth Conference. San Mateo, CA: Morgan Kaufmann.
32. Jensen, F. V. (1996). An introduction to Bayesian networks. London: UCL Press.
33. Jordan, M. I. (1994). A statistical approach to decision tree modeling. In M. Warmuth (Ed.), Proceedings of the Seventh Annual ACM Conference on Computational Learning Theory. New York: ACM Press.
34. Jordan, M. I., Ghahramani, Z., & Saul, L. K. (1997). Hidden Markov decision trees. In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems 9. Cambridge, MA: MIT Press.
35. Kanazawa, K., Koller, D., & Russell, S. (1995). Stochastic simulation algorithms for dynamic probabilistic networks. Uncertainty and Artificial Intelligence: Proceedings of the Eleventh Conference. San Mateo, CA: Morgan Kaufmann.
36. Kjærulff, U. (1990). Triangulation of graphs - Algorithms giving small total state space. (Research Report R-90-09). Department of Mathematics and Computer Science, Aalborg University, Denmark.
37. Kjærulff, U. (1994). Reduction of computational complexity in Bayesian networks through removal of weak dependences. Uncertainty and Artificial Intelligence: Proceedings of the Tenth Conference. San Mateo, CA: Morgan Kaufmann.
38. MacKay, D. J. C. (1997). Ensemble learning for hidden Markov models. Unpublished manuscript. Cambridge: Department of Physics, University of Cambridge.
39. McEliece, R. J., MacKay, D. J. C., & Cheng, J.-F. (1998). Turbo decoding as an instance of Pearl's "belief propagation algorithm." IEEE Journal on Selected Areas in Communication, 16, 140-152.
40. Merz, C. J., & Murphy, P. M. (1996). UCI repository of machine learning databases. Irvine, CA: Department of Information and Computer Science, University of California.
41. Neal, R. (1992). Connectionist learning of belief networks. Artificial Intelligence, 56, 71-113.
42. Neal, R. (1993). Probabilistic inference using Markov chain Monte Carlo methods. (Technical Report CRG-TR-93-1). Toronto: Department of Computer Science, University of Toronto.
43. Neal, R., & Hinton, G. E. (1999). A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. I. Jordan (Ed.), Learning in graphical models. Cambridge, MA: MIT Press.
44. Parisi, G. (1988). Statistical field theory. Redwood City, CA: Addison-Wesley.
45. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo, CA: Morgan Kaufmannn.
46. Peterson, C., & Anderson, J. R. (1987). A mean field theory learning algorithm for neural networks. Complex Systems, 1, 995-1019.
47. Rockafellar, R. (1972). Convex analysis. Princeton University Press.
48. Rustagi, J. (1976). Variational methods in statistics. New York: Academic Press.
49. Sakurai, J. (1985). Modern quantum mechanics. Redwood City, CA: Addison-Wesley.
50. Saul, L. K., & Jordan, M. I. (1994). Learning in Boltzmann trees. Neural Computation, 6, 1173-1183.
51. Saul, L. K., Jaakkola, T. S., & Jordan, M. I. (1996). Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4, 61-76.
52. Saul, L. K., & Jordan, M. I. (1996). Exploiting tractable substructures in intractable networks. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems 8. Cambridge, MA: MIT Press.
53. Saul, L. K., & Jordan, M. I. (1999). A mean field learning algorithm for unsupervised neural networks. In M. I. Jordan (Ed.), Learning in graphical models. Cambridge, MA: MIT Press.
54. Seung, S. (1995). Annealed theories of learning. In J.-H. Oh, C. Kwon, S. Cho (Eds.), Neural networks: The statistical mechanics perspectives. Singapore: World Scientific.
55. Shachter, R. D., Andersen, S. K., & Szolovits, P. (1994). Global conditioning for probabilistic inference in belief networks. Uncertainty and Artificial Intelligence: Proceedings of the Tenth Conference. San Mateo, CA: Morgan Kaufmann.
56. Shenoy, P. P. (1992). Valuation-based systems for Bayesian decision analysis. Operations Research, 40, 463-484.
57. Shwe, M. A., & Cooper, G. F. (1991). An empirical analysis of likelihood - Weighting simulation on a large, multiply connected medical belief network. Computers and Biomedical Research, 24, 453-475.
58. Shwe, M. A., Middleton, B., Heckerman, D. E., Henrion, M., Horvitz, E. J., Lehmann, H. P., & Cooper, G. F. (1991). Probabilistic diagnosis using a reformulation of the INTERNIST-1/QMR knowledge base. Meth. Inform. Med., 30, 241-255.
59. Smyth, P., Heckerman, D., & Jordan, M. I. (1997). Probabilistic independence networks for hidden Markov probability models. Neural Computation, 9, 227-270.
60. Waterhouse, S., MacKay, D. J. C., & Robinson, T. (1996). Bayesian methods for mixtures of experts. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems 8. Cambridge, MA: MIT Press.
61. Williams, C. K. I., & Hinton, G. E. (1991). Mean field networks that learn to discriminate temporally distorted strings. In D. S. Touretzky, J. Elman, T. Sejnowski, & G. E. Hinton (Eds.), Proceedings of the 1990 Connectionist Models Summer School. San Mateo, CA: Morgan Kaufmann.