The latent maximum entropy principle

ACM Transactions on Knowledge Discovery from Data

Volumn 6, Issue 2, 2012, Pages

  Wang, Shaojun ^a   Schuurmans, Dale ^b   Zhao, Yunxin ^c  

^a Wright State University   (United States)

^b UNIVERSITY OF ALBERTA   (Canada)

^c University of Missouri   (United States)

Author keywords

Expectation maximization; Information geometry; Iterative scaling; Latent variable models; Maximum entropy

Indexed keywords

EXPECTATION MAXIMIZATION; INFORMATION GEOMETRY; ITERATIVE SCALING; LATENT VARIABLE MODELS; MAXIMUM ENTROPY;

ALGORITHMS; ENTROPY; EQUIVALENCE CLASSES; ESTIMATION; MAXIMUM LIKELIHOOD ESTIMATION; REGRESSION ANALYSIS;

MAXIMUM PRINCIPLE;

EID: 84866340966     PISSN: 15564681     EISSN: 1556472X     Source Type: Journal    
DOI: 10.1145/2297456.2297460     Document Type: Article

References (52)

1. ACKLEY, D., HINTON, G., AND SEJNOWSKI, T. 1985. A learning algorithm for Boltzmann machines. Cognitive Sci. 9, 147-169.
2. AMARI, S. 1995. Information geometry of the EM and em algorithms for neural networks. Neural Netw. 8, 9, 1379-1408. (Pubitemid 26072898)
3. AMARI, S. AND NAGAOKA, H. 2000. Methods of Information Geometry. American Mathematical Society.
4. BERGER, A., DELLA PIETRA, S., AND DELLA PIETRA, V. 1996. A maximum entropy approach to natural language processing. Comput. Linguist. 22, 1, 39-71.
5. BERTSEKAS, D. 1999. Nonlinear Programming 2nd Ed. Athena Scientific.
6. BLEI, D., NG, A., AND JORDAN, M. 2002. Latent Dirichlet allocation. In Advances Neural Information Processing Systems 14, 601-608.
7. BYRNE, W. 1992. Alternating minimization and Boltzmann machine learning. IEEE Trans. Neural Netw. 3, 4, 612-620.
8. CENSOR, Y. AND ZENIOS, S. 1997. Parallel Optimization: Theory, Algorithms, and Applications. Oxford University Press.
9. COLLINS, M., SCHAPIRE, R., AND SINGER, Y. 2002. Logistic regression, AdaBoost and Bregman distances. Mach. Learn. 48, 3, 253-285. (Pubitemid 34247580)
10. COVER, T. AND THOMAS, J. 1991. Elements of Information Theory. Wiley.
11. CSISZAR, I. 1975. I-Divergence geometry of probability distributions and minimization problems. Ann. Probab. 3, 146-158.
12. CSISZAR, I. AND TUSNADY, G. 1984. Information geometry and alternating minimization procedures. Statistics and Decisions. Supplement Issue 1, 205-237.
13. DARROCH, J. AND RATCHLIFF, D. 1972. Generalized iterative scaling for log-linear models. Ann. Math. Stat. 43, 5, 1470-1480.
14. DELLA PIETRA, S., DELLA PIETRA, V., AND LAFFERTY, J. 1997. Inducing features of random fields. IEEE Trans. Pattern Anal. Mach. Intell. 19, 4, 380-393. (Pubitemid 127762893)
15. DEMPSTER, A., LAIRD, N., AND RUBIN, D. 1977. Maximum likelihood estimation from incomplete data via the EM algorithm. J. Royal Stat. Soc. Series B, 39, 1-38.
16. EISNER, J. 2002. Parameter estimation for probabilistic finite-state transducers. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 1-8.
17. FANG, S., RAJASEKERA, J., AND TSAO, H. 1997. Entropy Optimization and Mathematical Programming, Kluwer.
18. FISHER, R. 1936. The use of multiple measurements in taxonomic problems. Ann. Eugenics 7, II, 179-188.
19. GANCHEV, K., GRAFIA, J., GILLENWATER, J., AND TASKAR, B. 2010. Posterior regularization for structured latent variable models. J. Mach. Learn. Res. 11, 2001-2049.
20. GAUVAIN, J. AND LEE, C.-H. 1994. Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains. IEEE Trans. Speech Audio Process. 2, 2, 291-298.
21. GOLAN, A., MILLER, D., AND JUDGE, G. 1996. Maximum Entropy Econometrics: Robust Estimation with Limited Data. Wiley.
22. GOODMAN, J. 2002. Sequential conditional generalized iterative scaling. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 9-16.
23. GRACA, J., GANCHEV, K., AND TASKAR, B. 2007. Expectation maximization and posterior constraints. In Advances in Neural Information Processing Systems (NIPS).
24. HAGENAARS, J. 1993. Loglinear Models with Latent Variables. Sage Publications.
25. HOFMANN, T. 2001. Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42, 1, 177-196. (Pubitemid 32872403)
26. HUANG, F., HSIEH, C., CHANG, K., AND LIN, C. 2010. Iterative scaling and coordinate descent methods for maximum entropy models. J. Mach. Learn. Res. 11, 815-848.
27. JAAKKOLA, T.,MEILA, M., AND JEBARA, T. 1999. Maximum entropy discrimination. In Advances in Neural Information Processing Systems. 470-476.
28. JAYNES, E. 1983. Papers on Probability, Statistics, and Statistical Physics. R. Rosenkrantz Ed., D. Reidel Publishing.
29. JEBARA, T. 2000. Discriminative, generative and imitative learning. Ph.D. dissertation, MIT.
30. JELINEK, F. 1998. Statistical Methods for Speech Recognition. MIT Press.
31. LAFFERTY, J., MCCALLUM, A. AND PEREIRA, F. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data, In Proceedings of the International Conference on Machine Learning (ICML). 282-289.
32. LAU, R., ROSENFELD, R., AND ROUKOS, S. 1993. Trigger-based language models: A maximum entropy approach. In Proceedings of the 18th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). II, 45-48.
33. LAURITZEN, S. 1995. The EM-algorithm for graphical association models with missing data. Comput. Stat. Data Anal.19, 2, 191-201.
34. LAURITZEN, S. 1996. Graphical Models. Oxford University Press.
35. LITTLE, R. AND RUBIN, D. 2002. Statistical Analysis with Missing Data 2nd Ed. Wiley-Interscience.
36. MACKAY, D. AND PETO, L. 1995. A hierarchical Dirichlet language model. Natural Lang. Eng. 1, 3, 289-307.
37. MALOUF, R. 2002. A comparison of algorithms for maximum entropy parameter estimation. In Proceedings of the 6th Conference on Natural Language Learning. 49-55.
38. MENG, X. AND RUBIN, D. 1993. Maximum likelihood estimation via the ECL algorithm: A general framework. Biometrika 80, 2, 267-278.
39. MINKA, T. 2003. A comparison of numerical 1169 optimizers for logistic regression. Tech. rep. http://research.microsoft.com/en-us/um/people/minka/ papers/logreg/minka-logreg.pdf.
40. RIEZLER, S. 1999. Probabilistic constraint logic programming. Ph.D. dissertation, University of Stuttgart.
41. RIEZLER, S., PRESCHER, D., KUHN, J., AND JOHNSON, M. 2000. Lexicalized stochastic modeling of constraint-based grammars using log-linear measures and EM training. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics.
42. ROSENFELD, R. 1996. A maximum entropy approach to adaptive statistical language modeling. Comput. Speech Lang. 10, 2, 187-228.
43. WAINWRIGHT, M., JAAKKOLA, T., AND WILLSKY, A. 2003. Tree-based reparameterization framework for analysis of belief propagation and related algorithms. IEEE Trans. Inf. Theory 49, 5, 1120-1146.
44. WANG, S., ROSENFELD, R., AND ZHAO, Y. 2001. Latent maximum entropy principle for statistical language modeling. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding.
45. WANG, S., SCHUURMANS, D., PENG, F., AND ZHAO, Y. 2003a. Learning mixture models with the latent maximum entropy principle. In Proceedings of the 20th International Conference on Machine Learning (ICML). 784-791.
46. WANG, S., SCHUURMANS, D., PENG, F., AND ZHAO, Y. 2003b. Boltzmann machine learning with the latent maximum entropy principle. In Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence (UAI). 567-574.
47. WANG, S., SCHUURMANS, D., PENG, F., AND ZHAO, Y. 2005. Combining statistical language models via the latent maximum entropy principle. Mach. Learn. J. 60 (Special Issue on Learning in Speech and Language Technologies), 229-250. (Pubitemid 41218069)
48. WANG, S., GREINER, R., AND WANG, S. 2009. Consistency and generalization bounds for maximum entropy density. Tech. rep., Department of Computer Science, Wright State University.
49. WU, C. 1983. On the convergence properties of the EM algorithm. Ann. Stat. 11, 95-103.
50. YEDIDIA, J., FREEMAN, W., AND WEISS, Y. 2005. Constructing free energy approximations and generalized belief propagation algorithms. IEEE Trans. Inf. Theory 51, 7, 2282-2312. (Pubitemid 41136394)
51. ZHU, J., XING, E., AND ZHANG, B. 2008. Partially observed maximum entropy discrimination Markov networks. In Advances in Neural Information Processing Systems (NIPS). 1977-1984.
52. ZHU, J. AND XING, E. 2009. Maximum entropy discrimination Markov networks. J. Mach. Learn. Res. 10, 2531-2569.