A large-deviation analysis of the maximum-likelihood learning of markov tree structures

IEEE Transactions on Information Theory

Volumn 57, Issue 3, 2011, Pages 1714-1735

  Tan, Vincent Y F ^a   Anandkumar, Animashree ^b   Tong, Lang ^c   Willsky, Alan S ^d  

^a Wisconsin Electric Machines and Power Electronics Consortium   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c School of Operations Research and Industrial Engineering   (United States)

^d MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Error exponent; Euclidean information theory; large deviations principle; Markov structure; maximum likelihood (ML) distribution estimation; tree structured distributions

Indexed keywords

ERROR EXPONENT; EUCLIDEAN; LARGE-DEVIATIONS PRINCIPLE; MARKOV STRUCTURE; MAXIMUM-LIKELIHOOD (ML) DISTRIBUTION ESTIMATION; TREE-STRUCTURED DISTRIBUTIONS;

ESTIMATION; INFORMATION THEORY; INFORMATION USE; PROBABILITY; SIGNAL TO NOISE RATIO; TREES (MATHEMATICS);

MAXIMUM LIKELIHOOD ESTIMATION;

EID: 79951910784     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2011.2104513     Document Type: Article

References (51)

1. V. Y. F. Tan, A. Anandkumar, L. Tong, and A. S.Willsky, "A large-deviation analysis for the maximum likelihood learning of tree structures," in Proc. IEEE Int. Symp. Information Theory, Seoul, Korea, Jul. 2009, pp. 1140-1144.
2. S. Lauritzen, Graphical Models. Oxford, U,K,: Oxford Univ. Press, 1996.
3. C. K. Chow and C. N. Liu, "Approximating discrete probability distributions with dependence trees," IEEE Trans. Inf. Theory, vol. IT-14, no. 3, pp. 462-467, May 1968.
4. A. Chechetka and C. Guestrin, "Efficient principled learning of thin junction trees," presented at the Neural Inf. Process. Syst., 2007.
5. M. J. Wainwright, P. Ravikumar, and J. D. Lafferty, "High- dimensional graphical model selection using -regularized logistic regression," in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2006, pp. 1465-1472.
6. 1-regularization," presented at the Neural Inf. Process. Syst., 2006.
7. J. Johnson, V. Chandrasekaran, and A. S. Willsky, "Learning Markov structure by maximum entropy relaxation," presented at the Artificial Intelligence and Statistics (AISTATS), 2007.
8. N. Meinshausen and P. Buehlmann, "High-dimensional graphs and variable selection with the Lasso," Ann. Statist., vol. 34, no. 3, pp. 1436-1462, Jun 2006. (Pubitemid 44231168)
9. M. Dudik, S. Phillips, and R. Schapire, "Performance guarantees for regularized maximum entropy density estimation," presented at the Conf. Learning Theor. (COLT), 2004.
10. C. K. Chow and T.Wagner, "Consistency of an estimate of tree-dependent probability distributions," IEEE Trans. Inf. Theory, vol. IT-19, no. 3, pp. 369-371, May 1973.
11. F. D. Hollander, Large Deviations (Fields Institute Monographs, 14). Providence, RI: AMS, 2000.
12. D. B.West, Introduction to Graph Theory, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall, 2000.
13. I. Csiszár and F. Matúš, "Information projections revisited," IEEE Trans. Inf. Theory, vol. 49, no. 6, pp. 1474-1490, Jun. 2003.
14. S. Borade and L. Zheng, "I-projection and the geometry of error exponents," presented at the Allerton Conf. Communication, Control, Comput., 2006.
15. S. Borade and L. Zheng, "Euclidean information theory," presented at the Allerton Conf. Communication, Control, Comput., 2007.
16. D. Karger and N. Srebro, "Learning Markov networks: Maximum bounded tree-width graphs," in Proc. Symp. Discrete Algorithms, 2001, pp. 392-401.
17. F. Bach and M. I. Jordan, "Thin junction trees," presented at the Neural Inf. Process. Syst., 2002.
18. D. Heckerman and D. Geiger, Learning Bayesian networks, Microsoft Research, Redmond, WA, 1994, Tech. Rep. MSR-TR-95-02.
19. G. Schwarz, "Estimating the dimension of a model," Ann. Statist., vol. 6, no. 2, pp. 461-464, 1978.
20. R. Tibshirani, "Regression shrinkage and selection via the Lasso," J. Roy. Statist. Soc. B, vol. 58, no. 1, pp. 267-288, 1996.
21. O. Zuk, S. Margel, and E. Domany, "On the number of samples needed to learn the correct structure of a Bayesian network," presented at the 22nd Annu. Conf. Uncertainty Artificial Intell. (UAI-06), 2006.
22. M. J. Choi, V. Y. F. Tan, A. Anandkumar, and A. S. Willsky, "Learning latent tree graphical models," J. Mach. Learn. Res., Sep. 2010, ArXiv:1009.2722, submitted for publication.
23. L. Paninski, "Estimation of entropy and mutual information," Neural Comput., vol. 15, no. 6, pp. 1191-1253, Jun 1, 2003. (Pubitemid 37049793)
24. A. Antos and I. Kontoyiannis, "Convergence properties of functional estimates for discrete distributions," Random Struct. Alg., pp. 163-193, 2001.
25. J.-R. Chazottes and D. Gabrielli, "Large deviations for empirical entropies of g-measures," Nonlinearity, vol. 18, pp. 2545-2563, Nov. 2005. (Pubitemid 41483187)
26. M. Hutter, "Distribution of mutual information," in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2001, pp. 399-406.
27. V. Y. F. Tan, A. Anandkumar, and A. S. Willsky, "Learning Gaussian tree models: Analysis of error exponents and extremal structures," IEEE Trans. Signal Process., vol. 58, no. 5, pp. 2701-2714, May 2010.
28. A. Kester and W. Kallenberg, "Large deviations of estimators," Ann. Statist., pp. 648-664, 1986.
29. R. Bahadur, S. Zabell, and J. Gupta, "Large deviations, tests, and estimates," Asymptotic Theor. Stat. Tests Estim., pp. 33-64, 1980.
30. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. Hoboken, NJ: Wiley-Interscience, 2006.
31. K. Ryu, "Econometric analysis of mixed parameter models," J. Econ. Theor. Econ., vol. 5, no. 113-124, 1999.
32. H. L. V. Trees, Detection, Estimation, and Modulation Theory, Part I. Hoboken, NJ: Wiley, 1968.
33. I. Csiszár and G. Tusnády, "Information geometry and alternating minimization procedures," Stat. Decisions, Supplementary, no. 1, pp. 205-237, Jul. 1984.
34. T. Cormen, C. Leiserson, R. Rivest, and C. Stein, Introduction to Algorithms, 2nd ed. New York: McGraw-Hill Science/Engineering/Math, 2003.
35. J. B. Kruskal, "On the shortest spanning subtree of a graph and the traveling salesman problem," presented at the Amer. Math. Soc., Feb. 1956.
36. R. C. Prim, "Shortest connection networks and some generalizations," Bell Syst. Tech. J., vol. 36, 1957.
37. W. Rudin, Principles of Mathematical Analysis, 3rd ed. New York: McGraw-Hill Science/Engineering/Math, 1976.
38. V. Y. F. Tan, A. Anandkumar, and A. S. Willsky, "Learning highdimensional Markov forest distributions: Analysis of error rates," J. Mach. Learning Research,May 2010, ArXiv:1005.0766, submitted for publication.
39. P. J. Huber and V. Strassen, "Minimax tests and the Neyman-Pearson lemma for capacities," Ann. Statist., vol. 1, pp. 251-263.
40. C. Pandit and S. P. Meyn, "Worst-case large-deviations with application to queueing and information theory," Stochastic Processes Appl., vol. 116, no. 5, pp. 724-756, May 2006. (Pubitemid 44262037)
41. O. Zeitouni and M. Gutman, "On universal hypotheses testing via large deviations," IEEE Trans. Inf. Theory, vol. 37, no. 2, pp. 285-290, Mar. 1991.
42. J. Unnikrishnan, D. Huang, S. Meyn, A. Surana, and V. V. Veeravalli, "Universal and composite hypothesis testing via mismatched divergence," IEEE Trans. Inf. Theory, to be published.
43. V. Y. F. Tan, A. Anandkumar, and A. S.Willsky, "Error exponents for composite hypothesis testing of Markov forest distributions," in Proc. Int. Symp. Inf. Theor., Austin, TX, Jun. 2010, pp. 1613-1617.
44. E. Abbe and L. Zheng, "Linear universal decoding for compound channels: An Euclidean geometric approach," in Proc. Int. Symp. Inf. Theor., 2008, pp. 1098-1102.
45. M. S. Pinsker, Information and Information Stability of Random Variables. Oakland, CA: Holden-Day, 1964.
46. J. N. Laneman, "On the distribution of mutual information," presented at the Inf. Theor. Applicat. Workshop, 2006.
47. S.-I. Amari and H. Nagaoka, Methods of Information Geometry. Providence, RI: AMS, 2000.
48. S.-I. Amari, "Information geometry on hierarchy of probability distributions," IEEE Trans. Inf. Theory, vol. 47, no. 5, pp. 1701-1711, Jul. 2001. (Pubitemid 32644689)
49. H. B. Mann and A. Wald, "On the statistical treatment of linear stochastic difference equations," Econometrics, vol. 11, pp. 173-220, 1943.
50. P. Billingsley, Weak Convergence of Measures: Applications in Probability. Philadelphia, PA: SIAM, 1987.
51. I. Csiszár and J. Korner, Information Theory: Coding Theorems For Discrete Memoryless Systems. Budapest, Hungary: Akademiai Kiado, 1997.