The Minimum Description Length Principle in Coding and Modeling

IEEE Transactions on Information Theory

Volumn 44, Issue 6, 1998, Pages 2743-2760

  Barron, Andrew ^a,b   Rissanen, Jorma ^a,c   Yu, Bin ^a,d  

^a IEEE   (United States)

^b YALE UNIVERSITY   (United States)

^c IBM ALMADEN RESEARCH CENTER   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Complexity; Compression; Estimation; Inference; Universal modeling

Indexed keywords

COMPUTATIONAL COMPLEXITY; DATA COMPRESSION; MAXIMUM LIKELIHOOD ESTIMATION; RANDOM PROCESSES; STATISTICAL METHODS;

INFERENCE; MINIMUM DESCRIPTION LENGTH PRINCIPLE; STOCHASTIC COMPLEXITY; UNIVERSAL MODELING;

ENCODING (SYMBOLS);

EID: 0032183995     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.720554     Document Type: Article

References (60)

1. H. Akaike, "Information theory and an extension of the maximum likelihood principle," in Proc. 2nd Int. Symp. Information Theory, Petrov and Csaki, Eds. Budapest: Akademia Kiado, 1973, pp. 267-281.
2. A. R. Barren, "Logically smooth density estimation," Ph.D. dissertation, Dept. Elec. Eng., Stanford Univ., Stanford, CA, Sept. 1985.
3. "Are Bayes rules consistent in information?" in Problems in Communications and Computation, T. M. Cover and B. Gopinath, Eds. New York: Springer-Verlag, 1987, pp. 85-91.
4. "The exponential convergence of posterior probabilities with implications for Bayes estimators of density functions," Tech. Rep. 7, Dept. Statist., Univ. Illinois, Apr. 1988
5. A. R. Barron and T. M. Cover, "Minimum complexity density estimation," IEEE Trans. Inform. Theory, vol. 37, July 1991.
6. A. R. Barron and N. Hengartner, "Information theory and superefficiency," Ann. Statist., 1998, to be published.
7. A. R. Barron, Y. Yang, and B. Yu, "Asymptotically optimal function estimation by minimum complexity criteria," in Proc. 1994 IEEE Int. Symp. Information Theory.
8. L. Birgé, "Approximation dans les espaces métriques et théorie de l'estimation," Z. Wahrscheinlichkeitstheor. Verw. Geb., vol. 65, pp. 181-237, 1983.
9. B. S. Clarke and A. R. Barron "Information-theoretic asymptotics of Bayes methods," IEEE Trans. Inform. Theory, vol. 36, pp. 453-471, May 1990.
10. "Jeffreys' prior is asymptotically least favorable under entropy risk," J. Statist. Planning and Infer., vol. 41, pp. 37-60, 1994.
11. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
12. L. D. Davisson, "Universal noiseless coding," IEEE Trans. Inform. Theory, vol. IT-19, pp. 783-795, Nov. 1973.
13. "Minimax noiseless universal coding for Markov sources," IEEE Trans. Inform. Theory, vol. IT-29, pp. 211-215, Mar. 1983.
14. L. D. Da vison and A. Leon-Garcia, "A source matching approach to finding minimax codes," IEEE Trans. Inform. Theory, vol. IT-26, pp. 166-174, 1980.
15. A. P. Dawid, "Present position and potential developments: Some personal views, statistical theory, the prequential approach," J. Roy. Statist. Soc. A, vol. 147, PART. 2, pp. 278-292, 1984
16. "Prequential analysis, stochastic complexity and Bayesian in-ference," presented at the Fourth Valencia International Meeting on Bayesian Statistics, Peniscola, Spain, Apr. 15-20, 1991.
17. "Prequential data analysis," Current Issues in Statistical Infer-ence: Essays in Honor of D. Basu, IMS Monograph 17, M. Ghosh and P. K. Pathak, Eds., 1992.
18. L. Devroye, A Course in Density Estimation. Basel, Switzerland: Birkhäuser-Verlag, 1987.
19. B. Dom, "MDL estimation for small sample sizes and its application to linear regression," IBM Res. Rep. RJ 10030, June 13, 1996, also submitted for publication.
20. J. L. Doob, Stochastic Processes. New York: Wiley, 1953.
21. R. A. Fisher, "On the mathematical foundations of theoretical statistics," Phil. Trans. Roy. Soc. London, Ser. A, vol. 222, pp. 309-368, 1922.
22. E. J. Hannan, "The estimation of the order of an ARMA process" Ann. Statist., vol. 6, no. 2, pp. 461-464, 1978.
23. E. J. Hannan, A. J. McDougall, and D. S. Poskitt, "The determination of the order of an autoregression," J. Roy. Statist. Soc. Ser. B, vol. 51, pp. 217-233, 1989
24. J. A. Hartigan, Bayes Theory. New York: Springer-Verlag, 1983.
25. R. Z. Hasminskii, "A lower bound on the risks of nonparametric estimates of densities in the uniform metric." Theory Probab. Its Applic., vol. 23, pp. 794-796, 1978.
26. R. Z. Hasminsikii and I. A. Ibragimov, "On density estimation in the view of Kolmogorov's ideas in approximation theory," Ann. Statist., vol. 18, pp. 999-1010, 1990.
27. D. Haughton, "Size of the error in the choice of a model fit data from an exponential family," Sankhya Ser. A, vol. 51, pp. 45-58, 1989.
28. D. Haussier, "A general minimax result for relative entropy," IEEE Trans. Inform. Theory, vol. 43, pp. 1276-1280, July 1997.
29. D. Haussler and A. Barron, "How well do Bayes methods work for online prediction of {±1} values?" in Proc. NEC Symp. Computation and Cognition, 1992.
30. D. Haussler and M. Opper, "Mutual information, metric entropy and cumulative relative entropy risk," Ann. Statist., vol. 25, no. 6, 1997.
31. E. M. Hemerly and M. H. A. Davis, "Strong constistency of the predictive least squares criterion for order determination of autoregressive processes," Ann. Statist., vol. 17, pp. 941-946, 1989.
32. U. Hjorth, "Model selection and forward validation," Scand. J. Statist., vol. 9, pp. 95-105, 1982.
33. J. Jeffreys, Theory of Probability, 3rd ed. Oxford, U.K.: Oxford Univ. Press, 1961.
34. A. N. Kolmogorov, "Three approaches to the quantitative definition of information," Probl. Inform. Transm., vol. 1, pp. 4-7, 1965.
35. L. LeCam, "On some asymptotic properties of maximum likelihood and related Bayes' estimates," in University of California Publications in Statistics, J. Neyman, M. Loeve, and O. Struve, Eds. London, U.K.: Cambridge Univ. Press, 1953, pp. 277-329.
36. N. Merhav and J. Ziv, "Estimating the number of states of a finite-state source," IEEE Trans. Inform. Theory, vol. 41, pp. 61-65, Jan. 1992.
37. N. Merhav and M. Feder, "A strong version of the redundancy-capacity theory of universal coding," IEEE Trans. Inform. Theory, vol. 41, pp. 714-722, May 1995.
38. J. Rissanen, "Modeling by shortest data description," Automatica, vol. 14, pp. 465-471, 1978.
39. J. Rissanen and G. G. Langdon, Jr., "Universal modeling and coding," IEEE Trans. Inform. Theory, vol. IT-27, pp. 12-23, Jan. 1981.
40. J. Rissanen, "A universal data compression system," IEEE Trans. Inform. Theory, vol. IT-29, pp. 656-664, Sept. 1983.
41. "Universal coding, information, prediction, and estimation," IEEE Trans. Inform. Theory, vol. IT-30, pp. 629-636, July 1984.
42. "Stochastic complexity and modeling," Ann. Statist., vol. 14, pp. 1080-1100, 1986.
43. "Fisher information and stochastic complexity," IEEE Trans. Inform. Theory, vol. 42, pp. 40-47, Jan. 1996.
44. "Hypothesis selection and testing by the MDL principle," invited paper for the special issue of Comp. J. devoted to Kolmogorov complexity, 1997.
45. J. Rissanen, T. Speed, and B. Yu, "Density estimation by stochastic complexity," IEEE Trans. Inform. Theory, vol. 38, pp. 315-323, Mar. 1992.
46. Y. M. Shtarkov, "Universal sequential coding of single messages," Probl. Inform. Transm., vol. 23, no. 3, pp. 3-17, July-Sept. 1987.
47. R. J. Solomonoff, "A formal theory of inductive inference," Part I, Inform. Contr., vol. 7, pp. 1-22;
48. Part II, Inform. Contr., vol. 7, pp. 224-254. 1964.
49. T. P. Speed and B. Yu, "Model selection and prediction: Normal regression," Ann. Inst. Statist. Math., vol. 45, pp. 35-54, 1994.
50. G. Schwarz, "Estimation the dimension of a model" Ann. Statist., vol. 6, no. 2, pp. 461-464, 1978.
51. J. Takeuchi and A. R. Barron, "Asymptotically minimax regret for exponential and curved exponential families," in Proc. IEEE 1998 Int. Symp. Information Theory (Boston, MA, Aug. 1998).
52. A. I. Tulcea, "Contributions to information theory for abstract alphabets," Arkiv für Matematik, vol. 4, no. 18, pp. 235-247, 1960.
53. C. S. Wallace and D. M. Boulton, "An information measure for classification," Comp. J., vol. 11, no. 2, pp. 185-195, 1968.
54. C. Z. Wei, "On the predictive least squares principle," Ann. Statist., vol. 20, pp. 1-2, 1992.
55. M. J. Weinberger, J. Rissanen, and M. Feder, "A universal finite memory source," IEEE Trans. Inform. Theory, vol. 41, pp. 643-652, May 1995.
56. F. M. J. Willems, Y. M. Shtarkov, and T. J. Tjalkens, "The context-tree weighting method: Basic properties," IEEE Trans. Inform. Theory, vol. 41, pp. 653-664, May 1995.
57. Q. Xie and A. R. Barron, "Asymptotic minimax regret for data compression, gambling and prediction,"IEEE Trans. Inform. Theory, submitted for publication, 1996.
58. Y. H. Yang and A. R. Barron, "Information theoretic determination of minimax rates of convergence," Annals of Statistics, submitted for publication Oct. 1995, revised Mar. 1997.
59. B. Yu, "Assouad, Fano, and Le Cam," Festschrift in Honor ofL. Le Cam on His 70th Birthday, D. Pollard and G. Yang, Eds. Berlin, Germany: Springer-Verlag, 1997.
60. B. Yu and T. P. Speed, "Data compression and histograms," Probab. Theory Related Fields, vol. 92, pp. 195-229, 1992.