Kolmogorov's structure functions and model selection

IEEE Transactions on Information Theory

Volumn 50, Issue 12, 2004, Pages 3265-3290

  Vereshchagin, Nikolai K ^a   Vitányi, Paul M B ^b,c  

^a MOSCOW STATE UNIVERSITY   (Russian Federation)

^b CWI   (Netherlands)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Computability; Constrained best fit model selection; Constrained maximum likelihood (ML); Constrained minimum description length (MDL); Function prediction; Kolmogorov complexity; Kolmogorov structure function; Lossy compression; Minimal sufficient statistic; Nonprobabilistic statistics; Sufficient statistic

Indexed keywords

COMPUTATIONAL COMPLEXITY; CONSTRAINT THEORY; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; PROBABILITY; RANDOM PROCESSES; STATISTICAL METHODS;

CONSTRAINED BEST FIT MODEL SELECTION; CONSTRAINED MAXIMUM LIKELIHOOD; CONSTRAINED MINIMUM DESCRIPTION LENGTH (MDL); FUNCTION PREDICTION; KOLMOGOROV COMPLEXITY; KOLMOGOROV STRUCTURE FUNCTION; LOSSY COMPRESSION; MINIMAL SUFFICIENT STATISTIC; NONPROBABILISTIC STATISTIC;

INFORMATION THEORY;

EID: 10644226108     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2004.838346     Document Type: Article

References (30)

1. A. R. Barron and T. M. Cover, "Minimum complexity density estimation," IEEE Trans. Inform. Theory, vol. 37, pp. 1034-1054, July 1991.
2. A. R. Barron, J. Rissanen, and B. Yu, "The minimum description length principle in coding and modeling," IEEE Trans. Inform. Theory, vol. 44, pp. 2743-2760, Oct. 1998.
3. T. M. Cover, "Kolmogorov complexity, data compression, and inference," in The Impact of Processing Techniques on Communications, J. K. Skwirzynski, Ed. Dordrecht, Zuid-Holland, The Netherlands: Martinus Nijhoff, 1985, pp. 23-33.
4. T. M. Cover, "Attending A. N. Kolmogorov's Talk at the Information Theory Symposium in Tallinn, Estonia, 1974," e-mail to P. Vitányi, Jan. 24, 2002.
5. T. M. Cover, P. Gács, and R. M. Gray, "Kolmogorov's contributions to information theory and algorithmic complexity," Ann. Probab., vol. 17, pp. 840-865, 1989.
6. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
7. R. A. Fisher, "On the mathematical foundations of theoretical statistics," Philos. Trans. Roy. Soc. London, Ser. A, vol. 222, pp. 309-368, 1922.
8. P. Gács, "On the symmetry of algorithmic information," Sov. Math. - Dokl., vol. 15, pp. 1477-1480, 1974.
9. P. Gács, "Attending A. N. Kolmogorov's Talk at the Information Theory Symposium in Tallinn, Estonia, 1974 ," e-mail to P. Vitányi, Jan. 24, 2002.
10. P. Gács, J. Tromp, and P. M. B. Vitányi, "Algorithmic statistics," IEEE Trans. Inform. Theory, vol. 47, pp. 2443-2463, Sept. 2001.
11. Q. Gao, M. Li, and P. M. B. Vitányi, "Applying MDL to learn best model granularity," Artificial Intell., vol. 121, pp. 1-29, 2000.
12. P. S. Laplace, A Philosophical Essay on Probabilities, 1819. New York: Dover, 1951. English translation.
13. L. A. Levin, e-mails to P. Vitányi, Feb. 7,11,20, 2002.
14. M. Li and P. M. B. Vitányi, An Introduction to Kolmogorov Complexity and its Applications, 2nd ed. New York: Springer-Verlag, 1997.
15. A. N. Kolmogorov, "Three approaches to the quantitative definition of information," Probl. Inform. Transm., vol. 1, no. 1, pp. 1-7, 1965.
16. A. N. Kolmogorov, "Complexity of algorithms and objective definition of randomness," A Talk at Moscow Math. Soc. Meet., Apr. 16, 1974.
17. A. N. Kolmogorov, Talk at the Information Theory Symp., Tallinn, Estonia, 1974.
18. P. Martin-Löf, "The definition of random sequences," Inform. Contr., vol. 9, pp. 602-619, 1966.
19. J. J. Rissanen, "A universal prior for integers and estimation by minimum description length," Ann. Statist., vol. 11, no. 2, pp. 416-431, 1983.
20. J. J. Rissanen, "Kolmogorov's structure function for probability models," in Proc. IEEE Information Theory Workshop, 2002, pp. 98-99.
21. C. E. Shannon, "The mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, 1948.
22. A. K. Shen, "The concept of (α, β)-stochasticity in the Kolmogorov sense, and its properties," Sov. Math. - Dokl., vol. 28, no. 1, pp. 295-299, 1983.
23. A. K. Shen, "Discussion on Kolmogorov complexity and statistical analysis," Comput. J., vol. 42, no. 4, pp. 340-342, 1999.
24. P. M. B. Vitányi and M. Li, "Minimum description length induction, Bayesianism, and Kolmogorov complexity," IEEE Trans. Inform. Theory vol. 46, pp. 446-464, Mar. 2000.
25. P. M. B. Vitányi. Meaningful Information. Los Alamos National Laboratories Archives. [Online]. Available: http://xxx.lanl.gov/abs/cs. CC/0111053
26. V. G. Vovk, "Learning about the parameter of the Bernoulli model," J. Comput. System Sci., vol. 55, pp. 96-104, 1997.
27. V. V. V'yugin, "On the defect of randomness of a finite object with respect to measures with given complexity bounds," SIAM Theory Probab. Appl., vol. 32, no. 3, pp. 508-512, 1987.
28. V. V. V'yugin, "Algorithmic complexity and stochastic properties of finite binary sequences," Comput. J., vol. 42, no. 4, pp. 294-317, 1999.
29. V. V. V'yugin, "Does snooping help?," Theoret. Comput. Sci., vol. 276, no. 1/2, pp. 407-415, 2002.
30. C. S. Wallace and P. R. Freeman, "Estimation and inference by compact coding," J. Roy. Statist. Soc., Ser. B, vol. 49, pp. 240-251, 1987.