Exact minimax strategies for predictive density estimation, data compression, and model selection

IEEE Transactions on Information Theory

Volumn 50, Issue 11, 2004, Pages 2708-2726

  Liang, Feng ^a   Barron, Andrew ^b  

^a DUKE UNIVERSITY   (United States)

^b YALE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CHANNEL CAPACITY; DATA COMPRESSION; DECISION THEORY; ENCODING (SYMBOLS); INTEGRATION; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; PROBABILITY DENSITY FUNCTION; REGRESSION ANALYSIS; THEOREM PROVING;

HAAR MEASURE; HUNT-STEIN INVARIANCE; KULLBACK-LEIBER DIVERGENCE; MINIMAX RISK; MINIMUM DESCRIPTION LENGTH (MDL); MODEL SELECTION; PREDICTIVE DENSITY ESTIMATION; UNIVERSAL CODING;

INFORMATION THEORY;

EID: 8144227174     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2004.836922     Document Type: Article

References (58)

1. J. Aitchison, "Goodness of prediction fit," Biometrika, vol. 62, pp. 547-554, 1975.
2. M. Aslan, "Asymptotically minimax Bayes predictive densities," Ph.D. dissertation, Yale Univ., New Haven, CT, 2002.
3. A. R. Barron, "Logically smooth density estimation," Ph.D. dissertation, Stanford Univ., Stanford, CA, 1985.
4. A. R. Barron, "Information-theoretic characterization of Bayes performance and the choice of priors in parametric and nonparametric problems," in Bayesian Statistics, A. P. Dawid, J. M. Bernardo, J. O. Berger, and A. F. M. Smith, Eds. Oxford, U.K.: Oxford Univ. Press, 1998, vol. 6, pp. 27-52.
5. A. R. Barron, J. Rissanen, and B. Yu, "The minimum description length principle in coding and modeling," IEEE Trans. Inform. Theory, vol. 44, pp. 2743-2760, Oct. 1998.
6. J. O. Berger, Statistical Decision Theory and Bayesian Analysis. New York: Springer-Verlag, 1985.
7. D. Blackwell and M. A. Girshick, Theory of Games and Statistical Decisions. New York: Wieley, 1954.
8. J. V. Bondar and P. Milnes, "Amenability: A survey for statistical applications of Hunt-Stein and related conditions on groups," Z. Wahrscheinlichkeitstheorie und Verwandte Gebiete [Became: Ψ(ProbTher ], vol. 57, pp. 103-128, 1981.
9. L. D. Brown, "On the admissibility of invariant estimators of one or more location parameters," Ann. Math. Statist., vol. 37, pp. 1087-1136, 1966.
10. L. D. Brown, "Inadmissibility of the usual estimators of scale parameters in problems with unknown location and scale parameters," Ann. Math. Statist., vol. 39, pp. 29-48, 1968.
11. L. D. Brown, "Commentary on paper [19]," in Jack Carl Kiefer. Collected Papers, Supplementary Volume, L. D. Brown, I. Olkin, J. Sacks, and H. P. Wynn, Eds. New York: Springer-Verlag, 1986, pp. 20-27.
12. K. L. Chung, A Course in Probability Theory. New York: Academic, 1974.
13. B. S. Clarke and A. R. Barron, "Information-theoretic asymptotics of Bayes methods," IEEE Trans. Inform. Theory, vol. 36, pp. 453-471, May 1990.
14. B. S. Clarke, "Jeffrey's prior is asymptotically least favorable under entropy risk," J. Statist. Planning and Inference, vol. 41, pp. 37-60, 1994.
15. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
16. I. Csiszár, "Information-type measures of difference of probability distributions and indirect observations," Studia Sci. Math. Hungar., vol. 2, pp. 299-318, 1967.
17. I. Csiszár, "Generalized entropy and quantization problems," in Trans. 6th Prague Conf. Information Theory, Statistical Decision Functions and Random Processes, 1973, pp. 159-174.
18. L. D. Davisson, "Universal noiseless coding," IEEE Trans. Inform. Theory, vol. IT-19, pp. 783-795, Nov. 1973.
19. L. D. Davisson and A. Leon-Garcia, "A source matching approach to finding minimax codes," IEEE Trans. Inform. Theory, vol. IT-26, pp. 166-174, Mar. 1980.
20. M. L. Eaton, Group Invariance Applications in Statistics. Beachwood, OH: Institute of Mathematical Statistics, 1989, IMS Lecture Notes - Monograph Series.
21. T. S. Ferguson, Mathematical Statistics, A Decision Theoretic Approach. New York: Academic, 1967.
22. R. Gallager, "Source coding with side information and universal coding," MIT Lab. Inform. Decision Syst., Cambridge, MA, Tch. Rep. LIDS-P-937, 1979.
23. M. A. Girshick and L. J. Savage, "Bayes and minimax estimates for quadratic loss functions," in Proc. 2nd Berkeley Symp. Mathematical Statistics and Probability, Berkeley, CA, 1951, pp. 53-73.
24. R. M. Gray, Entropy and Information Theory. New York: Springer-Verlag, 1990.
25. J. Hartigan, Bayes Theory. New York: Springer-Verlag, 1983.
26. J. Hartigan, "The maximum likelihood prior," Ann. Statist., vol. 26, no. 6, pp. 2083-2103, 1998.
27. D. Haussler, "A general minimax result for relative entropy," IEEE Trans. Inform. Theory, vol. 43, pp. 1276-1280, July 1997.
28. J. A. Hoeting, D. Madigan, A. E. Raftery, and C. T. Volinsky, " Bayesian model averaging: A tutorial," Statist. Sci., vol. 14, no. 4, pp. 382-401, 1999.
29. W. James and C. Stein, "Estimation with quadratic loss," in Proc. 4th Berkeley Symp. Mathematical Statistics and Probability, vol. 1, Berkeley, CA, 1960, pp. 361-379.
30. H. Jeffreys, Theory of Probability. New York: Oxford Univ. Press, 1961.
31. O. Kallenberg, Foundations of Modern Probability. Berlin, Germany: Springer-Verlag, 1997.
32. J. Kiefer, "Invariance, minimax sequential estimation, and continuous time processes," Ann. Math. Statist., vol. 28, pp. 537-601, 1957.
33. J. Kiefer, "Multivariate optimality results," Multivariate Anal., pp. 255-274, 1966.
34. R. E. Krichevsky and V. K. Trofimov, "The performance of universal encoding," IEEE Trans. Inform. Theory, vol. IT-27, pp. 199-207, Mar. 1981.
35. S. Kullback, J. C. Keegel, and J. H. Kullback, Topics in Statistical Information Theory. Berlin, Germany: Springer-Verlag, 1980.
36. L. M. Le Cam, Asymptotic Methods in Statistical Decision Theory. New York: Springer-Verlag, 1986.
37. E. L. Lehmann, Testing Statistical Hypotheses. New York: Springer-Verlag, 1986.
38. E. L. Lehmann and G. Casella, Theory of Point Estimation. New York: Springer-Verlag, 1998.
39. F. Liang, "Exact minimax procedures for predictive density estimation and data compression," Ph.D. dissertation, Yale Univ., New Haven, CT, 2002.
40. N. Merhav and M. Feder, "A strong version of the redundancy-capacity theorem of universal coding," IEEE Trans. Inform. Theory, vol. 41, pp. 714-722, May 1995.
41. M. S. Pinsker, Information and Information Stability of Random Variables. San Francisco, CA: Holden Day, 1964. Translated by A. Feinstein.
42. E. J. G. Pitman, "The estimation of location and scale parameters of a continuous population of any given form," Biometrika, vol. 30, pp. 391-421, 1939.
43. D. Pollard, A User's Guide to Measure Theoretic Probability. Cambridge, U.K.: Cambridge Univ. Press, 2002.
44. J. Rissanen, "Modeling by shortest data description," Automatica, vol. 14, pp. 465-471, 1978.
45. J. Rissanen, "Universal coding, information, prediction and estimation," IEEE Trans. Inform. Theory, vol. IT-30, pp. 629-636, 1984.
46. J. Rissanen, "Stochastic complexity," J. Roy. Statist. Soc., vol. 49, no. 3, pp. 223-239, 1987.
47. J. Rissanen, Stochastic Complexity and Statistical Inquiry. Singapore: World Scientific, 1989.
48. J. Rissanen, "Fisher information and stochastic complexity," IEEE Trans. Inform. Theory, vol. 42, pp. 40-47, Jan. 1996.
49. B. Y. Ryabko, "The encoding of a source with unknown but ordered probabilities," Probl. Inform. Transm., vol. 14, pp. 71-77, 1979.
50. B. Y. Ryabko, "Twice-universal coding," Probl. Inform. Transm., vol. 3, pp. 173-177, 1984.
51. B. Y. Ryabko, "Prediction of random sequences and universal coding," Probl. Inform. Transm., vol. 24, pp. 87-96, 1988.
52. C. Stein, "Inadmissibility of the usual estimator for the mean of a multivariate normal distribution," in Proc. 3rd Berkeley Symp. Mathematical Statistics and Probability, vol. 1, Berkeley, CA, 1956, pp. 197-206.
53. C. Stein, "The sdmissibility of Pitman's estimator of a single location parameter," Ann. Math. Statist., vol. 30, pp. 970-979, 1959.
54. H. Strasser, Mathematical Theory of Statistics: Statistical Experiments and Asymptotic Theory. Berlin, Germany: De Gruyter. 1985, De Gruyter Studies in Mathematics.
55. W. E. Strawderman, "Proper Bayes minimax estimators of the multivariate normal mean," Ann. Math. Statist., vol. 42, pp. 385-388, 1971.
56. I. Vajda, "On convergence of information contained in quantized observations," IEEE Trans. Inform. Theory, vol. 48, pp. 2163-2172, 2002.
57. O. Wesler, "Invariance theory and a modified minimax principle," Ann. Math. Statist., vol. 30, pp. 1-20, 1959.
58. Q. Xie and A. R. Barron, "Minimax redundancy for the class of memoryless sources," IEEE Trans. Inform. Theory, vol. 43, pp. 646-657, Mar. 1997.