The empirical distribution of rate-constrained source codes

IEEE Transactions on Information Theory

Volumn 51, Issue 11, 2005, Pages 3718-3733

  Weissman, Tsachy ^a   Ordentlich, Erik ^b  

^a Stanford University   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

Compression based denoising; Empirical distribution; Good codes; Lossy source codes; Pointwise bounds; Shannon lower bound; Strong converse

Indexed keywords

BLOCK CODES; CONVERGENCE OF NUMERICAL METHODS; DATA COMPRESSION; PROBABILITY DISTRIBUTIONS; RANDOM PROCESSES; THEOREM PROVING;

COMPRESSION-BASED DENOISING; EMPIRICAL DISTRIBUTION; LOSSY SOURCE CODES; POINTWISE BOUNDS; RATE DISTORTION FUNCTION; RATE-CONSTRAINED SOURCE CODES; SHANNON LOWER BOUND; STRONG CONVERSE;

INFORMATION THEORY;

EID: 27744572437     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.856982     Document Type: Article

References (35)

1. T. Berger, Rate-Distortion Theory: A Mathematical Basis for Data Compression. Englewood Cliffs, N.J.: Prentice-Hall, 1971.
2. T. Berger and J. D. Gibson, "Lossy source coding," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2693-2723, Oct. 1998.
3. G. Chang, B. Yu, and M. Vetterli, "Bridging compression to wavelet thresholding as a denoising method," in Proc. Conf.Information Sciences and Systems, Johns Hopkins Univ., Baltimore, MD, Mar. 1997.
4. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
5. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
6. A. Dembo and T. Weissman, "The minimax distortion redundancy in noisy source coding," IEEE Trans. Inf. Theory, vol. 49, no. 11, pp. 3020-3030, Nov. 2003.
7. D. Donoho. (2002, Jan.) The Kolmogorov Sampler. [Online]. Available: http://www-stat.stanford.edu/donoho/
8. Y. Ephraim and N. Merhav, "Hidden Markov processes," IEEE Trans. Inf. Theory, vol. 48, no. 6, pp. 1518-1569, Jun. 2002.
9. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
10. R. M. Gray, "Information rates of autoregressive processes," IEEE Trans. Inf. Theory, vol. IT-16, no. 4, pp. 412-421, Jul. 1970.
11. R. M. Gray, "Rate distortion functions for finite-state finite-alphabet Markov sources," IEEE Trans. Inf. Theory, vol. IT-17, no. 2, pp. 127-134, Mar. 1971.
12. J. Hannan, "Approximation to Bayes risk in repeated play," in Contributions to the Theory of Games, Ann. Math. Study. Princeton, NJ: Princeton Univ. Press, 1957, vol. III, pp. 97-139.
13. R. Jörnstein and B. Yu, "Adaptive Quantization," Dep. Statistics, Univ. Calif., Berkeley, Tech. Rep., 2001.
14. A. Kanlis, "Compression and transmission of information at multiple resolutions," Ph.D. dissertation, Elec. Eng. Dep., Univ. Maryland, College Park, Aug. 1997.
15. A. Kanlis, S. Khudanpur, and P. Narayan, "Typicality of a good rate-distortion code" (in Russian), Probl. Pered. Inform. (Probl. Inf Transm.), vol. 32, no. 1, pp. 96-103, 1996. Special issue in honor of M. S. Pinsker.
16. J. C. Kieffer, "A unified approach to weak universal source coding," IEEE Trans. Inf. Theory, vol. IT-24, no. 6, pp. 674-682, Nov. 1978.
17. J. C. Kieffer, "An almost sure convergence theorem for sequences of random variables selected from log-convex sets," in Proc. Conf.Almost Everywhere Convergence in Problems in Ergodic Theory II. Boston, MA: Academic, 1991, pp. 151-166.
18. J. C. Kieffer, "Sample converses in source coding theory," IEEE Trans. Inf. Theory, vol. 37, no. 2, pp. 263-268, Mar. 1991.
19. I. Kontoyiannis, "Pointwise redundancy in lossy data compression and universal lossy data compression," IEEE Trans. Inf. Theory, vol. 46, no. 1, pp. 136-152, Jan. 2000.
20. N. Merhav and M. Feder, "Universal prediction," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2124-2147, Oct. 1998.
21. B. Natarajan, "Filtering random noise via data compression," in Proc. Data Compression Conf. (DCC'93), Snowbird, UT, Mar. 1993, pp. 60-69.
22. B. Natarajan, "Filtering random noise from deterministic signals via data compression," IEEE Trans. Signal Process., vol. 43, no. 11, pp. 2595-2605, Nov. 1995.
23. B. Natarajan, K. Konstantinides, and C. Herley, "Occam filters for stochastic sources with application to digital images," IEEE Trans. Signal Process., vol. 46, no. 11, pp. 1434-1438, Nov. 1998.
24. D. L. Neuhoff and P. C. Shields, "Simplistic universal coding," IEEE Trans. Inf. Theory, vol. 44, no. 2, pp. 778-781, Mar. 1998.
25. J. Rissanen, "MDL denoising," IEEE Trans. Inf.Theory, vol. 46, no. 7, pp. 2537-2543, Nov. 2000.
26. E. Samuel, "An empirical Bayes approach to the testing of certain parametric hypotheses," Ann. Math. Statist., vol. 34, no. 4, pp. 1370-1385, 1963.
27. S. Shamai and S. Verdd, "The empirical distribution of good codes," IEEE Trans. Inf. Theory, vol. 43, no. 3, pp. 836-846, May 1997.
28. I. Tabus, J. Rissanen, and J. Astola, "Normalized maximum likelihood models for Boolean regression with application to prediction and classification in genomics," in Computational Statististical Approaches to Genomics, W. Zhang and I. Shmulevich, Eds. Norwell, MA: Kluwer Academic, 2002, pp. 173-196.
29. N. Tishby, F. Pereira, and W. Bialek, "The information bottleneck method," in Proc. 37th Allerton Conf. Communication and Computation, Monticello, IL, Sep. 1999, pp. 368-377.
30. T. Weissman, Not All Universal Source Codes are Pointwise Universal. [Online]. Available: http://www.stanford.edu/~tsachy/interest.htm
31. T. Weissman and N. Merhav, "On competitive prediction and its relationship to rate-distortion theory," IEEE Trans. Inf. Theory, vol. 49, no. 12, pp. 3185-3193, Dec. 2003.
32. T. Weissman, E. Ordentlich, G. Seroussi, S. Verdú, and M. Weinberger, "Universal discrete denoising: Known channel," IEEE Trans. Inf. Theory, vol. 51, no. 1, pp. 5-28, Jan. 2005.
33. E. H. Yang and J. Kieffer, "Simple universal lossy data compression schemes derived from the Lempel-Ziv algorithm," IEEE Trans. Inf. Theory, vol. 42, no. 1, pp. 239-245, Jan. 1996.
34. J. Ziv, "Coding of sources with unknown statistics-Part II: Distortion relative to a fidelity criterion," IEEE Trans. Inf. Theory, vol. IT-18, no. 2, pp. 389-394, May 1972.
35. J. Ziv, "Distortion-rate theory for individual sequences," IEEE Trans. Inf. Theory, vol. IT-26, no. 2, pp. 137-143, Mar. 1980.