Rényi information dimension: Fundamental limits of almost lossless analog compression

IEEE Transactions on Information Theory

Volumn 56, Issue 8, 2010, Pages 3721-3748

  Wu, Yihong ^a   Verdú, Sergio ^a  

^a Princeton University   (United States)

Author keywords

Analog compression; compressed sensing; information measures; Rnyi information dimension; Shannon theory; source coding

Indexed keywords

COMPRESSED SENSING; DISCRETE SOURCES; FUNDAMENTAL LIMITS; INFORMATION DIMENSIONS; LIPSCHITZ CONTINUITY; LOSSLESS; LOSSLESS CODING; LOSSLESS SOURCE CODING; MEMORYLESS SOURCE; OPTIMAL COMPRESSION; REGULARITY CONDITION; SHANNON THEORY; SOURCE-CODING;

COMPRESSORS; INFORMATION THEORY;

SIGNAL RECONSTRUCTION;

EID: 77954596738     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2010.2050803     Document Type: Article

References (55)

1. C. E. Shannon, "A mathematical theory of communication," in Bell Syst. Tech. J., 1948, vol.27, pp. 379-423, 623-56.
2. D. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol.52, no.4, pp. 1289-1306, Apr. 2006.
3. E. Candes, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol.52, no.2, pp. 489-509, Feb. 2006.
4. M.Wainwright, "Information-theoretic bounds on sparsity recovery in the high-dimensional and noisy setting," in Proc. IEEE Int. Symp. Inf. Theory, Nice, France, Jun. 2007.
5. D. L. Donoho, A. Maleki, and A. Montanari, "Message-passing algorithms for compressed sensing," Proc. Nat. Acad. Sci., vol.106, no.45, pp. 18914-18919, Nov. 2009.
6. D. L. Donoho, A. Maleki, and A. Montanari, "Construction of message passing algorithms for compressed sensing," to be submitted to IEEE Trans Inf. Theory.
7. Y. Eftekhari, A. H. Banihashemi, and I. Lambadaris, "An efficient approach toward the asymptotic analysis of node-based recovery algorithms in compressed sensing," 2010 [Online]. Available: http://arxiv. org/abs/1001.2284
8. P. Schniter, "Turbo reconstruction of structured sparse signals," in Proc. Conf. Inf. Sci. Syst., Princeton, NJ, Mar. 2010.
9. J. R. Munkres, Topology, 2nd ed. Englewood Cliffs, NJ: Prentice- Hall, 2000.
10. A. Montanari and E. Mossel, "Smooth compression, Gallager bound and nonlinear sparse graph codes," in Proc. IEEE Int. Symp. Inf. Theory, Toronto, ON, Canada, Jul. 2008.
11. A. Rényi, "On the dimension and entropy of probability distributions," Acta Mathematica Hungarica, vol.10, no.1-2, Mar. 1959.
12. T. Kawabata and A. Dembo, "The rate-distortion dimension of sets and measures," IEEE Trans. Inf. Theory, vol.40, no.5, pp. 1564-1572, Sep. 1994.
13. Y. B. Pesin, Dimension Theory in Dynamical Systems: Contemporary Views and Applications. Chicago, IL: Univ. Chicago Press, 1997.
14. B. R. Hunt and V. Y. Kaloshin, "How projections affect the dimension spectrum of fractal measures," Nonlinearity, vol.10, pp. 1031-1046, 1997.
15. E. Çinlar, Probability and Stochastics. New York: Springer-Verlag, 2010.
16. A. Rényi, Probability Theory. Amsterdam, The Netherlands: North- Holland, 1970.
17. K. Falconer, Techniques in Fractal Geometry. New York: Wiley, 1997.
18. K. Falconer, Fractal Geometry: Mathematical Foundations and Applications, 2nd ed. New York: Wiley, 2003.
19. A. György, T. Linder, and K. Zeger, "On the rate-distortion function of random vectors and stationary sources with mixed distributions," IEEE Trans. Inf. Theory, vol.45, pp. 2110-2115, 1999.
20. T. Linder and R. Zamir, "On the asymptotic tightness of the Shannon lower bound," IEEE Trans. Inf. Theory, vol.40, no.6, pp. 2026-2031, Nov. 1994.
21. I. Csiszár, "On the dimension and entropy of order of the mixture of probability distributions," Acta Mathematica Hungarica, vol.13, no.3-4, pp. 245-255, Sep. 1962.
22. P. Halmos, Naive Set Theory. Princeton, NJ: D. Van Nostrand, 1960.
23. K. Kuratowski, Topology. New York: Academic, 1966, vol.I.
24. G. Folland, Real Analysis: Modern Techniques and Their Applications, 2nd ed. New York: Wiley-Interscience, 1999.
25. P. Mattila, Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability. Cambridge, U.K.: Cambridge Univ. Press, 1999.
26. E. Candés, J. Romberg, and T. Tao, "Stable signal recovery from incomplete and inaccurate measurements," Commun. Pure Appl. Math., vol.59, no.8, pp. 1207-1223, Aug. 2006.
27. R. Calderbank, S. Howard, and S. Jafarpour, "Construction of a large class of deterministic matrices that satisfy a statistical isometry property," IEEE J. Sel. Topics Signal Process., vol.29, no.4, 2009.
28. W. Xu and B. Hassibi, "Compressed sensing over the Grassmann manifold: A unified analytical framework," in Proc. 46th Annu. Allerton Conf. Commun. Control Comput., 2008, pp. 562-567.
29. I. Csiszár and J. G. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1982.
30. I. Csiszár, "Generalized cutoff rates and Rényi's information measures," IEEE Trans. Inf. Theory, vol.41, no.1, pp. 26-34, Jan. 1995.
31. P. N. Chen and F. Alajaji, "Csiszár's cutoff rates for arbitrary discrete sources," IEEE Trans. Inf. Theory, vol.47, no.1, pp. 330-338, Jan. 2001.
32. H. Shimokawa, "Rényi's entropy and error exponent of source coding with countably infinite alphabet," in Proc. IEEE Int. Symp. Inf. Theory, Seattle, WA, Jul. 2006.
33. C. Chang and A. Sahai, "Universal quadratic lower bounds on source coding error exponents," in Proc. Conf. Inf. Sci. Syst., 2007.
34. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
35. M. S. Pinsker, Information and Information Stability of Random Variables and Processes. San Francisco, CA: Holden-Day, 1964.
36. K. Alligood, T. Sauer, and J. A. Yorke, Chaos: An Introduction to Dynamical Systems. New York: Springer-Verlag, 1996.
37. R. Mañé, "On the dimension of the compact invariant sets of certain non-linear maps," in Dynamical Systems and Turbulence, Warwick 1980, ser. Lecture Notes in Mathematics. Berlin, Germany: Springer-Verlag, 1981, vol.898, pp. 230-242.
38. B. R. Hunt and V. Y. Kaloshin, "Regularity of of embeddings of infinite- dimensional fractal sets into finite-dimensional spaces," Nonlinearity, vol.12, no.5, pp. 1263-1275, 1999.
39. A. Ben-Artzi, A. Eden, C. Foias, and B. Nicolaenko, "Hölder continuity for the inverse of Mañé's projection," J. Math. Anal. Appl., vol.178, pp. 22-29, 1993.
40. H. Steinhaus, "Sur les distances des points des ensembles de mesure positive," Fundamenta Mathematicae, vol.1, pp. 93-104, 1920.
41. G. J. Minty, "On the extension of Lipschitz, Lipschitz-Hölder continuous, and monotone functions," Bull. Amer. Math. Soc., vol.76, no.2, pp. 334-339, 1970.
42. H. Federer, Geometric Measure Theory. New York: Springer-Verlag, 1969.
43. D. Preiss, "Geometry of measures in : Distribution, rectifiability, and densities," Ann. Math., vol.125, no.3, pp. 537-643, 1987.
44. G. A. Edgar, Integral, Probability, and Fractal Measures. New York: Springer-Verlag, 1997.
45. M. Keane and M. Smorodinsky, "Bernoulli schemes of the same entropy are finitarily isomorphi," Ann. Math., vol.109, no.2, pp. 397-406, 1979.
46. A. Del Junco, "Finitary codes between one-sided Bernoulli shifts," Ergodic Theory Dyn. Syst., vol.1, pp. 285-301, 1981.
47. J. G. Sinai, "A weak isomorphism of transformations with an invariant measure," Dokl. Akad. Nauk SSSR., vol.147, pp. 797-800, 1962.
48. K. E. Petersen, Ergodic Theory. Cambridge, U.K.: Cambridge Univ. Press, 1990.
49. W. H. Schikhof, Ultrametric Calculus: An Introduction to p-Adic Analysis. New York: Cambridge Univ. Press, 2006.
50. M. A. Martin and P. Mattila, "dimensional regularity classifications for fractals," Trans. Amer. Math. Soc., vol.305, no.1, pp. 293-315, 1988.
51. V. Milman, "Dvoretzky's theorem: Thirty years later," Geom. Funct. Anal., vol.2, no.4, pp. 455-479, Dec. 1992.
52. W. Johnson and J. Lindenstrauss, "Extensions of Lipschitz maps into a Hilbert space," Contemp. Math., vol.26, pp. 189-206, 1984.
53. E. C. Posner and E. R. Rodemich, "Epsilon entropy and data compression," Ann. Math. Stat., vol.42, no.6, pp. 2079-2125, Dec. 1971.
54. G. Szegö, Orthogonal Polynomials. Providence, RI: AMS, 1975.
55. Y. Wu and S. Verdú, "Fundamental limits of almost lossless analog compression," in Proc. IEEE Int. Symp. Inf. Theory, Seoul, Korea, Jun. 2009.