The interactions between ergodic theory and information theory

IEEE Transactions on Information Theory

Volumn 44, Issue 6, 1998, Pages 2079-2093

  Shields, Paul C ^a,b  

^a IEEE   (United States)

^b UNIVERSITY OF TOLEDO   (United States)

Author keywords

Data compression; Entropy; Probability; Sequential coding; Source coding; Stochastic processes

Indexed keywords

DATA COMPRESSION; ENCODING (SYMBOLS); HISTORY; PROBABILITY; RANDOM PROCESSES;

ERGODIC THEORY; SEQUENTIAL THEORY; SOURCE CODING;

INFORMATION THEORY;

EID: 0032184321     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.720532     Document Type: Article

References (77)

1. R. L. Adler, D. Coppersmith, and M. Hassner, "Algorithms for slidingblock codes-An application of symbolic dynamics to information theory," IEEE Trans. Inform. Theory, vol. IT-29, pp. 5-22, 1983.
2. P. Algoet and T. Cover, "Asymptotic optimality and asymptotic equipartition properties of log-optimum investment," Ann. Probab., vol. 16, pp. 876-898, 1988.
3. A. Barron, "The strong ergodic theorem for densities: Generalized Shannon-McMillan-Breiman theorem," Ann. Probab., vol. 13, pp. 1292-1303, 1985.
4. T. Berger, Rate Distortion Theory: A Mathematical Basis for Data Compression. Englewood Cliffs, NJ: Prentice-Hall, 1971.
5. P. Billingsley, Ergodic Theory and Information. New York: Wiley, 1965.
6. L. Breiman, "The individual ergodic theorem of information theory," Ann. Math. Statist., vol. 28, pp. 809-811, 1957; correction, vol. 31, pp. 809-810, 1960.
7. T. Cover, "A proof of the data compression theorem of Slepian and Wolf for ergodic sources," IEEE Trans. Inform. Theory, vol. IT-21, pp. 226-228, 1975.
8. I. Csisza"circ;r and J. Körner, Information Theory. Coding Theorems for Discrete Memoryless Systems. Budapest, Hungary: Akade"acute;miai Kiadö, 1981.
9. L. D. Davisson, "Universal noiseless coding," IEEE Trans. Inform. Theory, vol. IT-19, pp. 783-795, 1973.
10. A. Dembo, "Information inequalities and concentration of measure," Ann. Probab., vol. 25, pp. 927-939, 1997.
11. J. Feldman, "r-Entropy, equipartition, and Ornstein's isomorphism theorem," Israel J. Math., vol. 36, pp. 321-343, 1980.
12. R. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
13. R. Gray, "Sliding-block source coding," IEEE Trans. Inform. Theory, vol. IT-21, pp. 357-368, 1975.
14. Entropy and Information Theory. New York: Springer-Verlag, 1990.
15. R. Gray and L. D. Davisson, "Source coding without the ergodic assumption," IEEE Trans. Inform. Theory, vol. IT-20, pp. 502-516, 1975.
16. R. Gray and J. Kieffer, "Asymptotically mean stationary measures," Ann. Probab., vol. 8, pp. 962-973, 1980.
17. R. Gray, D. Neuhoff, and P. Shields, "A generalization of Ornstein's (i-distance with applications to information theory," Ann. Probab., vol. 3, pp. 315-328, 1975.
18. R. Gray and D. Ornstein, "Sliding-block joint source/noisy channel coding theorems," IEEE Trans. Inform. Theory, vol. IT-22, pp. 682-690, 1976.
19. R. Gray, D. Ornstein, and R. Dobrushin, "Block synchronization, sliding-block coding, invulnerable sources and zero error codes for discrete noisy channels," Ann. Probab., vol. 8, pp. 639-674, 1980.
20. P. Halmos, Lectures on Ergodic Theory. New York: Chelsea, 1956.
21. T. S. Han and S. Verdti, "Approximation theory of output statistics," IEEE Trans. Inform. Theory, vol. 39, pp. 752-772, 1993.
22. Y. Hershkovits and J. Ziv, "On sliding-window universal data compression with limited memory," IEEE Trans. Inform. Theory, vol. 44, pp. 66-78, 1998.
23. M. Kac, "On the notion of recurrence in discrete stochastic processes," Ann. Math. Statist., vol. 53, pp. 1002-1010, 1947.
24. S. Kakutani, "Induced measure-preserving transformations," Proc. Japan Acad., vol. 19, pp. 635-641, 1943.
25. J. C. Kieffer, "Sample converses in source coding theory," IEEE Trans. Inform. Theory, vol. 37, pp. 263-268, 1991.
26. "A direct proof that VWB processes are closed in the d-bar metric," Israel J. Math., vol. 41, pp. 154-160, 1982.
27. "An ergodic theorem for constrained sequences of functions," Bull. Amer. Math. Soc., vol. 21, pp. 249-254, 1989.
28. A. N. Kolmogorov, "A new invariant for transitive dynamical systems," Dokl. Akad. Nauk SSSR, vol. 119, pp. 861-864, 1958.
29. U. Krengel, Ergodic Theorems. Berlin, Germany: W. de Gruyter, 1985.
30. D. Lind and B. Marcus, An Introduction to Symbolic Dynamics and Coding. Cambridge, U.K.: Cambridge Univ. Press, 1995.
31. G. A. Margulis, "Probabilistic characteristics of graphs with large connectivity," Probl. Pered. Inform., vol. 10, pp. 101-108, 1974.
32. K. Marton, "A simple proof of the blowing-up lemma," IEEE Trans. Inform. Theory, voL_42, pp. 445-447, 1986.
33. "Bounding d-distance by information divergence: A method to prove concentration inequalities," Ann. Probab., vol. 24, pp. 857-866, 1966.
34. K. Marton and P. Shields, "The positive-divergence and blowing-up properties," Israel J. Math., vol. 86, pp. 331-348, 1994.
35. "Entropy and the consistent estimation of joint distributions," Ann. Probab., vol. 22, pp. 960-977, 1994; correction, vol. 24, pp. 541-545, 1996.
36. "Almost sure waiting time results for weak and very weak Bernoulli processes," Ergodic Theory and Dyn. Syst., vol. 15, pp. 951-960, 1995.
37. "How many future measures can there be?," in preparation.
38. C. McDiarmid, "On the method of bounded differences," in Surveys in Combinatorics, in London Mathematical Society Lecture Notes, vol. 141, J. Simons, Ed. Cambridge, U.K.: Cambridge Univ. Press, 1989, pp. 148-188.
39. B. McMillan, "The basic theorems of information theory," Ann. Math. Stat., vol. 24, pp. 196-216, 1953.
40. D. Neuhoff, R. Gray, and L. D. Davisson, "Fixed rate universal block source coding with a fidelity criterion," IEEE Trans. Inform. Theory, vol. IT-21, pp. 511-523, 1975.
41. D. Neuhoff and P. Shields, "Block and sliding-block source coding," IEEE Trans. Inform. Theory, vol. IT-23, pp. 211-215, 1977.
42. D. Neuhoff, P. Shields, L. D. Davisson, and F. Ledrappier, "The distortion-rate function for nonergodic sources," Ann. Probab., vol. 6, pp. 138-143, 1978.
43. D. Neuhoff and P. Shields, "Channels with almost finite memory," IEEE Trans. Inform. Theory, vol. IT-25, pp. 440447, 1979.
44. "Simplistic universal coding," IEEE Trans. Inform. Theory, vol. 44, pp. 778-781, 1998.
45. D. S. Ornstein, "Bernoulli shifts with the same entropy are isomorphic," Adv. in Math., vol. 4, pp. 337-352, 1970.
46. "Ergodic theory, randomness, and dynamical systems," in Yale Mathematical Monographs 5. New Haven, CT: Yale Univ. Press, 1974.
47. D. Ornstein, D. Rudolph, and B. Weiss, "Equivalence of measure preserving transformations," Memoirs AMS, vol. 262, 1982.
48. D. Ornstein and P. Shields, "Universal almost sure data compression," Ann. Probab.,_yol 18, pp. 441-452, 1990.
49. "The d-recognition of processes," Adv. in Math., vol. 104, pp. 182-224, 1994.
50. D. Ornstein and B. Weiss, "The Shannon-McMillan-Breiman theorem for amenable groups," Israel J. Math., vol. 44, pp. 53-60, 1983.
51. "How sampling reveals a process," Ann. Probab., vol. 18, pp. 905-930, 1990.
52. "Entropy and data compression," IEEE Trans. Inform. Theory, vol. 39, pp. 78-83, 1993.
53. K. Petersen, Ergodic Theory. Cambridge, U.K.: Cambridge Univ. Press, 1983.
54. B. Ryabko, "Twice-universal coding," Probl. Inform. Transm., vol. 20, pp. 173-178, 1984.
55. P. Shields, "The ergodic theory of discrete sample paths," AMS Graduate Studies in Mathematics, Amer. Math. Soc., 1996.
56. "Stationary coding of processes," IEEE Trans. Inform. Theory, vol. IT-25, pp. 283-291, 1979.
57. "Almost block independence," Z. für Wahr., vol. 49, pp. 119-123, 1979.
58. "Cutting and stacking. A method for constructing stationary processes," IEEE Trans. Inform. Theory, vol. 37, pp. 1605-1617, 1991.
59. "The ergodic and entropy theorems revisited," IEEE Trans. Inform. Theory, vol. IT-33, pp. 263-266, 1987.
60. "The entropy theorem via coding bounds," IEEE Trans. Inform. Theory, vol. 37, pp. 1645-1647, 1991.
61. "Entropy and prefixes," Ann. Probab., vol. 20, pp. 403-409, 1992.
62. "String matching-The general ergodic case," Ann. Probab., vol. 20, pp. 1199-1203, 1992.
63. "Two divergence-rate counterexamples," J. Theor. Prob., vol. 6, pp. 521-545, 1993.
64. "Waiting times: Positive and negative results on the Wyner-Ziv problem," J. Theor. Prob., vol. 6, pp. 499-519, 1993.
65. P. Shields and B. Weiss, "Universal redundancy rates for B-processes do not exist," IEEE Trans. Inform. Theory, vol. 41, pp. 508-512, 1995.
66. J. Sinai, "On the notion of entropy of a dynamical system," Dokl. Akad. Nauk SSSR, vol. 124, pp. 768-771, 1959.
67. M. Steele, "Kingman's subadditive ergodic theorem," Ann. Inst. Henri Poincare"acute;, vol. 25, pp. 93-98, 1989.
68. M. Talagrand, "Concentration of measure and isoperimetric inequalities in product spaces," Publ. of I.H.E.S., vol. 81, pp. 73-205, 1995.
69. "A new look at independence," Ann. Probab., vol. 24, pp. 1-34, 1996.
70. J. Moser, E. Phillips, and S. Varadhan, Ergodic Theory: A Seminar, Courant Inst. Math. Sci., New York Univ., New York, 1975.
71. L. Vasershtein, "Markov processes on countable product spaces describ-ing large systems of automata," Probl. Inform. Transm., vol. 5, pp. 64-73, 1969.
72. F. M. J. Willems, "Universal data compression and repetition times," IEEE Trans. Inform. Theory, vol. 35, pp. 54-58, 1989.
73. A. Wyner and J. Ziv, "Some asymptotic properties of the entropy of a stationary ergodic data source with applications to data compression," IEEE Trans. Inform. Theory, vol. 35, pp. 125-1258, 1989.
74. S. Xu, "An ergodic process of zero divergence-distance from the class of all stationary processes," J. Theor. Prob., vol. 11, pp. 181-196, 1997.
75. J. Ziv and A. Lempel, "A universal algorithm for sequential data compression," IEEE Trans. Inform. Theory, vol. IT-23, pp. 337-343, 1977.
76. "Compression of individual sequences via variable rate coding," IEEE Trans. Inform. Theory, vol. IT-24, pp. 530-536, 1978.
77. J. Ziv, "Coding theorems for individual sequences," IEEE Trans. Inform. Theory, vol. IT-24, pp. 40512, 1978.