A Survey of the Theory of Source Coding with a Fidelity Criterion

IEEE Transactions on Information Theory

Volumn 39, Issue 5, 1993, Pages 1473-1490

  Kieffer, John C ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

data compression; distortion measure; fidelity criterion; quantization; rate distortion theory; Source coding

Indexed keywords

CODES (SYMBOLS); DATA COMPRESSION; INFORMATION THEORY;

FIDELITY CRITERION; QUANTIZATION; SOURCE CODING;

DATA PROCESSING;

EID: 0027667901     PISSN: 00189448     EISSN: 15579654     Source Type: Journal    
DOI: 10.1109/18.259634     Document Type: Article

References (137)

1. H. Abut, Vector Quantization. New York: IEEE Press, 1990.
2. S.C. Ahalt, A.K. Krishnamurthy, P. Chen, and D.E. Melton, “Competitive learning algorithms for vector quantization,” Neural Networks, vol. 3, pp. 277–290, 1990.
3. R. Ahlswede and J. Körner, “Source coding with side information and a converse for degraded broadcast channels,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 629–637, 1975.
4. R. Ahlswede, “Coloring hypergraphs: A new approach to multi-user source coding I,” J. Comb. Inform. Syst. Sci., vol, 4, pp. 76–115, 1979.
5. ——, “Coloring hypergraphs: A new approach to multi-user source coding II,” J. Comb. Inform. Syst. Sci., vol. 5, pp. 220–268, 1980.
6. ——, “The rate distortion region for multiple descriptions without excess rate,” IEEE Trans. Inform. Theory, vol. IT-31, pp. 721–726, 1985.
7. ——, “On multiple descriptions and team guessing,” IEEE Trans. Inform. Theory, vol. IT-32, pp. 543–549, 1986.
8. A. N. Akansu and R. A. Haddad, Multiresolution Signal Decomposition. Boston, MA: Academic, 1992.
9. V. Anantharam, “A large deviations approach to error exponents in source coding and hypothesis testing,” IEEE Trans. Inform. Theory, vol. 36, pp. 938–943, 1990.
10. M. Antonini, M. Barlaud, P. Mathieu, and I. Daubechies, “Image coding using wavelet transform,” IEEE Trans. Image Processing, vol. 1, pp. 205–220, 1992.
11. F. Avram and T. Berger, “On critical distortion for Markov sources,” IEEE Trans. Inform. Theory, vol. IT-31, pp. 688–690, 1985.
12. E. Ayanoglu and R.M. Gray, “The design of joint source and channel trellis waveform coders,” IEEE Trans. Inform. Theory, vol. 33, pp. 855–865, 1987.
13. M. Barnsley, Fractals Everywhere. Boston, MA: Academic, 1988.
14. M. F. Barnsley and L. P. Hurd, Fractal Image Compression. Wellesley, MA: Peters, 1993.
15. L. A. Bassalygo and R. L. Dobrushin, “Є-entropy of the random field,” Prob. Peredach. Inform., vol. 23, pp. 3–15, 1987.
16. T. Berger, “Rate distortion theory for sources with abstract alphabet and memory,” Inform. Contr., vol. 13, 254–273, 1968.
17. ——, Rate Distortion Theory. Englewood Cliffs, NJ: Prentice-Hall, 1971.
18. ——, “Optimum quantizers and permutation codes,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 759–765, 1972.
19. ——, “Explicit bounds to R (D) for a binary symmetric Markov source,” IEEE Trans. Inform. Theory, vol. IT-23, 52–59, 1977.
20. T. Berger, K. B. Housewright, J. K. Omura, S. Tung, and J. Wolfowitz, “An upper bound on the rate distortion function for source coding with partial side information at the decoder,” IEEE Trans. Inform. Theory, vol. IT-25, 664–666, 1979.
21. T. Berger and Z. Zhang, “Minimum breakdown degradation in binary source encoding,” IEEE Trans. Inform. Theory, vol. IT-29, pp. 807–814, 1983.
22. T. Berger and R. W. Yeung, “Multiterminal source encoding with one distortion criterion,” IEEE Trans. Inform. Theory, vol. 35, pp. 228–236, 1989.
23. ——, “Multiterminal source encoding with encoder breakdown,” IEEE Trans. Inform. Theory, vol. 35, pp. 237–244, 1989.
24. T. Berger and Z. Ye, “Є-entropy and critical distortion of random fields,” IEEE Trans. Inform. Theory, vol. 36, pp. 717–725, 1990.
25. K. Birmiwal, “Block source coding theory for one-sided asymptotically mean stationary sources,” paper presented at the IEEE Int. Symp. Information Theory, Kobe, Japan, June 1988.
26. R. E. Blahut, “Computation of channel capacity and rate distortion functions,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 460–473, 1972.
27. S. Cambanis and N. L. Gerr, “A simple class of asymptotically optimal quantizers,” IEEE Trans. Inform. Theory, vol. IT-29, pp. 664–676, 1983.
28. A. E. Cetin and V. Weerackody, “Design vector quantizers using simulated annealing,” IEEE Trans. Circuits Syst., vol. 35, p. 1550, 1988.
29. J. Chatard, “Applications des proprietes de moyenne d'un groupe localement compact a la theorie ergodique,” Ann. Inst. H. Poincare, Sec. B, vol. 6, pp. 307–326, 1970.
30. P.A. Chou, T. Lookabaugh, and R.M. Gray, “Entropy-constrained vector quantization,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 31–42, 1989.
31. J.H. Conway and N.J.A. Sloane, “Fast quantizing and decoding algorithms for lattice quantizers and codes,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 227–232, 1982.
32. ——, Sphere Packings, Lattices, and Groups. New York: Springer-Verlag, 1988.
33. T. M. Cover, “A proof of the data compression theorem of Slepian and Wolf for ergodic sources,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 226–228, 1975.
34. T. M. Cover and A. El Gamal, “Achievable rates for multiple descriptions,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 851–857, 1982.
35. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
36. R. E. Chrochiere, S. A. Webber, and J. L. Flanagan, “Digital coding of speech in subbands,” Bell Syst. Tech. J., vol. 55, pp. 1069–1085, 1976.
37. I. Csiszar and G. Longo, “On the error exponent for source coding and for testing simple statistical hypotheses,” Studia Scientiarum Math. Hungarica, vol. 6, pp. 181–191, 1971.
38. I. Csiszar and J. Körner, “Towards a general theory of source networks,” IEEE Trans. Inform. Theory, vol. IT-26, pp. 155–165, 1980.
39. ——, “Graph decomposition: A new key to coding theorems,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 5–12, 1981.
40. I. Csiszar, “On the error exponent of source-channel transmission with a distortion threshold,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 823–828, 1982.
41. L. D. Davisson, “Universal noiseless coding,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 783–795, 1973.
42. L.D. Davisson, G. Longo, and A. Sgarro, “The error exponent for the noiseless encoding of finite ergodic Markov sources,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 431–438, 1981.
43. R. A. DeVore, B. Jawerth, and B. J. Lucier, “Image compression through wavelet transform coding,” IEEE Trans. Inform. Theory, vol. 38, pp. 719–746, 1992.
44. W. H. Equitz and T. M. Cover, “Successive refinement of information,” IEEE Trans. Inform. Theory, vol. 37, pp. 269–274, 1991.
45. S. Even, “On information lossless automata of finite order,” IEEE Trans. Electron. Comput., vol. EC-14, pp. 561–569, 1965.
46. K. Falconer, Fractal Geometry. Chichester, England: Wiley, 1990.
47. N. Farvardin and J. Modestino, “Optimum quantizer performance for a class of non-Gaussian memoryless sources,” IEEE Trans. Inform. Theory, vol. IT-30, pp. 485–497, 1984.
48. N. Farvardin and V. Vaishampayan, “Optimum quantizer design for noisy channels: An approach to combined source-channel coding,” IEEE Trans. Inform. Theory, vol. IT-33, pp. 827–838, 1987.
49. P. Fleischer, “Sufficient conditions for achieving minimum distortion in a quantizer,” IEEE Int. Conf. Rec., part I, vol. 12, pp. 104–111, 1964.
50. G. C. Freeland and T. S. Durrani, “IFS Fractals and the wavelet transform,” in Proc. 1990 Int. Conf. ASSP, Albuquerque, NM, 1990, pp. 2345–2348.
51. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
52. R. Garcia-Munoz and D. L. Neuhoff, “Strong universal source coding subject to a rate-distortion constraint,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 285–295, 1982.
53. M. R. Garey, D. S. Johnson, and H. S. Witsenhausen, “The complexity of the generalized Lloyd-Max problem,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 255–256, 1982.
54. A. Gersho, “Asymptotically optimal block quantization,” IEEE Trans. Inform. Theory, vol. IT-25, pp. 373–380, 1979.
55. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. Boston, MA: Kluwer, 1992.
56. H. Gish and J.N. Pierce, “Asymptotically efficient quantizing,” IEEE Trans. Inform. Theory, vol. IT-14, pp. 676–681, 1968.
57. R.M. Gray, “Information rates of autoregressive processes,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 412–421, 1970.
58. ——, “Rate-distortion functions for finite-state finite-alphabet Markov sources,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 127–134, 1971.
59. R.M. Gray and L.D. Davisson, “Source coding theorems without the ergodic assumption,” IEEE Trans. Inform. Theory, vol. IT-20, pp. 502–516, 1974.
60. R. M. Gray, D. L. Neuhoff, and P. C. Shields, “A generalization of Ornstein's d distance with applications to information theory,” Ann. Probab., vol. 3, pp. 315–328, 1975.
61. R.M. Gray and J. Kieffer, “Asymptotically mean stationary measures,” Ann. Probab., vol. 8, pp. 962–973, 1980.
62. R. M. Gray and E. D. Karnin, “Multiple local optima in vector quantizers,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 256–261, 1982.
63. R. M. Gray and F. Saadat, “Block source coding theory for asymptotically mean stationary sources,” IEEE Trans. Inform. Theory, vol. IT-30, 54–68, 1984.
64. R. M. Gray, Source Coding Theory. Boston, MA: Kluwer, 1990.
65. ——, Entropy and Information Theory. New York: Springer-Verlag, 1990.
66. M. Gutman, “On uniform quantization with various distortion measures,” IEEE Trans. Inform. Theory, vol. IT-33, pp. 169–171, 1987.
67. B. E. Hajek and T. Berger, “A decomposition theorem for binary Markov random Ann. vol. 1112–1125, 1987.
68. T. Hashimoto, “Source coding for average rate and average distortion: New variable-length coding theorems,” IEEE Trans. Inform. Theory, vol. IT-29, 785–792, 1983.
69. C. Heegard and T. Berger, “Rate-distortion when side information may be absent,” IEEE Trans. Inform. Theory, vol. IT-31, pp. 727–734, 1985.
70. D. A. Huffman, “A method for the construction of minimum redundancy Proc. IRE, vol. 40, 1098–1101, 1952.
71. A. E. Jacquin, “Image coding based on a fractal theory of iterated contractive image transformations,” IEEE Trans. Image Processing, vol. 1, 18–30, 1992.
72. N. S. Jayant and P. Noll, Digital Coding of Waveforms. Englewood Cliffs, NJ: Prentice-Hall, 1984.
73. A. H. Kaspi, “Two-way source coding with a fidelity criterion,” IEEE Trans. Inform. Theory, vol. IT-31, pp. 735–740, 1985.
74. A. H. Kaspi and T. Berger, “Rate-distortion for correlated sources and partially separated encoders,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 828–840, 1982.
75. A. Khinchin, “The entropy concept in probability theory,” Usp. Mat. Nauk, vol. 8, 3–20, 1953.
76. J.C. Kieffer, “A unified approach to weak universal source coding,” IEEE Trans. Inform. Theory, vol. 24, pp. 674–682, 1978.
77. ——, “On the minimum rate for strong universal block coding of a class of ergodic sources,” IEEE Trans. Inform. Theory, vol. IT-26, pp. 693–702, 1980.
78. ——, “A method for proving multiterminal source coding theorems,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 565–570, 1981.
79. ——, “Uniqueness of locally optimal quantizer for log-concave density and convex error weighting function,” IEEE Trans. Inform. Theory, vol. IT-29, pp. 42–47, 1983.
80. ——, “Fixed-rate encoding of nonstationary information sources,” IEEE Trans. Inform. Theory, vol. IT-33, pp. 651–655, 1987.
81. J.C. Kieffer, T. M. Jahns, and V. A. Obuljen, “New results on optimal entropy-constrained quantization,” IEEE Trans. Inform. Theory, vol. 34, pp. 1250–1258, 1988.
82. J.C. Kieffer, “Finite-state adaptive block to variable-length noiseless coding of a nonstationary information source,” IEEE Trans. Inform. Theory, vol. 35, pp. 1259–1263, 1989.
83. ——, “Sample converses in source coding theory,” IEEE Trans. Inform. Theory, vol. 37, pp. 263–268, 1991.
84. ——, “Strong converses in source coding relative to a fidelity criterion,” IEEE Trans. Inform. Theory, vol. 37, pp. 257–262, 1991.
85. A. N. Kolmogorov, “Three approaches to the quantitative definition of information,” Probl. Inform. Transm., vol. 1, pp. 1–7, 1965.
86. V. N. Koshelev, “Hierarchical coding of discrete sources,” Probl. Pered. vol. 31–49, 1980.
87. R.E. Krichevsky and V.K. Trofimov, “The performance of universal encoding,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 199–207, 1981.
88. A. Lempel and J. Ziv, “On the complexity of finite sequences,” IEEE Trans. Inform. Theory, vol. IT-22, pp. 75–81, 1976.
89. A. Leon-Garcia, L. D. Davisson, and D. L. Neuhoff, “New results on coding of stationary nonergodic sources,” IEEE Trans. Inform. Theory, vol. IT-25, pp. 137–144, 1979.
90. Y. Linde, A. Buzo, and R. M. Gray, “An algorithm for vector quantizer design,” IEEE Trans. Commun., vol. COM-28, pp. 84–95, 1980.
91. S. P. Lloyd, “Least squares quantization in PCM,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 129–137, 1982.
92. G. Longo and A. Sgarro, “The source coding theorem revisited: A combinatorial approach,” IEEE Trans. Inform. Theory, vol. IT-25, pp. 544–548, 1979.
93. K. M. Mackenthun and M. B. Pursley, “Strongly and weakly universal source coding,” in Proc. 1977 Conf. Information Science and Systems, Johns Hopkins Univ., 1977, pp. 286–291.
94. ——, “Variable-rate block source coding subject to a fidelity constraint,” IEEE Trans. Inform. Theory, vol. IT-24, pp. 349–360, 1978.
95. M.W. Marcellin and T. R. Fischer, “Trellis coded quantization of memoryless and Gauss-Markov sources,” IEEE Trans. Commun., vol. 38, pp.82-93, 1990.
96. K. Marton, “Error exponent for source coding with a fidelity criterion,” IEEE Trans. Theory, vol. IT-20, 197–199, 1974.
97. J. Max, “Quantizing for minimum distortion,” IRE Trans. Inform. Theory, vol. IT-6, pp. 7–12, 1960.
98. S. W. McLaughlin, “Source-channel coding with applications to digital magnetic recording,” Ph.D. dissertation, Univ. Michigan, 1992.
99. N. Merhav and D. L. Neuhoff, “Variable-to-fixed length codes provide better large deviations performance than fixed-to-variable length codes,” IEEE Trans. Inform. Theory, vol. 38, pp. 135–140, 1992.
100. S. Natarajan, “Large deviations, hypothesis testing, and source coding for finite Markov chains,” IEEE Trans. Inform. Theory, vol. IT-31, pp. 360–365, 1985.
101. D.L. Neuhoff, R. M. Gray, and L. D. Davisson, “Fixed rate universal block source coding with a fidelity criterion,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 511–523, 1975.
102. D. L. Neuhoff and P. C. Shields, “Fixed-rate universal codes for Markov sources,” IEEE Trans. Inform. Theory, vol. IT-24, pp. 360–367, 1978.
103. D. L. Neuhoff and R. K. Gilbert, “Causal source codes,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 701–713, 1982.
104. C. Newman, “Decomposition of binary random fields and zeroes of partition functions,” Ann. Probab., vol. 15, pp. 1126–1130, 1987.
105. D. S. Omstein and P. C. Shields, “Universal almost sure data compression,” Ann. Probab., vol. 18, pp. 441–452, 1990.
106. L.H. Ozarow, “On the source coding problem with two channels and three receivers,” Bell Syst. Tech. J., vol. 59, pp. 1909–1922, 1980.
107. P. F. Panter and W. Dite, “Quantization distortion in pulse-count modulation with nonuniform spacing of levels,” Proc. IRE, vol. 39, pp. 44–48, 1951.
108. K. Parthasarathy, “Effective entropy rate and transmission of information through channels with additive random noise,” Sankhya, Ser. A, vol. 25, pp. 75–84, 1963.
109. J. Rissanen, “Universal coding, information, prediction, and estimation,” IEEE Trans. Inform. Theory, vol. IT-30, pp. 629–636, 1984.
110. J. Rissanen, Stochastic Complexity in Statistical Inquiry. Singapore: World Scientific, 1989.
111. ——, “Reflections of the Associate Editor for data compression,” IEEE Inform. Theory Soc. Newslett., vol. 42, no. 1, p. 5, 1992.
112. K. Rose, E. Gurewitz, and G.C. Fox, “Vector quantization by deterministic annealng,” IEEE Trans. Inform. Theory, vol. 38, pp. 1249–1257, 1992.
113. C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 1948.
114. ——, “Coding theorems for a discrete source with a fidelity criterion,” IRE Nat. Conv. Rec., part 4, pp. 142–163, 1959.
115. P.C. Shields, D.L. Neuhoff, L.D. Davisson, and F. Ledrappier, “The distortion-rate function for non-ergodic sources,” Ann. Probab., vol. 6, pp. 138–143, 1978.
116. D. Slepian and J. K. Wolf, “Noiseless coding of correlated information sources,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 471–480, 1973.
117. A. Trushkin, “Monotony of Lloyd's Method II for log-concave density and convex error weighting function,” IEEE Trans. Inform. Theory, vol. IT-30, pp. 380–383, 1984.
118. J. Vaisey and A. Gersho, “Simulated annealing and codebook design,” in Proc. IEEE Int. Conf. ASSP, 1988, pp. 1176–1179.
119. K. Vasek, “On the error exponent for ergodic Markov source,” Kybernetika, vol. 16, pp. 318–329, 1980.
120. M.J. Weinberger, A. Lempel, and J. Ziv, “On the coding delay of a general coder,” Image and Text Compression, J. A. Storer, Ed. Boston, MA: Kluwer, 1992, pp. 216–252.
121. H.S. Witsenhausen, “On source networks with minimum breakdown degradation,” Bell Syst. Tech. J., vol. 59, pp. 1083–1087, 1980.
122. H.S. Witsenhausen and A.D. Wyner, “Source coding for multiple descriptions II: A binary source,” Bell Syst. Tech. J., vol. 60, pp. 2281–2292, 1981.
123. J.K. Wolf, A. D. Wyner, and J. Ziv, “Source coding for multiple descriptions,” Bell Syst. Tech. J., vol. 59, pp. 1417–1426, 1980.
124. J. W. Woods and S. D. O'Neil, “Subband coding of images,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp. 1278–1280, 1986.
125. X. Wu and K. Zhang, “A better tree-structured vector quantizer,” in Proc. 1991 Data Compression Conf, J. Storer and J. Reif, Eds. Los Alamitos, CA: IEEE Computer Society Press, 1991, pp. 392–401.
126. A.D. Wyner, “On source coding with side information at the decoder,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 294–300, 1975.
127. A. D. Wyner and J. Ziv, “The rate-distortion function for source coding with side information at the decoder,” IEEE Trans. Inform. Theory, vol. IT-22, pp. 1–10, 1976.
128. P. L. Zador, “Asymptotic quantization error of continuous signals and the quantization dimension,” IEEE Trans. Inform. Theory, vol. IT-28, pp. 139–149, 1982.
129. R. Zamir and M. Feder, “On universal quantization by randomized uniform/lattice quantizers,” IEEE Trans. Inform. Theory, vol. 38, pp. 428–436, 1992.
130. K. Zeger and A. Gersho, “Stochastic relaxation algorithm for improved codebook design,” Electron. Lett., vol. 25, pp. 896–898, 1989.
131. K. Zeger, J. Vaisey, and A. Gersho, “Globally optimal vector quantizer design by stochastic relaxation,” IEEE Trans. Signal Processing, vol. 40, pp. 310–322, 1992.
132. Z. Zhang and T. Berger, “New results in binary multiple descriptions,” IEEE Trans. Inform. Theory, vol. IT-33, pp. 502–521, 1987.
133. J. Ziv, “Coding of sources with unknown statistics—Part II: Distortion relative to a fidelity criterion,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 389–394, 1972.
134. J. Ziv and A. Lempel, “A universal algorithm for sequential data compression,” IEEE Trans. Inform. Theory, vol. IT-33, pp. 337–343, 1977.
135. ——, “Compression of individual sequences via variable-rate coding,” IEEE Trans. Inform. Theory, vol. IT-24, pp. 530–536, 1978.
136. J. Ziv, “Distortion-rate theory for individual sequences,” IEEE Trans. Inform. Theory, vol. IT-26, pp. 137–143, 1980.
137. ——, “On universal quantization,” IEEE Trans. Inform. Theory, vol. IT-31, pp. 344–347, 1985.