On universal types

IEEE Transactions on Information Theory

Volumn 52, Issue 1, 2006, Pages 171-189

  Seroussi, Gadiel ^a,b  

^a HEWLETT PACKARD LABORATORIES   (United States)

^b MATHEMATICAL SCIENCES RESEARCH INSTITUTE   (United States)

Author keywords

Lempel Ziv coding; Method of types; Random process simulation; Type classes; Universal simulation; Universal types

Indexed keywords

CODES (SYMBOLS); DATA PROCESSING; ESTIMATION; PROBABILITY DISTRIBUTIONS; RANDOM PROCESSES;

LEMPEL-ZIV CODING; NORMALIZED LOGARITHMS; RANDOM PROCESS SIMULATION; TYPE CLASSES; UNIVERSAL SIMULATION; UNIVERSAL TYPES;

ENCODING (SYMBOLS);

EID: 29844446731     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.860437     Document Type: Article

References (56)

1. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
2. I. Csiszár, "The method of types," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2505-2523, Oct. 1998.
3. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
4. E. L. Lehmann, Theory of Point Estimation. New York: Wiley, 1983.
5. N. Merhav and M. J. Weinberger, "On universal simulation of information sources using training data," IEEE Trans. Inf. Theory, vol. 50, no. 1, pp. 5-20, Jan. 2004.
6. J. Ziv and A. Lempel, "Compression of individual sequences via variable-rate coding," IEEE Trans. Inf. Theory, vol. IT-24, no. 5, pp. 530-536, Sep. 1978.
7. J. Rissanen, "A universal data compression system," IEEE Trans. Inf. Theory, vol. IT-29, no. 5, pp. 656-664, Sep. 1983.
8. P. Whittle, "Some distribution and moment formulae for the Markov chain," J. Roy. Statist. Soc., Ser. B, vol. 17, pp. 235-242, 1955.
9. L. D. Davisson, G. Longo, and A. Sgarro, "The error exponent for the noiseless encoding of finite ergodic Markov sources," IEEE Trans. Inf. Theory, vol. IT-27, no. 4, pp. 431-438, Jul. 1981.
10. P. Jacquet and W. Szpankowski, "Markov types and minimax redundancy for Markov sources," IEEE Trans. Inf. Theory, vol. 50, no. 7, pp. 1393-1402, Jul. 2004.
11. M. J. Weinberger, N. Merhav, and M. Feder, "Optimal sequential probability assignment for individual sequences," IEEE Trans. Inf. Theory, vol. 40, no. 2, pp. 384-396, Mar. 1994.
12. T. M. Cover, "Enumerative source encoding," IEEE Trans. Inf. Theory, vol. IT-19, no. 1, pp. 73-77, Jan. 1973.
13. D. E. Knuth and A. Yao, "The complexity of nonuniform random number generation," in Algorithms and Complexity, New Directions and Results, J. F. Traub, Ed. New York: Acadenic, 1976, pp. 357-428.
14. T. S. Han and S. Verdú, "Approximation theory of output statistics," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 752-772, May 1993.
15. Y. Steinberg and S. Verdú, "Simulation of random processes and ratedistortion theory," IEEE Trans. Inf. Theory, vol. 42, no. 1, pp. 63-86, Jan. 1996.
16. T. S. Han and M. Hoshi, "Interval algorithm for random number generation," IEEE Trans. Inf. Theory, vol. 43, no. 2, pp. 599-611, Mar. 1997.
17. I. Csiszár and P. C. Shields, "Redundancy rates for renewal and other processes," IEEE Trans. Inf. Theory, vol. 42, no. 6, pp. 2065-2072, Nov. 1996.
18. N. Merhav, "Universal coding with minimum probability of codeword length overflow," IEEE Trans. Inf. Theory, vol. 37, no. 3, pp. 556-563, May 1991.
19. J. Ziv and N. Merhav, "Estimating the number of states of a finite-state source," IEEE Trans. Inf. Theory, vol. 38, no. 1, pp. 61-65, Jan. 1992.
20. D. E. Knuth, The Art of Computer Programming. Fundamental Algorithms, 3rd ed. Reading, MA: Addison-Wesley, 1997, vol. 1.
21. G. Seroussi, "On the Number of t-Ary Trees with a Given Path Length," Hewlett-Packard Labs., Tech. Rep. HPL-2004-127, July 2004. Available Online at http://arxiv.org/abs/cs.DM/509046. To be published in Algorithmica.
22. M. Feder, N. Merhav, and M. Gutman, "Universal prediction of individual sequences," IEEE Trans. Inf. Theory, vol. 38, pp. 1258-1270, Jul. 1992.
23. D. E. Knuth, The Art of Computer Programming. Sorting and Searching. Reading, MA: Addison-Wesley, 1998, vol. 3.
24. G. H. Gonnet and J. I. Munro, "Heaps on heaps," SIAM J. Comput., vol. 15, no. 4, pp. 964-971, Nov. 1986.
25. H.-K. Hwang and J.-M. Steyaert, "On the number of heaps and the cost of heap construction," in Mathematics and Computer Science, II, Trends in Mathematics. Basel, Switzerland: Birkhäuser, 2002, pp. 295-310.
26. D. G. Champerknowne, "The construction of decimals normal in the scale of ten," J. London Math. Soc., vol. 8, pp. 254-260, 1933.
27. N. J. A. Sloane. The On-Line Encyclopedia of Integer Sequences. Online. Available: http://www.research.att.com/njas/sequences/
28. F. J. MacWilliams and N. J. A. Sloane, The Theory of Error Correcting Codes. Amsterdam, The Netherlands: North-Holland, 1983.
29. F. Willems, Y. Shtarkov, and T. Tjalkens, "The context-tree weighting method: Basic properties," IEEE Trans. Inf. Theory, vol. 41, no. 3, pp. 653-664, May 1995.
30. C. Knessl and W. Szpankowski, "Enumeration of binary trees, Lempel-Ziv'78 parsings, and universal types," in Proc. 2nd Workshop on Analytic Algorithmics and Combinatorics (ANALCO05), Vancouver, BC, Canada, Jan. 2005, pp. 222-229.
31. G. Louchard and W. Szpankowski, "On the average redundancy rate of the Lempel-Ziv code," IEEE Trans. Inf. Theory, vol. 43, no. 1, pp. 2-8, Jan. 1997.
32. S. A. Savari, "Redundancy of the Lempel-Ziv incremental parsing rule," IEEE Trans. Inf. Theory, vol. 43, no. 1, pp. 9-21, Jan. 1997.
33. D. H. Lehmer, "The machine tools of combinatorics," in Applied Combinatorial Mathematics, E. F. Beckenbach, Ed. New York:WIley, 1964, pp. 5-30.
34. D. E. Knuth, The Art of Computer Programming. Seminumerical Algorithms, 3rd ed. Reading, MA: Addison-Wesley, 1998, vol. 2.
35. F. Jelinek, Probabilistic Information Theory. New York: Mc-Graw-Hill, 1968.
36. J. Rissanen, "Generalized Kraft inequality and arithmetic coding," IBM J. Res. Develop., vol. 20, pp. 197-300, May 1976.
37. R. Pasco, "Source coding algorithms for fast data compression," Ph.D. dissertation, Stanford Univ., Stanford, CA, 1976.
38. J. Rissanen and G. G. Langdon, "Universal modeling and coding," IEEE Trans. Inf. Theory, vol. IT-27, no. 1, pp. 12-23, Jan. 1981.
39. M. J.Weinberger, A. Lempel, and J. Ziv, "A sequential algorithm for the universal coding of finite memory sources," IEEE Trans. Inf. Theory, vol. 38, no. 3, pp. 1002-1014, May 1992.
40. N. G. de Bruijn, "A combinatorial problem," Proc. Koninklijke Nederlands Akademie van Wetenschappen, pt. 2, vol. 49, pp. 758-764, 1946.
41. S. W. Golomb, Shift Register Sequences. San Francisco, CA: Holden-Day, 1967.
42. J. M. Hall, Combinatorial Theory, 2nd ed. New York: Wiley, 1986.
43. P. C. Shields, "Performance of LZ algorithms on individual sequences," IEEE Trans. Inf. Theory, vol. 45, no. 4, pp. 1283-1288, May 1999.
44. L. A. Pierce II and P. Shields, "Sequences incompressible by SLZ (LZW), yet fully compressible by ULZ," in Numbers, Information and Complexity, I. Althöfer, N. Cai, G. Dueck, L. Khachatrian, M. Pinsker, A. Sarközy, I. Wegener, and Z. Zhang, Eds. Norwell, MA: Kluwer, 2000, pp. 385-390.
45. J. Ziv and A. Lempel, "A universal algorithm for sequential data compression," IEEE Trans. Inf. Theory, vol. IT-23, no. 3, pp. 337-343, May 1977.
46. E. Plotnik, M. J. Weinberger, and J. Ziv, "Upper bounds on the probability of sequences emitted by finite-state sources and on the redundancy of the Lempel-Ziv algorithm," IEEE Trans. Inf. Theory, vol. 38, no. 1, pp. 66-72, Jan. 1992.
47. A. R. Barron, "Logically smooth density estimation," Ph.D. dissertation, Stanford Univ., Stanford, CA, 1985.
48. J. C. Kieffer, "Sample converses in source coding theory," IEEE Trans. Inf. Theory, vol. 37, no. 2, pp. 263-268, Mar. 1991.
49. A. Lempel and J. Ziv, "Compression of two-dimensional data," IEEE Trans. Inf. Theory, vol. IT-32, no. 1, pp. 2-8, Jan. 1986.
50. P. Gács, J. T. Tromp, and P. M. B. Vitányi, "Algorithmic statistics," IEEE Trans. Inf. Theory, vol. 47, no. 6, pp. 2443-2463, Sep. 2001.
51. T. Weissman, E. Ordentlich, G. Seroussi, S. Verdú, and M. Weinberger, "Universal discrete denoising: Known channel," IEEE Trans. Inf. Theory, vol. 51, no. 1, pp. 5-28, Jan. 2005.
52. B. Carpentieri, M. J. Weinberger, and G. Seroussi, "Lossless compression of continuous-tone images," in Proc. IEEE, vol. 88, Nov. 2000, pp. 1797-1809.
53. G. Motta, E. Ordentlich, I. Ramírez, G. Seroussi, and M. Weinberger, "The DUDE framework for continuous tone image denoising," in Proc. IEEE Int. Conf. Image Processing, Genoa, Italy, Sep. 2005, pp. 345-348.
54. J. Rissanen, "Universal coding, information, prediction, and estimation," IEEE Trans. Inf. Theory, vol. IT-30, no. 4, pp. 629-636, Jul. 1984.
55. J. Portilla and E. P. Simoncelli, "A parametric texture model based on joint statistics of complex wavelet coefficients," Int. J. Computer Vision, vol. 40, no. 1, pp. 49-71, Oct. 2000.
56. G. Brown, G. Sapiro, and G. Seroussi, "Texture mixing via universal simulation," in Proc. 4th Int. Workshop on Texture Analysis and Synthesis (Texture 2005), Beijing, China, Oct. 2005.