Universal compression of memoryless sources over unknown alphabets

IEEE Transactions on Information Theory

Volumn 50, Issue 7, 2004, Pages 1469-1481

  Orlitsky, Alon ^a   Santhanam, Narayana P ^a   Zhang, Junan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Large and unknown alphabets; Patterns; Set and integer partitions; Universal compression

Indexed keywords

ASYMPTOTIC STABILITY; ENCODING (SYMBOLS); ESTIMATION; PROBABILITY; REDUNDANCY;

MEMORYLESS SOURCES; UNIVERSAL COMPRESSION; UNKNOWN ALPHABET;

DATA COMPRESSION;

EID: 3042606358     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2004.830761     Document Type: Article

References (46)

1. B. Fittingoff, "Universal methods of coding for the case of unknown statistics," in Proc. 5th Symp. Information Theory, Moscow-Gorky, U.S.S.R., 1972, pp. 129-135.
2. L. Davisson, "Universal noiseless coding," IEEE Trans. Inform. Theory, vol. IT-19, pp. 783-795, Nov. 1973.
3. J. Shtarkov, "Coding of discrete sources with unknown statistics," in Topics in Information Theory (Coll. Math. Soc. J. Bolyai, no. 16), I. Csiszár and P. Elias, Eds. Amsterdam, The Netherlands: North Holland, 1977, pp. 559-574.
4. J. Kieffer, "A unified approach to weak universal source coding," IEEE Trans. Inform. Theory, vol. IT-24, pp. 674-682, Nov. 1978.
5. J. Rissanen, "Universal coding, information, prediction, and estimation," IEEE Trans. Inform. Theory, vol. IT-30, pp. 629-636, July 1984.
6. N. Merhav and M. Feder, "Universal prediction," IEEE Trans. Inform. Theory, vol. 44, pp. 2124-2147, Oct. 1998.
7. R. Krichevsky and V. Trofimov, "The performance of universal coding," IEEE Trans. Inform. Theory, vol. IT-27, pp. 199-207, Mar. 1981.
8. T. Cover, "Universal portfolios," Math. Finance, vol. 1, no. 1, pp. 1-29, Jan. 1991.
9. Y. Shtarkov, T. Tjalkens, and F. Willems, "Multialphabet universal coding of memoryless sources," Probl. Inform. Transm., vol. 31, no. 2, pp. 114-127, 1995.
10. T. Cover and E. Ordentlich, "Universal portfolios with side information," IEEE Trans. Inform. Theory, vol. 42, pp. 348-363, Mar. 1996.
11. J. Rissanen, "Fisher information and stochastic complexity," IEEE Trans. Inform. Theory, vol. 42, pp. 40-47, Jan. 1996.
12. W. Szpankowski, "On asymptotics of certain recurrences arising in universal coding," Probl. Inform. Transm., vol. 34, no. 2, pp. 142-146, 1998.
13. Q. Xie and A. Barron, "Asymptotic minimax regret for data compression, gambling and prediction," IEEE Trans. Inform. Theory, vol. 46, pp. 431-445, Mar. 2000.
14. M. Drmota and W. Szpankowski, "The precise minimax redundancy," in Proc. IEEE Symp. Inform. Theory, Lausanne, Switzerland, June/July 2002, p. 35.
15. A. Orlitsky and N. Santhanam, "Performance of universal codes over infinite alphabets," in Proc. Data Compression Conf., Snowbird, UT, Mar. 2003, pp. 402-410.
16. P. Elias, "Universal codeword sets and representations of integers," IEEE Trans. Inform. Theory, vol. IT-21, pp. 194-203, Mar. 1975.
17. L. Györfi, I. Pali, and E. V. der Meulen, "On universal noiseless source coding for infinite source alphabets," Europ. Trans. Telecommun. and Related Technologies, vol. 4, pp. 125-132, 1993.
18. D. Foster, R. Stine, and A. Wyner, "Universal codes for finite sequences of integers drawn from a monotone distribution," IEEE Trans. Inform. Theory, vol. 48, pp. 1713-1720, June 2002.
19. T. Uyematsu and F. Kanaya, "Asymptotic optimality of two variations of Lempel-Ziv codes for sources with countably infinite alphabet," in Proc. IEEE Symp. Information Theory, Lausanne, Switzerland, June/July 2002, p. 122.
20. J. Kieffer and E. Yang, "Grammar based codes: A new class of universal lossless source codes," IEEE Trans. Inform. Theory, vol. 46, pp. 737-754, May 2000.
21. D. He and E. Yang, "On the universality of grammar-based codes for sources with countably infinite alphabets," in Proc. IEEE Symp. Information Theory, Yokohama, Japan, June/July 2003.
22. N. Jevtić, A. Orlitsky and N. Santhanam, "Universal compression of unknown alphabets," in Proc. IEEE Symp. Information Theory, Lausanne, Switzerland, 2002, p. 320.
23. J. Åberg, Y. Shtarkov, and B. Smeets, "Multialphabet coding with separate alphabet description," in Proc. Compression and Complexity of Sequences, Salerno, Italy, 1997, pp. 56-65.
24. K. Church and W. Gale, "Probability scoring for spelling correction," Statist. and Comput., vol. 1, pp. 93-103, 1991.
25. W. Gale, K. Church, and D. Yarowsky, "A method for disambiguating word senses," Comput. and Humanities, vol. 26, pp. 415-419, 1993.
26. S. Chen and J. Goodman, "An empirical study of smoothing techniques for language modeling," in Proc. 34th Annu. Meeting Association for Computational Linguistics, San Francisco, CA, 1996, pp. 310-318.
27. K. Yamanishi, "A decision-theoretic extension of stochastic complexity and its application to learning," IEEE Trans. Inform. Theory, vol. 44, pp. 1424-1439, July 1998.
28. V. Vovk, "A game of prediction with expert advice," J. Comput. Syst. Sci., vol. 56, no. 2, pp. 153-173, 1998.
29. N. Cesa-Bianchi and G. Lugosi, "Minimax regret under log loss for general classes of experts," in Proc. 12th Annu. Conf. Computational Learning Theory, Santa Cruz, CA, 1999, pp. 12-18.
30. F. Song and W. Croft, "A general language model for information retrieval (poster abstract)," in Proc. Conf. Research and Development in Information Retrieval, Berkeley, CA, 1999, pp. 279-280.
31. A. Dhulipala and A. Orlitsky, "On the redundancy of hmm patterns," in Proc. 2004 Proc. IEEE Int. Symp. Information Theory, to be published.
32. A. Orlitsky, N. Santhanam, and J. Zhang, "Always Good Turing: Asymptotically optimal probability estimation," Science, vol. 302, no. 5644, pp. 427-431, Oct. 2003.
33. E. Bell, "Exponential numbers," Amer. Math. Monthly, vol. 41, pp. 411-419, 1934.
34. G. Rota, "The number of partitions of a set," Amer. Math. Monthly, vol. 71, pp. 498-504, 1964.
35. N. Sloane. Online Encyclopedia of Integer Sequences [Online]. Available: http://www.research.att.com/~njas/sequences/
36. J. van Lint and R. Wilson, A Course in Combinatorics. Cambridge, U.K.: Cambridge Univ. Press, 1996.
37. Y. Shtarkov, "Universal sequential coding of single messages," Probl. Inform. Transm., vol. 23, no. 3, pp. 3-17, 1987.
38. W. Feller, An Introduction to Probability Theory. New York: Wiley, 1968.
39. A. Orlitsky, N. Santhanam, K. Viswanathan, and J. Zhang, "On modeling profiles instead of values," manuscript, submitted for publication.
40. G. Hardy and S. Ramanujan, "Asymptotic formulae in combinatory analysis," Proc. London Math. Soc., vol. 17, no. 2, pp. 75-115, 1918.
41. G. Hardy and E. Wright, An Introduction to the Theory of Numbers. Oxford, U.K.: Oxford Univ. Press, 1985.
42. I. Csiszár and P. Shields, "Redundancy rates for renewal and other processes," IEEE Trans. Inform. Theory, vol. 42, pp. 2065-2072, Nov. 1996.
43. N. Jevtić, A. Orlitsky, and N. Santhanan, "A Lower bound on compression of unknown alphabets," manuscript, submitted for publication.
44. A. Orlitsky and N. Santhanam, "Speaking of infinity," manuscript, submitted for publication.
45. G. Shamir and L. Song, "On the entropy of patterns of iid sequences," in Proc. 41st Annu. Allerton Conf. Communication, Control, and Computing, Allerton, IL, Oct. 2003, pp. 160-170.
46. G. Shamir, "Universal lossless compression with unknown alphabets - The average case," IEEE Trans. Inform. Theory, submitted for publication.