Universal entropy estimation via block sorting

IEEE Transactions on Information Theory

Volumn 50, Issue 7, 2004, Pages 1551-1561

  Cai, Haixiao ^a   Kulkarni, Sanjeev R ^a   Verdú, Sergio ^a  

^a Princeton University   (United States)

Author keywords

Block sorting; Burrows wheelers transform (BWT); Entropy estimation; Piecewise stationary memoryless source; Tree source

Indexed keywords

ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; ENTROPY; ESTIMATION; PROBABILITY; TREES (MATHEMATICS);

BLOCK SORTING; UNIVERSAL ENTROPY ESTIMATION;

INFORMATION THEORY;

EID: 3042515346     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2004.830771     Document Type: Letter

References (39)

1. A. Anto and I. Kontoiannis, "Convergence properties of functional estimates of discrete distributions," Random Structures and Algorithms, vol. 19, pp. 163-193, Oct. 2002.
2. D. Baron and Y. Bresler, "Linear complexity MDL universal coding with the BWT," presented at the IEEE International Symposium on Information Theory, Washington, DC, June 2001.
3. A. R. Barron, "Logically smooth density estimation," Ph.D. dissertation, Stanford Univ., Stanford, CA, 1985.
4. P. Billingsley, "Statistical methods in Markov chains," Ann. Math. Statist., vol. 32, no. 1, pp. 12-40, Mar. 1961.
5. M. Burrows and D. J. Wheeler, "A block-sorting lossless data compression algorithm," Digital Systems Res. Ctr., Palo Alto, CA, Tech. Rep. SRC 124, May 1994.
6. J. G. Cleary and I. H. Witten, "Data compression using adaptive coding and partial string matching," IEEE Trans. Commun. Technol., vol. COM-32, pp. 396-402, Apr. 1984.
7. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
8. M. Effros, K. Visweswariah, S. R. Kulkarni, and S. Verdú, "Universal lossless source coding with the Burrows Wheeler transform," IEEE Trans. Inform. Theory, vol. 48, pp. 1061-1081, May 2002.
9. Y. Gao, I. Kontoiannis, and E. Bienenstock, "Estimating the entropy rate of spike trains," preprint, 2004.
10. M. Gutman, "Asymptotically optimal classification for multiple tests with empirically observed statistics," IEEE Trans. Inform. Theory, vol. 35, pp. 401-408, Mar. 1989.
11. A. O. Hero and O. Michel, "Asymptotic theory of greedy approximations to minimal k-point random graphs," IEEE Trans. Inform. Theory, vol. 45, pp. 1921-1938, Sept. 1999.
12. A. O. Hero, B. Ma, O. Michel, and J. Gorman, "Applications of entropic spanning graphs," IEEE Signal Processing Mag., vol. 19, pp. 85-95. Sept. 2002.
13. J. C. Kieffer, "Sample converses in source coding theory," IEEE Trans. Inform. Theory, vol. 37, pp. 263-268 Mar. 1991.
14. J. C. Kieffer and E.-h. Yang, "Grammar based codes: A new class of universal lossless source codes," IEEE Trans. Inform. Theory, vol. 46, pp. 737-754, May 2000.
15. I. Kontoyiannis, P. H. Algoet, Y. M. Suhov, and A. J. Wyner, "Nonparametric entropy estimation for stationary processes and random fields, with applications to english text," IEEE Trans. Inform. Theory, vol. 44, pp. 1319-1327, May 1998.
16. J. Lanctot, M. Li, and E.-H. Yang, "Estimating DNA sequence entropy," in Proc. Symp. Discrete Algorithms, San Francisco, CA, Jan. 2000, pp. 409-418.
17. A. Martin, G. Seroussi, and M. Weinberger, "Linear time universal coding and time reversal of tree sources via FSM closure," IEEE Trans. Inform. Theory, vol. 50, pp, 1442-1468, July 2004.
18. E. M. Mccreight, "A space-economical suffix tree construction algorithm," J. Assoc. Comput. Mach., vol. 23, pp. 262-272, 1976.
19. I. Nemenman, W. Bialek, and R. Steveninck, "Entropy and information in neural spike trains: Progress on the sampling problem," Phys. Rev. E, to be published.
20. D. S. Ornstein and B. Weiss, "Entropy and data compression schemes," IEEE Trans. Inform. Theory, vol. 39, pp. 78-83, Jan. 1993.
21. L. Paninski, "Estimation of entropy and mutual information," Neural Comput., vol. 15, pp. 1191-11253, 2003.
22. A. N. Quas, "An entropy estimation for a class of infinite alphabet processes," Theory Probab. Appl., vol. 43, pp. 496-507, 1995.
23. A. O. Schmitt and H. Herzel, "Estimating the entropy of DNA sequences," J. Theor. Biol., vol. 188, no. 3, pp. 369-377, 1997.
24. J. Seward, "On the performance of BWT sorting algorithms," in Proc. Data Compression Conf., Snowbird, UT, Mar. 2000, p. 173.
25. G. I. Shamir and D. J. Costello, Jr, "Asymptotically optimal low-complexity sequential lossless coding for piecewise-stationary memoryless sources - Part I: The regular case," IEEE Trans. Inform. Theory, vol. 46, pp. 2444-2467, Nov. 2000.
26. C. E. Shannon, "Prediction and entropy of printed english," Bell Syst. Tech. J., pp. 50-64, 1951.
27. P. C. Shields, "Entropy and prefixes," Ann. Probab., vol. 20, no. 1, pp. 403-409, Jan. 1992.
28. ____, "The ergodic theory of discrete sample paths," in Graduate Studies in Mathematics Providence, RI, 1996, vol. 13.
29. S. P. Strong, R. Koberle, R. Steveninck, and W. Bialek, "Entropy and information in neural spike trains," Phys. Rev. Lett., vol. 80, pp. 197-200, 1998.
30. W. Szpankowski, "Asymptotic properties of data compression and suffix trees," IEEE Trans. Inform. Theory, vol. 39, pp. 1647-1659, Sept. 1993.
31. K. Viswesariah, S. R. Kulkarni, and S. Verdú, "Output distribution of the Burrows-Wheeler transform," in Proc. IEEE Int. Symp. Information Theory, Sorrento, Italy, June 2000, p. 53.
32. M. J. Weinberger, A. Lempel, and J. Ziv, "A sequential algorithm for the universal coding of finite memory sources," IEEE Trans. Inform. Theory, vol. 38, 1002-1014, May 1992.
33. M. J. Weinberger, J. Rissanen, and M. Feder, "A universal finite memory source," IEEE Trans. Inform. Theory, vol. 41, pp. 643-652, May 1995.
34. F. M. J. Willems, Y. M. Shtarkov, and T. J. Tjalkens, "The context tree weighting method: Basic properties," IEEE Trans. Inform. Theory, vol. 41, pp. 653-664 May 1995.
35. A. D. Wyner and J. Ziv, "Some asymptotic properties of the entropy of a stationary ergodic data data source with applications to data compression," IEEE Trans. Inform. Theory, vol. 35, pp. 1250-1258, Nov. 1989.
36. A. J. Wyner and D. Foster, "On the lower limits of entropy estimation," IEEE Trans. Inform. Theory, submitted for publication.
37. E.-h. Yang and J. C. Kieffer, "Efficient universal lossless data compression algorithms based on a greedy sequential grammar transform - Part one: Without context models," IEEE Trans. Inform. Theory, vol. 46, pp. 755-788, May 2000.
38. E.-H. Yang and S. Shen, "Chaitin complexity, Shannon information content of a single event and infinite random sequences (i)," Science in China, ser. A, vol. 34, pp. 1183-1193, 1991.
39. J. Ziv, "On classification with empirically observed statistics and universal data compression," IEEE Trans. Inform. Theory, vol. 34, pp. 278-286, Mar. 1988.