Nonparametric entropy estimation for stationary processesand random fields, with applications to english text

IEEE Transactions on Information Theory

Volumn 44, Issue 3, 1998, Pages 1319-1327

  Kontoyiannis, I ^b   Algoet, P H ^b   Suhov, Yu M ^c,d   Wyner, A J ^a,e  

^a IEEE   (United States)

^b STANFORD UNIVERSITY   (United States)

^c UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^d INSTITUTE FOR INFORMATION TRANSMISSION PROBLEMS   (Russian Federation)

^e UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Entropy of english; Entropy rate; Pattern matching; Universal data compression

Indexed keywords

ALGORITHMS; DATA COMPRESSION; ENCODING (SYMBOLS); ESTIMATION; MATHEMATICAL TRANSFORMATIONS; PATTERN RECOGNITION; PROBABILITY; RANDOM PROCESSES;

ENGLISH TEXT; ENTROPY RATE; STATIONARY ERGODIC PROCESS;

INFORMATION THEORY;

EID: 0032073870     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.669425     Document Type: Article

References (33)

1. P. H. Algoet, "The strong law of large numbers for sequential decisions under uncertainty," IEEE Trans. Inform. Theory, vol. 40, pp. 609-634, May 1994.
2. A. R. Barron, "The strong ergodic theorem for densities: Generalized Shannon-McMillan-Breiman theorem," Ann. Probab., vol. 13, pp. 1292-1303, Nov. 1985.
3. L. Breiman, "The individual ergodic theorem of information theory," Ann. Math. Statist., vol. 28, pp. 809-811, 1957. Correction: ibid., vol. 31, pp. 809-810, 1960.
4. S. Chen and J. H. Reif, "Using difficulty of prediction to decrease computation: Fast sort, priority queue and convex hull on entropy bounded inputs," in Proc. 34th Symp. Foundations of Computer Science. Los Alamitos, CA: IEEE Computer Soc. Press, 1993, pp. 104-112.
5. _, "Fast pattern matching for entropy bounded text," in Proc. DCC'95 Data Compression Conf. (Snowbird, UT). Los Alamitos, CA: IEEE Computer Soc. Press, 1995, pp. 282-291.
6. T. M. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
7. W. Doeblin, "Sur le propriétés asymptotiques de mouvement régis par certains types de chaines simples," Bull. Math. Soc. Raum. Sci., vol. 39, no. 1 pp. 57-115,
8. and Bull. Math. Soc. Raum. Sci., vol. 39, no. 2 pp. 3-61, 1937.
9. M. Farach, M. Noordewier, S. Savari, L. Shepp, A. J. Wyner, and J. Ziv, "On the entropy of DNA: Algorithms and measurements based on memory and rapid convergence," in Proc. ACM-SIAM Symp. Discrete Algorithms. Philadelphia, PA: Soc. Industr. Appl. Math., 1995, pp. 48-57.
10. P. Grassberger, "Estimating the information content of symbol sequences and efficient codes," IEEE Trans. Inform. Theory, vol. 35, pp. 669-675, May 1989.
11. P. Juola, "What can we do with small corpora: Document categorization via cross-entropy," Comparative Linguistics, to be published in SimCat'97.
12. I. Kontoyiannis and Yu. M. Suhov, "Prefixes and the entropy rate for long-range sources," in Probability Statistics and Optimization, F. P. Kelly, Ed. Chichester, U.K.: Wiley, 1994, pp. 89-98.
13. _, "Stationary entropy estimation via string matching," in Proc. Data Compression Conf. DCC'96 (Snowbird, UT, Apr. 1996).
14. U. Krengel, Ergodic Theorems. Berlin, Germany: De Gruyter, 1985.
15. P. T. Maker, "The ergodic theorem for a sequence of functions," Duke Math. J., vol. 6, pp. 27-30, 1940.
16. D. S. Ornstein and B. Weiss, "Entropy and data compression schemes," IEEE Trans. Inform. Theory, vol. 39, pp. 78-83, Jan. 1993.
17. _, "Entropy and recurrence rates for stationary random fields," preprint;
18. also presented at the IEEE Int. Symp. on Information Theory, Trondheim, Norway, June 27-July 1, 1994.
19. _, "The Shannon-McMillan-Breiman theorem for countable partitions," unpublished manuscript.
20. M. S. Pinsker, Information and Information Stability of Random Variables and Processes. Moscow, USSR: Izd. Akad. Nauk SSSR. (Translated and edited by A. Feinstein, San Francisco: Holden-Day, 1964).
21. B. Pittel, "Asymptotic growth of a class of random trees," Ann. Probab., vol. 13, no. 2, pp. 414-427, 1985.
22. A. N. Quas, "An entropy estimator for a class of infinite alphabet processes," Statistical Lab., Univ. of Cambridge, Cambridge, U.K., Tech. Rep. 95-3, May 1995.
23. P. C. Shields, "Universal redundancy rates do not exist," IEEE Trans. Inform. Theory, vol. 39, pp. 520-524, Mar. 1993.
24. P. C. Shields, "Entropy and prefixes," Ann. Probab., vol. 20, pp. 403-409, 1992.
25. P. C. Shields and B. Weiss, "Universal redundancy rates for the class of B-processes do not exist," IEEE Trans. Inform. Theory, vol. 41, pp. 508-512, Mar. 1995.
26. W. Szpankowski, "Asymptotic properties of data compression and suffix trees," IEEE Trans. Inform. Theory, vol. 39, pp. 1647-1659, Sept. 1993.
27. W. J. Teahan and J. G. Cleary, "Models of English text," in Proc. DCC'97 Data Compression Conf. (Snowbird, UT). Los Alamitos, CA: IEEE Computer Soc. Press, 1997, pp. 12-21.
28. W. J. Teahan and J. G. Cleary, "The entropy of English using PPM-based models," in Proc. DCC'96 Data Compression Conf. (Snowbird, UT). Los Alamitos, CA: IEEE Computer Soc. Press, 1996, pp. 53-62.
29. A. J. Wyner, "Entropy and patterns," in Proc. IEEE Inform. Theory Workshop (Haifa, Israel, June 1996).
30. _, "The redundancy and distribution of the phrase lengths of the fixed-database Lempel-Ziv algorithm," IEEE Trans. Inform. Theory, vol. 43, pp. 1452-1464, Sept. 1997.
31. A. D. Wyner and J. Ziv, "Some asymptotic properties of entropy of a stationary ergodic data source with applications to data compression," IEEE Trans. Inform. Theory, vol. 35, pp. 1250-1258, Nov. 1989.
32. J. Ziv and A. Lempel, "A universal algorithm for sequential data compression," IEEE Trans. Inform. Theory, vol. IT-23, pp. 337-343, May 1977.
33. _, "Compression of individual sequences via variable rate coding," IEEE Trans. Inform. Theory, vol. IT-24, pp. 530-536, Sept. 1978.