A near-optimal algorithm for estimating the entropy of a stream

ACM Transactions on Algorithms

Volumn 6, Issue 3, 2010, Pages

  Chakrabarti, Amit ^a   Cormode, Graham ^b   McGregor, Andrew ^c  

^a Dartmouth College   (Canada)

^b AT AND T LABS RESEARCH   (United States)

^c The Manning College of Information and Computer Sciences   (United States)

Author keywords

Approximation algorithms; Data streams; Entropy

Indexed keywords

DATA STREAM; DATA STREAMS; LINEAR SPACES; LOWER BOUNDS; MULTIPLICATIVE FACTORS; NEAR-OPTIMAL ALGORITHMS; PROCESSING TIME; RANDOM WALK; SIMPLE ALGORITHM; SINGLE PASS; UNDIRECTED GRAPH;

DATA COMMUNICATION SYSTEMS; ENTROPY; HYDRAULICS; OPTIMIZATION;

APPROXIMATION ALGORITHMS;

EID: 77954392247     PISSN: 15496325     EISSN: 15496333     Source Type: Journal    
DOI: 10.1145/1798596.1798604     Document Type: Article

References (31)

1. ALON, N., MATIA S , Y., AND SZEGEDY, M. 1999. The space complexity of approximating the frequency moments. J. Comput. Syst. Sci. 58, 1, 137-147.
2. ARASU, A. AND MANKU, G. S. 2004. Approximate counts and quantiles over sliding windows. In Proceedings of the ACM Symposium on Principles of Database Systems. 286-296.
3. BABCOCK, B., DATA R , M., AND MOTWANI, R. 2002. Sampling from a moving window over streaming data. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 633-634.
4. BAR-YOSSEF, Z., JAYRAM, T. S., KUMAR, R., SIVAKUMAR, D., AND TREVISAN, L. 2002a. Counting distinct elements in a data stream. In Proceedings of the International Workshop on Randomization and Approximation Techniques (RANDOM). 1-10.
5. BAR-YOSSEF, Z., KUMAR, R., AND SIVAKUMAR, D. 2002b. Reductions in streaming algorithms, with an application to counting triangles in graphs. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 623-632.
6. BATU , T., DASGUPTA, S., KUMAR, R., AND RUBINFELD, R. 2005. The complexity of approximating the entropy. SIAM J. Comput. 35, 1, 132-150.
7. BHUVANAGIRI, L. AND GANGULY, S. 2006. Estimating entropy over data streams. In Proceedings of the European Symposium on Algorithms. 148-159.
8. BHUVANAGIRI, L., GANGULY, S., KESH, D., AND SAHA, C. 2006. Simpler algorithm for estimating frequency moments of data streams. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 708-713.
9. BOSE, P., KRANAKIS, E., MORIN, P., AND TANG, Y. 2003. Bounds for frequency estimation of packet streams. In Proceedings of the International Colloquium on Structural Information Complexity. 33-42.
10. CHAKRABARTI, A., CORMODE, G., AND MCGREGOR, A. 2007. A near-optimal algorithm for computing the entropy of a stream. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 328-335.
11. CHAKRABARTI, A., DO BA, K., AND MUTHUKRISHNAN, S. 2006. Estimating entropy and entropy norm on data streams. In Proceedings of the Symposium on Theoretical Aspects of Computer Science. 196-205.
12. CHAKRABARTI, A., KHOT, S., AND SUN, X. 2003. Near-Optimal lower bounds on the multi-party communication complexity of set disjointness. In Proceedings of the Annual IEEE Conference on Computational Complexity (CCC). 107-117.
13. CORMODE, G. AND MUTHUKRISHNAN, S. 2005a. An improved data stream summary: The count-min sketch and its applications. J. Algor. 55, 1, 58-75.
14. CORMODE, G. AND MUTHUKRISHNAN, S. 2005b. Space efficient mining of multigraph streams. In Proceedings of the ACM Symposium on Principles of Database Systems. 271-282.
15. COVER, T. AND THOMAS, J. 1991. Elements of Information Theory. John Wiley and Sons.
16. DATAR , M., GIONIS, A., INDYK, P., AND MOTWANI, R. 2002. Maintaining stream statistics over sliding windows. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms.
17. FEIGENBAUM, J., KANNAN, S., MCGREGOR, A., SURI, S., AND ZHANG, J. 2005. On graph problems in a semi-streaming model. Theor. Comput. Sci. 348, 2-3, 207-216.
18. GANGULY, S. A hybrid algorithm for estimating frequency moments of data streams. Unpublished manuscript, http://www.cse.iitk.ac.in/users/sganguly/ HybridFk.pdf.
19. GIBBONS, P. AND TIRTHAPURA, S. 2002. Distributed streams algorithms for sliding windows. In Proceedings of the ACM Symposium on Parallel Algorithms and Architectures.
20. GU, Y., MCCALLUM, A., AND TOWSLEY, D. 2005. Detecting anomalies in network traffic using maximum entropy estimation. In Proceedings of the Internet Measurement Conference.
21. GUHA, S., MCGREGOR, A., AND VENKATASUBRAMANIAN, S. 2006. Streaming and sublinear approximation of entropy and information distances. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 733-742.
22. HARVEY, N. J. A., NELSON, J., AND ONAK, K. 2008. Sketching and streaming entropy via approximation theory. In Proceedings of the IEEE Symposium on Foundations of Computer Science. 489-498.
23. INDYK, P. 2001. A small approximately min-wise independent family of hash functions. J. Algor. 38, 1, 84-90.
24. INDYK, P. AND WOODRUFF, D. P. 2003. Tight lower bounds for the distinct elements problem. In Proceedings of the IEEE Symposium on Foundations of Computer Science. 283-289.
25. INDYK, P. AND WOODRUFF, D. P. 2005. Optimal approximations of the frequency moments of data streams. In Proceedings of the ACM Symposium on Theory of Computing. 202-208.
26. LALL, A., SEKAR, V., OGIHARA, M., XU, J., AND ZHANG, H. 2006. Data streaming algorithms for estimating entropy of network traffic. In ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems.
27. MISRA, J. AND GRIES, D. 1982. Finding repeated elements. Sci. Comput. Program. 2, 143-152.
28. VITTER, J. S. 1985. Random sampling with a reservoir. ACM Trans. Math. Softw. 11, 1, 37-57.
29. WAGNER, A. AND PLATTNER, B. 2005. Entropy based worm and anomaly detection in fast IP networks. In Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises (WET ICE).
30. WOODRUFF, D. P. 2004. Optimal space lower bounds for all frequency moments. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 167-175.
31. XU, K., ZHANG, Z.-L., AND BHATTACHARYYA, S. 2005. Profiling internet backbone traffic: Behavior models and applications. In Proceedings of the ACM SIGCOMM Data Communications Festival. 169-180.