An integrated efficient solution for computing frequent and top-k elements in data streams

ACM Transactions on Database Systems

Volumn 31, Issue 3, 2006, Pages 1095-1133

  Metwally, Ahmed ^a,b   Agrawal, Divyakant ^a,b   El Abbadi, Amr ^a,b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b University of California   (United States)

Author keywords

Advertising networks; Approximate queries; Continuous queries; Data streams; Exact queries; Frequent elements; Top k elements; Zipfian data

Indexed keywords

APPROXIMATE QUERIES; CONTINUOUS QUERIES; DATA STREAMS; EXACT QUERIES; FREQUENT ELEMENTS; TOP-K ELEMENTS; ZIPFIAN DATA;

ALGORITHMS; APPROXIMATION THEORY; ERROR CORRECTION; PROBLEM SOLVING; QUERY LANGUAGES;

DATA STRUCTURES;

EID: 33750185779     PISSN: 03625915     EISSN: 03625915     Source Type: Journal    
DOI: 10.1145/1166074.1166084     Document Type: Conference Paper

References (47)

1. ALON, N., MATIAS, Y., AND SZEGEDY, M. 1996. The space complexity of approximating the frequency moments. In Proceedings of the 28th ACM STOC Symposium on the Theory of Computing. 20-29.
2. ARASU, A., BABU, S., AND WIDOM, J. 2003a. CQL: A language for continuous queries over streams and relations. In Proceedings of the 9th DBPL International Conference on Data Base and Programming Languages. 1-11.
3. ARASU, A., BABU, S., AND WIDOM, J. 2003b. The CQL Continuous Query Language: Semantic foundations and query execution. Tech. rep. 2002-67. Stanford University, Stanford, CA.
4. BABCOCK, B., BABU, S., DATAR, M., MOTWANI, R., AND WIDOM, J. 2002. Models and issues in data stream systems. In Proceedings of the 21st ACM PODS Symposium on Principles of Database Systems. 1-16.
5. BABCOCK, B. AND OLSTON, C. 2003. Distributed top-k monitoring. In Proceedings of the 22nd ACM SIGMOD International Conference on Management of Data. 28-39.
6. BLOOM, B. H. 1970. Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13, 7, 422-426.
7. BONNET, P., GEHRKE, J., AND SEBHADRI, P. 2001. Towards sensor database systems. In Proceedings of the 2nd IEEE MDM International Conference on Mobile Data Management. 3-14.
8. BOSE, P., KRANAKIS, E., MORIN, P., AND TANG, Y. 2003. Bounds for frequency estimation of packet streams. In Proceedings of the 10th SIROCCO International Colloquium on Structural Information and Communication Complexity, 33-42.
9. BOYER, R. AND MOORE, J. 1981. A fast majority vote algorithm. Tech. rep. 1981-32. Institute for Computing Science, University of Texas, Austin, Austin, TX.
10. CHARIKAR, M., CHEN, K., AND FARACH-COLTON, M. 2002. Finding frequent items in data streams. In Proceedings of the 29th ICALP International Colloquium on Automata, Languages and Programming. 693-703.
11. CHEN, J., DEWITT, D., TIAN, F., AND WANG, Y. 2000. NiagaraCQ: A scalable continuous query system for Internet databases. In Proceedings of the 19th ACM SIGMOD International Conference on Management of Data. 379-390.
12. CORMODE, G., KORN, F., MUTHUKRISHNAN, S., AND SRIVASTAVA, D. 2003. Finding hierarchical heavy hitters in data streams. In Proceedings of the 29th VLDB International Conference on Very Large Data Bases. 464-475.
13. CORMODE, G., KORN, F., MUTHUKRISHNAN, S., AND SRIVASTAVA, D. 2004. Diamond in the rough: Finding hierarchical heavy hitters in multi-dimensional data. In Proceedings of the 23rd ACM SIGMOD International Conference on Management of Data. 155-166.
14. CORMODE, G. AND MUTHUKRISHNAN, S. 2003. What's hot and what's not: Tracking most frequent items dynamically. In Proceedings of the 22nd ACM PODS Symposium on Principles of Database Systems. 296-306.
15. CORTES, C., FISHER, K., PREGIBON, D., ROGERS, A., AND SMITH, F. 2000. Hancock: A language for extracting signatures from data streams. In Proceedings of the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 9-17.
16. DATAR, M., GIONIS, A., INDYK, P., AND MOTWANI, R. 2002. Maintaining stream statistics over sliding windows. In Proceedings of the 13th ACM SIAM Symposium on Discrete Algorithms. 635-644.
17. DEMAINE, E., LÓPEZ-ORTIZ, A., AND MUNRO, J. 2002. Frequency estimation of internet packet streams with limited space. In Proceedings of the 10th ESA Annual European Symposium on Algorithms. 348-360.
18. ESTAN, C. AND VARGHESE, G. 2003. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst. 21, 3, 270-313.
19. FANG, M., SHIVAKUMAR, S., GARCIA-MOLINA, H., MOTWANI, R., AND ULLMAN, J. 1998. Computing iceberg queries efficiently. In Proceedings of the 24th VLDB International Conference on Very Large Data Bases. 299-310.
20. FEIGENBAUM, J., KANNAN, S., STRAUSS, M., AND VISWANATHAN, M. 1999. An approximate L1-difference algorithm for massive data streams. In Proceedings of 40th FOCS Annual Symposium on Foundations of Computer Science. 501-511.
21. FISCHER, M. AND SALZBERG, S. 1982. Finding a majority among N votes: Solution to problem 81-5. J. Algorith. 3, 376-379.
22. FLAJOLET, P. AND MARTIN, G. 1985. Probabilistic counting algorithms. J. Comput. Syst. Sci. 31, 182-209.
23. GEHRKE, J., KORN, F., AND SRIVASTAVA, D. 2001. On computing correlated aggregates over continual data streams. In Proceedings of the 20th ACM SIGMOD International Conference on Management of Data. 13-24.
24. GIBBONS, P. AND MATIAS, Y. 1998. New sampling-based susmary statistics for improving approximate query answers. In Proceedings of the 17th ACM SIGMOD International Conference on Management of Data. 331-342.
25. GILBERT, A., KOTIDIS, Y., MUTHUKRISHNAN, S., AND STRAUSS, M. 2001. Surfing wavelets on streams: One-pass summaries for approximate aggregate queries. In Proceedings of the 27th VLDB International Conference on Very Large Data Bases. 79-88.
26. GOLAB, L., DEHAAN, D., DEMAINE, E., LÓPEZ-ORTIZ, A., AND MUNRO, J. 2003. Identifying frequent items in sliding windows over online packet streams. In Proceedings of the 1st ACM SIGCOMM Internet Measurement Conference. 173-178.
27. GOLAB, L. AND OZSU, M. 2003. Issues in data stream management. ACM SIGMOD Rec. 32, 2, 5-14.
28. GREENWALD, M. AND KHANNA, S. 2001. Space-efficient online computation of quantile summaries. In Proceedings of the 19th ACM SIGMOD International Conference on Management of Data. 58-66.
29. GUHA, S., INDYK, P., MUTHUKRISHNAN, M., AND STRAUSS, M. 2002. Histogramming data streams with fast per-item processing. In Proceedings of the 29th ICALP International Colloquium on Automata, Languages and Programming. 681-692.
30. GUHA, S., KOUDAS, N., AND SHIM, K. 2001. Data-streams and histograms. In Proceedings of the 33rd ACM STOC Symposium on the Theory of Computing. 471-475.
31. GUNDUZ, S. AND OZSU, M. 2003. A Web page prediction model based on click-stream tree representation of user behavior. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 535-540.
32. GUPTA, P. AND MCKEOWN, N. 1999. Packet classification on multiple fields. In Proceedings of the ACM SIGCOMM International Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication. 147-160.
33. HAAS, P., NAUGHTON, J., SEHADRI, S., AND STOKES, L. 1995. Sampling-based estimation of the number of distinct values of an attribute. In Proceedings of the 21st VLDB International Conference on Very Large Data Bases. 311-322.
34. HOARE, C. 1961. Algorithm 65: Find. Commun. ACM 4, 7, 321-322.
35. JIN, C., QIAN, W., SHA, C., YU, J., AND ZHOU, A. 2003. Dynamically maintaining frequent items over a data stream. In Proceedings of the 12th ACM CIKM International Conference on Information and Knowledge Management. 287-294.
36. KARP, R., SHENKER, S., AND PAPADIMITRIOU, C. 2003. A simple algorithm for finding frequent elements in streams and bags. ACM Trans. Database Syst. 28, 1, 51-55.
37. KIRSCHENHOFER, P., PRODINGER, H., AND MARINEZ, C. 1997. Analysis of Hoare's FIND algorithm with median-of-three partition. Random Struct. Algorith. 10, 1-2, 143-156.
38. LIN, X., LU, H., XU, J., AND YU, J. 2004. Continuously maintaining quantile summaries of the most recent N elements over a data stream. In Proceedings of the 20th IEEE ICDE International Conference on Data Engineering. 362-374.
39. MANKU, G. AND MOTWANI, R. 2002. Approximate frequency counts over data streams. In Proceedings of the 28th VLDB International Conference on Very Large Data Bases. 346-357.
40. MANKU, G., RAJAGOPALAN, S., AND LINDSAY, B. 1999. random sampling techniques for space efficient online computation of order statistics of large datasets. In Proceedings of the 18th ACM SIGMOD International Conference on Management of Data. 251-262.
41. MATIAS, Y., VITTER, J., AND WANG, M. 2000. Dynamic maintenance of wavelet-based histograms. In Proceedings of the 26th VLDB International Conference on Very Large Data Bases. 101-110.
42. METWALLY, A., AGRAWAL, D., AND EL ABBADI, A. 2005a. Duplicate detection in click streams. In Proceedings of the 14th WWW International World Wide Web Conference. 12-21. An extended version appears as a University of California, Santa Barbara, Department of Computer Science Technical Report 2004-23.
43. METWALLY, A., AGRAWAL, D., AND EL ABBADI, A. 2005b. Efficient computation of frequent and top-k elements in data streams. In Proceedings of the 10th ICDT International Conference on Database Theory. 398-412. An extended version appears as a University of California, Santa Barbara, Department of Computer Science, Technical Report 2005-23.
44. MISRA, J. AND GRIES, D. 1982. Finding repeated elements. Sci. Comput. Programm. 2, 143-152.
45. WHANG, K., VANDER-ZANDEN, B., AND TAYLOR, H. 1990. A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15, 208-229.
46. ZHU, Y. AND SHASHA, D. 2002. StatStream: Statistical monitoring of thousands of data streams in real time. In Proceedings of the 28th VLDB International Conference on Very Large Data Bases. 358-369.
47. ZIPF, G. 1949. Human Behavior and The Principle of Least Effort. Addison-Wesley, Reading, MA.