Leveraging discarded samples for tighter estimation of multiple-set aggregates

SIGMETRICS/Performance'09 - Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems

Volumn 37, Issue 1, 2009, Pages 251-262

  Cohen, Edith ^a   Kaplan, Haim ^b  

^a AT AND T LABS RESEARCH   (United States)

^b TEL AVIV UNIVERSITY   (Israel)

Author keywords

Algorithms; G.3: probabilistic algorithms; H.2 database management; Measurement; Performance

Indexed keywords

APPROXIMATE QUERY; DATA SETS; DATABASE MANAGEMENT; EMPIRICAL EVALUATIONS; ESTIMATION ERRORS; EXACT COMPUTATIONS; H.2 DATABASE MANAGEMENT; HYPERTEXT DOCUMENTS; MARKET BASKET; MASSIVE DATA SETS; PROBABILISTIC ALGORITHM; RANDOM SAMPLE; SYNTHETIC AND REAL DATA; TIME PERIODS; TYPICAL APPLICATION;

ALGORITHMS; ASSOCIATIVE PROCESSING; HYPERTEXT SYSTEMS; KEYS (FOR LOCKS); MANAGEMENT INFORMATION SYSTEMS;

ESTIMATION;

EID: 70449636034     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1555349.1555379     Document Type: Conference Paper

References (50)

1. R. Agrawal, T. Imielinski, and A. N. Swami. Mining association rules between sets of items in large databases. In SIGMOD, pages 207-216, 1993.
2. K. S. Beyer, P. J. Haas, B. Reinwald, Y. Sismanis, and R. Gemulla. On synopses for distinct-value estimation under multiset operations. In SIGMOD, pages 199-210, 2007.
3. K. Bharat and A. Z. Broder. Mirror, mirror on the web: A study of host pairs with replicated content. In WWW, pages 501-512, 1999.
4. B. Bloom. Space/time tradeoffs in in hash coding with allowable errors. Communications of the ACM, 13:422-426, 1970.
5. K. R. W. Brewer, L. J. Early, and S. F. Joyce. Selecting several samples from a single population. Australian Journal of Statistics, 14(3):231-239, 1972.
6. A. Z. Broder. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences, pages 21-29. ACM, 1997.
7. A. Z. Broder. Identifying and filtering near-duplicate documents. In CPM, pages 1-10, 2000.
8. A. Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-wise independent permutations. J. Comput. System Sci., 60(3):630-659, 2000.
9. M. Charikar, S. Chaudhuri, R. Motwani, and V. Narasayya. Towards estimation error guarantees for distinct values. In PODS, 2000.
10. M. S. Charikar. Similarity estimation techniques from rounding algorithms. In STOC, 2002.
11. A. Chowdhury, O. Frieder, D. Grossman, and M. C. McCabe. Collection statistics for fast duplicate document detection. ACM Transactions on Information Systems, 20(1):171-191, 2002.
12. E. Cohen. Size-estimation framework with applications to transitive closure and reachability. J. Comput. System Sci., 55:441-453, 1997.
13. E. Cohen, N. Duffield, H. Kaplan, C. Lund, and M. Thorup. Stream sampling for variance-optimal estimation of subset sums. In SODA, 2009.
14. E. Cohen, A. Fiat, and H. Kaplan. Associative search in Peer to Peer networks: Harnessing latent semantics. In INFOCOM, 2003.
15. E. Cohen and H. Kaplan. Efficient estimation algorithms for neighborhood variance and other moments. In SODA, 2004.
16. E. Cohen and H. Kaplan. Spatially-decaying aggregation over a network: model and algorithms. J. Comput. System Sci., 73:265-288, 2007.
17. E. Cohen and H. Kaplan. Summarizing data using bottom-k sketches. In PODC, 2007.
18. E. Cohen and H. Kaplan. Estimating aggregates over multiple sets. In ICDM, 2008.
19. E. Cohen and H. Kaplan. Tighter estimation using bottom-k sketches. In VLDB, 2008.
20. E. Cohen and M. Strauss. Maintaining time-decaying stream aggregates. J. Algorithms, 59:19-36, 2006.
21. E. Cohen, Y.-M. Wang, and G. Suri. When piecewise determinism is almost true. In Proc. Pacific Rim International Symposium on Fault-Tolerant Systems, pages 66-71, 1995.
22. J. G. Conrad and C. P. Schriber. Constructing a text corpus for inexact duplicate detection. In SIGIR, pages 582-583, 2004.
23. T. Dasu, T. Johnson, S. Muthukrishnan, and V. Shkapenyuk. Mining database structure; or, how to build a data quality browser. In SIGMOD, pages 240-251, 2002.
24. N. Duffield, M. Thorup, and C. Lund. Priority sampling for estimating arbitrary subset sums. J. Assoc. Comput. Mach., 54(6), 2007.
25. L. Fan, P. Cao, J. Almeida, and A. Z. Broder. Summary cache: a scalable wide-area Web cache sharing protocol. IEEE/ACM Transactions on Networking, 8(3):281-293, 2000.
26. J. Feigenbaum, S. Kannan, M. Strauss, and M. Viswanathan. An approximate L1-difference algorithm for massive data streams. In FOCS, pages 501-511, 1999.
27. P. Gibbons and S. Tirthapura. Estimating simple functions on the union of data streams. In Proceedings of the 13th Annual ACM Symposium on Parallel Algorithms and Architectures. ACM, 2001.
28. P. B. Gibbons. Distinct sampling for highly-accurate answers to distinct values queries and event reports. In VLDB, pages 541-550, 2001.
29. M. Hadjieleftheriou, X. Yu, N. Koudas, and D. Srivastava. Hashed samples: Selectivity estimators for set similarity selection queries. In VLDB, 2008.
30. J. Hájek. Sampling from a finite population. Marcel Dekker, New York, 1981.
31. M. R. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In SIGIR, pages 284-291, 2006.
32. D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663-685, 1952.
33. H. Kaplan and M. Sharir. Randomized incremental constructions of three-dimensional convex hulls and planar voronoi diagrams, and approximate range counting. In SODA, pages 484-493, 2006.
34. A. Kolcz, A. Chowdhury, and J. Alspector. Improved robustness of signature-based near-replica detection via lexicon randomization. In SIGKDD, pages 605-610, 2004.
35. P. Li and K. W. Church. A sketch algorithm for estimating two-way and multi-way associations. Computational Linguistics, 33:305-354, 2007.
36. U. Manber. Finding similar files in a large file system. In Usenix, pages 1-10, 1994.
37. G. S. Manku, A. Jain, and A. D. Sarma. Detecting nearduplicates for web crawling. In WWW, 2007.
38. D. Mosk-Aoyama and D. Shah. Computing separable functions via gossip. In PODC, 2006.
39. R. Motwani, E. Cohen, M. Datar, S. Fujiware, A. Gronis, P. Indyk, J. Ullman, and C. Yang. Finding interesting associations without support pruning. IEEE Transactions on Knowledge and Data Engineering, 13:64-78, 2001.
40. The Netflix Prize. http://www.netflixprize.com/.
41. E. Ohlsson. Sequential poisson sampling. J. Official Statistics, 14(2):149-162, 1998.
42. B. Rosén. Asymptotic theory for successive sampling with varying probabilities without replacement, I. The Annals of Mathematical Statistics, 43(2):373-397, 1972.
43. B. Rosén. Asymptotic theory for order sampling. J. Statistical Planning and Inference, 62(2):135-158, 1997.
44. B. Rosen. On sampling with probability proportional to size. J. Statistical Planning and Inference, 62(2):159-191, 1997.
45. S. Schleimer, D. Wilkerson, and A. Aiken. Winnowing: local algorithms for document fingerprinting. In SIGMOD, 2003.
46. N. T. Spring and D. Wetherall. A protocol-independent technique for eliminating redundant network traffic. In SIGCOMM, 2000.
47. K. Sripanidkulchai, B. Maggs, and H. Zhang. Efficient content location using interest-based locality in peer-to-peer systems. In INFOCOM, 2003.
48. M. Szegedy. The DLT priority sampling is essentially optimal. In STOC, 2006.
49. M. Szegedy and M. Thorup. On the variance of subset sum estimation. In ESA, 2007.
50. C. Tang, Z. Xu, and S. Dwarkadas. Peer-to-peer information retrieval using self-organizing semantic overlay networks. In SIGCOMM, 2003.