Bottom-k and priority sampling, set similarity and subset sums with minimal independence

Proceedings of the Annual ACM Symposium on Theory of Computing

Volumn , Issue , 2013, Pages 371-380

  Thorup, Mikkel ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

Estimation; Independence; Sampling

Indexed keywords

CONCENTRATION BOUNDS; CONSTANT FACTORS; INDEPENDENCE; PRIORITY SAMPLINGS; PROBABILITY ERRORS; RANDOM HASHING; RELATIVE ERRORS; SET SIMILARITY;

COMPUTATION THEORY; ERROR ANALYSIS; ESTIMATION; SAMPLING;

HASH FUNCTIONS;

EID: 84879835309     PISSN: 07378017     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2488608.2488655     Document Type: Conference Paper

References (36)

1. R. Adler, R. Feldman, and M. Taqqu. A Practical Guide to Heavy Tails. Birkhauser, 1998.
2. Y. Bachrach, R. Herbrich, and E. Porat. Sketching algorithms for approximating rank correlations in collaborative filtering systems. In Proc. 16th SPIRE, pages 344-352, 2009.
3. Y. Bachrach and E. Porat. Fast pseudo-random fingerprints. CoRR, abs/1009.5791, 2010.
4. Y. Bachrach, E. Porat, and J. S. Rosenschein. Sketching techniques for collaborative filtering. In Proc. 21st IJCAI, pages 2016-2021, 2009.
5. Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting distinct elements in a data stream. In International Workshop on Randomization and Approximation Techniques in Computer Science (RANDOM), pages 1-10, 2002.
6. A. Z. Broder. On the resemblance and containment of documents. In Proc. Compression and Complexity of Sequences (SEQUENCES), pages 21-29, 1997.
7. A. Z. Broder. Identifying and filtering near-duplicate documents. In Proc. 11th CPM, pages 1-10, 2000.
8. A. Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-wise independent permutations. Journal of Computer and System Sciences, 60(3):630-659, 2000. See also STOC'98.
9. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. Computer Networks, 29:1157-1166, 1997.
10. E. Cohen, M. Datar, S. Fujiwara, A. Gionis, P. Indyk, R. Motwani, J. D. Ullman, and C. Yang. Finding interesting associations without support pruning. IEEE Trans. Knowl. Data Eng., 13(1):64-78, 2001.
11. E. Cohen, N. Duffield, H. Kaplan, C. Lund, and M. Thorup. Efficient stream sampling for variance-optimal estimation of subset sums. SIAM Journal on Computing, 40(5):1402-1431, 2011. Announced at SODA'09.
12. E. Cohen and H. Kaplan. Summarizing data using bottom-k sketches. In Proc. 26th PODC, pages 225-234, 2007.
13. M. Datar and S. Muthukrishnan. Estimating rarity and similarity over data stream windows. In Proc. 10th ESA, pages 323-334, 2002.
14. M. Dietzfelbinger. Universal hashing and k-wise independent random variables via integer arithmetic without primes. In Proc. 13th STACS, pages 569-580, 1996.
15. M. Dietzfelbinger, J. Gil, Y. Matias, and N. Pippenger. Polynomial hash functions are reliable (extended abstract). In Proc. 19th ICALP, pages 235-246, 1992.
16. N. Duffield, C. Lund, and M. Thorup. Learn more, sample less: control of volume and variance in network measurements. IEEE Transactions on Information Theory, 51(5):1756-1775, 2005.
17. N. Duffield, C. Lund, and M. Thorup. Priority sampling for estimation of arbitrary subset sums. J. ACM, 54(6):Article 32, 2007. Announced at SIGMETRICS'04.
18. G. Feigenblat, E. Porat, and A. Shiftan. Even better framework for min-wise based algorithms. CoRR, abs/1102.3537, 2011. Accepted as "Exponential space improvement for min-wise based algorithms" for FSTTCS'12.
19. J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online aggregation. In Proc. SIGMOD Conference, pages 171-182, 1997.
20. M. R. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In Proc. 29th SIGIR, pages 284-291, 2006.
21. G. S. Manku, A. Jain, and A. D. Sarma. Detecting near-duplicates for web crawling. In Proc. 16th WWW, pages 141-150, 2007.
22. M. Mitzenmacher and S. P. Vadhan. Why simple hash functions work: exploiting the entropy in a data stream. In Proc. 19th ACM/SIAM Symposium on Discrete Algorithms (SODA), pages 746-755, 2008.
23. D. Moore, V. Paxson, S. Savage, C. Shannon, S. Staniford, and N. Weaver. Inside the slammer worm. IEEE Security and Privacy Magazine, 1(4):33-39, 2003.
24. R. Motwani and P. Raghavan. Randomized algorithms. Cambridge University Press, 1995.
25. A. Pagh, R. Pagh, and M. Ružić. Linear probing with constant independence. SIAM Journal on Computing, 39(3):1107-1120, 2009. See also STOC'07.
26. M. Pǎtraşcu and M. Thorup. On the k-independence required by linear probing and minwise independence. In Proc. 36th ICALP, Part I, LNCS 6198, pages 715-726, 2010.
27. M. Pǎatraşcu and M. Thorup. Twisted tabulation hashing. In Proc. 23nd SODA, pages 209-228, 2013.
28. C.-E. Särndal, B. Swensson, and J. Wretman. Model Assisted Survey Sampling. Springer, 1992.
29. S. Schleimer, D. S. Wilkerson, and A. Aiken. Winnowing: Local algorithms for document fingerprinting. In Proc. SIGMOD, pages 76-85, 2003.
30. J. P. Schmidt, A. Siegel, and A. Srinivasan. Chernoff-Hoeffding bounds for applications with limited independence. SIAM Journal on Discrete Mathematics, 8(2):223-250, 1995. See also SODA'93.
31. M. Szegedy. The DLT priority sampling is essentially optimal. In Proc. 38th STOC, pages 150-158, 2006.
32. M. Thorup. Confidence intervals for priority sampling. In Proc. SIGMETRICS, pages 252-263, 2006.
33. M. Thorup. Bottom-k and priority sampling, set similarity and subset sums with minimal independence. CoRR, 2013.
34. M. Thorup and Y. Zhang. Tabulation-based 5-independent hashing with applications to linear probing and second moment estimation. SIAM Journal on Computing, 41(2):293-331, 2012. Announced at SODA'04 and ALENEX'10.
35. M. N. Wegman and L. Carter. New classes and applications of hash functions. Journal of Computer and System Sciences, 22(3):265-279, 1981. See also FOCS'79.
36. H. Yang and J. P. Callan. Near-duplicate detection by instance-level constrained clustering. In Proc. 29th SIGIR, pages 421-428, 2006. 380.