Tabulation-based 5-independent hashing with applications to linear probing and second moment estimation

SIAM Journal on Computing

Volumn 41, Issue 2, 2012, Pages 299-331

  Thorup, Mikkel ^a   Zhang, Yin ^b  

^a AT AND T LABS RESEARCH   (United States)

^b University of Texas at Austin   (United States)

Author keywords

Hashing; K independence; Tabulation

Indexed keywords

DATA STREAM; HASHING; K-INDEPENDENCE; LINEAR PROBING; LOOKUPS; POLYNOMIAL METHODS; PRIME FIELD; SECOND MOMENTS; TABULATION;

EXPERIMENTS;

HASH FUNCTIONS;

EID: 84861633258     PISSN: 00975397     EISSN: None     Source Type: Journal    
DOI: 10.1137/100800774     Document Type: Article

References (33)

1. N. Alon, Y. Matias, and M. Szegedy, The space complexity of approximating the frequency moments, J. Comput. System Sci., 58 (1999), pp. 137-147.
2. B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom, Models and issues in data stream systems, in Proceedings of the 21st ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, 2002, pp. 1-16. (Pubitemid 35017047)
3. J. R. Black, C. U. Martel, and H. Qi, Graph and hashing algorithms for modern architectures: Design and performance, in Proceedings of the 2nd International Workshop on Algorithm Engineering, MPI Saarbrücken, 1998, pp. 37-48.
4. J. Bloemer, M. Kalfane, M. Karpinski, R. Karp, M. Luby, and D. Zuckerman, An XOR-based Erasure-resilient Coding Scheme, Technical Report TR-95-048, ICSI, Berkeley, CA, 1995; available online at http://www.icsi.berkeley.edu/~luby/ PAPERS/ cauchypap.ps.
5. A. Z. Broder and A. R. Karlin, Multilevel adaptive hashing, in Proceedings of the First Annual ACM-SIAM Symposium on Discrete Algorithms, San Francisco, 1990, pp. 43-53.
6. J. L. Carter and M. N. Wegman, Universal classes of hash functions, J. Comput. System Sci., 18 (1979), pp. 143-154.
7. M. Charikar, K. Chen, and M. Farach-Colton, Finding frequent items in data streams, in Proceedings of the 29th ICALP, Lecture Notes in Comput. Sci. 2380, Springer, New York, 2002, pp. 693-703.
8. C. D. Cranor, T. Johnson, O. Spatscheck, and V. Shkapenyuk, Gigascope: A stream database for network applications, in Proceedings of the 22nd ACM SIGMOD Conference, 2003, pp. 647-651.
9. M. Dietzfelbinger, Universal hashing and k-wise independent random variables via integer arithmetic without primes, in Proceedings of the 13th STACS, Lecture Notes in Comput. Sci. 1046, Springer, New York, 1996, pp. 569-580. (Pubitemid 126040206)
10. M. Dietzfelbinger, J. Gil, Y. Matias, and N. Pippenger, Polynomial hash functions are reliable, in Proceedings of the 19th ICALP, Lecture Notes in Comput. Sci. 623, Springer, New York, 1992, pp. 235-246.
11. M. Dietzfelbinger, T. Hagerup, J. Katajainen, and M. Penttonen, A reliable randomized algorithm for the closest-pair problem, J. Algorithms, 25 (1997), pp. 19-51. (Pubitemid 127451249)
12. M. Dietzfelbinger and P. Woelfel, Almost random graphs with simple hash functions, in Proceedings of the 35th Annual ACM Symposium on Theory of Computing, 2003, pp. 629-638.
13. G. L. Heileman and W. Luo, How caching affects hashing, in Proceedings of the 7thWorkshop on Algorithm Engineering and Experiments, SIAM, Philadelphia, 2005, pp. 141-154. (Pubitemid 43206506)
14. C. A. R. Hoare, Algorithm 64: Quicksort, Comm. ACM, 4 (1961), p. 321.
15. H. J. Karloff and P. Raghavan, Randomized algorithms and pseudorandom numbers, J. ACM, 40 (1993), pp. 454-476. (Pubitemid 23707890)
16. D. E. Knuth, Notes on "Open" Addressing, unpublished memorandum, 1963; available online at http://citeseer.ist.psu.edu/knuth63notes. html.
17. D. E. Knuth, The Art of Computer Programming, Volume 3: Sorting and Searching, Addison-Wesley, Reading, MA, 1973.
18. B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen, Sketch-based change detection: Methods, evaluation, and applications, in Proceedings of the ACM Internet Measurement Conference (IMC-03), 2003, pp. 234-247. (Pubitemid 40730656)
19. M. Mitzenmacher and S. P. Vadhan, Why simple hash functions work: Exploiting the entropy in a data stream, in Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, San Francisco, 2008, pp. 746-755.
20. S. Muthukrishnan, Data streams: Algorithms and applications, Found. Trends Theor. Comput. Sci., 1 (2005), pp. 117-236. (Pubitemid 44111222)
21. A. Pagh, R. Pagh, and M. Ruzic, Linear probing with constant independence, in Proceedings of the 39th Annual ACM Symposium on Theory of Computing, 2007, pp. 318-327. (Pubitemid 47630750)
22. A. Pagh, R. Pagh, and M. Ruzić, Linear probing with constant independence, SIAM J. Comput., 39 (2009), pp. 1107-1120.
23. M. Patrascu and M. Thorup, On the k-independence required by linear probing and minwise independence, in Proceedings of the 37th International Colloquium on Automata, Languages, and Programming, 2010, pp. 715-726.
24. M. P̌atrasçu and M. Thorup, The power of simple tabulation hashing, in Proceedings of the 43rd Annual ACM Symposium on Theory of Computing, ACM, New York, 2011, pp. 1-10; J. ACM, to appear.
25. L. Rizzo, Effective erasure codes for reliable computer communication protocols, ACM Comput. Comm. Rev., 27 (1997), pp. 24-36. (Pubitemid 127566056)
26. A. Schönhage and V. Strassen, Schnelle multiplication großer zahlen, Computing, 7 (1971), pp. 281-292.
27. A. Siegel, On universal classes of extremely random constant-time hash functions, SIAM J. Comput., 33 (2004), pp. 505-543.
28. M. Thorup, Even strongly universal hashing is pretty fast, in Proceedings of the Eleventh Annual ACM-SIAM Symposium on Discrete Algorithms, San Francisco, 2000, pp. 496-497. (Pubitemid 30549294)
29. M. Thorup, String hashing for linear probing, in Proceedings of the Twentieth Annual ACMSIAM Symposium on Discrete Algorithms, New York, 2009, pp. 655-664.
30. M. Thorup and Y. Zhang, Tabulation based 4-universal hashing with applications to second moment estimation, in Proceedings of the Fifteenth Annual ACM-SIAM Symposium on Discrete Algorithms, New Orleans, 2004, pp. 608-617.
31. M. Thorup and Y. Zhang, Tabulation based 5-universal hashing and linear probing, in Proceedings of the Twelfth Workshop on Algorithm Engineering and Experiments (ALENEX10), SIAM, Philadelphia, 2010, pp. 62-76.
32. M. Thorup and Y. Zhang, Source code for tabulation-based hashing, 2011; available online at http://www.cs.utexas.edu/~yzhang/software/hash.tar.gz.
33. M. N. Wegman and J. L. Carter, New hash functions and their use in authentication and set equality, J. Comput. System Sci., 22 (1981), pp. 265-279. (Pubitemid 12444638)