Distributed hash sketches: Scalable, efficient, and accurate cardinality estimation for distributed multisets

ACM Transactions on Computer Systems

Volumn 27, Issue 1, 2009, Pages

  Ntarmos, N ^a   Triantafillou, P ^a   Weikum, G ^b  

^a University of Patras and Computer Technology Institute   (Greece)

^b MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

Author keywords

Distributed cardinality estimation; Distributed data summary structures; Distributed estimation; Distributed information systems; Hash sketches; Peer to peer networks and systems

Indexed keywords

DISTRIBUTED CARDINALITY ESTIMATION; DISTRIBUTED DATA SUMMARY STRUCTURES; DISTRIBUTED ESTIMATION; DISTRIBUTED INFORMATION SYSTEMS; HASH SKETCHES; PEER-TO-PEER NETWORKS AND SYSTEMS;

APPLICATIONS; ESTIMATION; INFORMATION SERVICES; INFORMATION SYSTEMS; MOBILE TELECOMMUNICATION SYSTEMS; TELECOMMUNICATION NETWORKS;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 60449101900     PISSN: 07342071     EISSN: 15577333     Source Type: Journal    
DOI: 10.1145/1482619.1482621     Document Type: Article

References (60)

1. ABERER, K., DATTA, A., HAUSWIRTH, M., AND SCHMIDT, R. 2005. Indexing data-oriented overlay networks. In Proceedings of the International Conference on Very Large Databases (VLDB).
2. ALON, N., GIBBONS, P. B., MATIAS, Y., AND SZEGEDY, M. 1999. Tracking join and self-join sizes in limited storage. In Proceedings of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS).
3. ALON, N., MATIAS, Y., AND SZEGEDY, M. 1996. The space complexity of approximating the frequency moments. In Proceedings of the Annual ACM Symposium on Theory of Computing (STOC).
4. ASPNES, J. AND SHAH, G. 2003. Skip Graphs. In Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms (SODA).
5. BABAOǦLU, Ö., MELING, H., AND MONTRESOR, A. 2002. Anthill: A framework for the development of agent-based peer-to-peer systems. In Proceedings of the IEEE International Conference on Distributed Computing and Systems (ICDCS).
6. BAR-YOSSEP, Z., JAYRAM, T., KUMAR, R., SIVAKUMAR, D., AND TREVISAN, L. 2002. Counting distinct elements in a data stream. In Proceedings of the International Workshop on Randomization and Approximation Techniques (RANDOM).
7. BAWA, M., GARCIA-MOLINA, H., GIONIS, A., AND MOTWANI, R. 2003. Estimating aggregates on a peer-to-peer network. Tech. rep., Computer Science Department, Stanford University.
8. BAWA, M., GIONIS, A., GARCIA-MOLINA, H., AND MOTWANI, R. 2004. The price of validity in dynamic networks. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
9. BEYER, K., HAAS, P. J., REINWALD, B., SISMANIS, Y., AND GEMULLA, R. 2007. On synopses for distinctvalue estimation under multiset operations. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
10. BHARAMBE, A., AGRAWAL, M., AND SESHAN, S. 2004. Mercury: Supporting scalable multi-attribute range queries. In Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM).
11. CHAUDHURI, S., MOTWANI, R., AND NARASAYYA, R. 1998. Random sampling for histogram construction: How much is enough? In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
12. CONSIDINE, J., LI, F., KOLLIOS, G., AND BYERS, J. 2004. Approximate aggregation techniques for sensor databases. In Proceedings of the International Conference on Data Engineering (ICDE).
13. CORMODE, G. AND GAROFALAKIS, M. N. 2005. Sketching streams through the net: Distributed approximate query tracking. In Proceedings of the International Conference on Very Large Databases (VLDB).
14. CORMODE, G. AND MUTHUKRISHNAN, S. 2004. An improved data stream summary: The count-min sketch and its applications. In Proceedings of the Latin American Symposium on Theoretical Informatics (LATIN).
15. DABEK, F., LI, J., SIT, E., ROBERTSON, J., KAASHOEK, M. F., AND MORRIS, R. 2004. Designing a DHT for low latency and high throughput. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI).
16. DAMGAARD, C. AND WEINER, J. 2000. Describing inequality in plant size or fecundity. Ecology 81, 1139-1142.
17. DOBRA, A., GAROFALAKIS, M., GEHRKE, J., AND RASTOGI, R. 2004. Sketch-Based multi-query processing over data streams. In Proceedings of the International Conference on Extending Database Technology (EDBT).
18. DRUSCHEL, P. AND ROWSTRON, A. 2001. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In Proceedings of the IFIP/ ACM IFIP/ ACM International Conference on Distributed Systems Platforms (Middleware).
19. DURAND, M. AND FLAJOLET, P. 2003. Loglog counting of large cardinalities. In Proceedings of the Annual European Symposium on Algorithms (ESA).
20. FLAJOLET, P. AND MARTIN, G. N. 1985. Probabilistic counting algorithms for data base applications. J. Comput. Syst. Sci. 31, 2, 182-209.
21. FEEEDHS. 2006. Homepage. http://netcins.ceid.upatras.gr/DHS.php.
22. FREEPASTRY. 2002. Homepage. http://freepastry.org/ FreePastry/.
23. GANGULY, S., GAROFALAKIS, M., AND RASTOGI, R. 2003. Processing set expressions over continuous update streams. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
24. GANGULY, S., GIBBONS, P. B., MATIAS, Y., AND SILBERSCHATZ, A. 1996. Bifocal sampling for skewresistant join size estimation. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
25. GNUTELLA. 2001. Homepage, http://gnutella.wego.com/.
26. GUMMADI, K., GUMMADI, R., GRIBBLE, S., RATNASAMY, S., SHENKER, S., AND STOICA, I. 2003. The impact of DHT routing geometry on resilience and proximity. In Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM).
27. GUPTA, A., AGRAWAL, D., AND EL ABBADI, A. 2003. Approximate range selection queries in peer-to-peer systems. In Proceedings of the ACM SIGMODIVLDB Biennial Conference on Innovative Data Systems Research (CIDR).
28. HADJIELEFTHERIOU, M., BYERS, J. W., AND KOLLIOS, G. 2005. Robust sketching and aggregation of distributed data streams. Tech. rep. 2005-011, Computer Science Department, Boston University.
29. HARREN, M., HELLERSTEIN, J. M., HUEBSCH, R., LOO, B. T., SHENKER, S., AND STOICA, I. 2002. Complex queries in DHT-based peer-to-peer networks. In Proceedings of the International Workshop on Peer-to-Peer Systems (IPTPS).
30. HARVEY, N., JONES, M., SAROIU, S., THEIMER, M., AND WOLMAN, A. 2003. Skipnet: A scalable overlay network with practical locality properties. In Proceedings of the USENIX Symposium on Internet Technologies and Systems (USITS).
31. HUEBSCH, R., CHUN, B. N., HELLERSTEIN, J. M., LOO, B. T., MANIATIS, P., ROSCOE, T., SHENKER, S., STOICA, I., AND YUMEREFENDI, A. R. 2005. The architecture of PIER: An Internet-scale query processor. In Proceedings of the ACM SIGMOD / VLDB Biennial Conference on Innovative Data Systems Research (CIDR).
32. HUEBSCH, R., HELLERSTEIN, J. M., LANHAM, N., LOO, B. T., SHENKER, S., AND STOICA, I. 2003. Querying the Internet with PIER. In Proceedings of the International Conference on Very Large Data Bases (VLDB).
33. IVES, Z., KHANDELWAL, N., KAPUR, A., AND CAKIR, M. 2005. ORCHESTRA: Rapid, collaborative sharing of dynamic data. In Proceedings of the ACM SIGMOD / VLDB Biennial Conference on Innovative Data Systems Research (CIDR).
34. JELASITY, M. AND MONTRESOR, A. 2004. Epidemic-Style proactive aggregation in large overlay networks. In Proceedings of the IEEE International Conference on Distributed Computing and Systems (ICDCS).
35. KARGER, D., LEHMAN, E., LEIGHTON, T., LEVINE, M., LEWIN, D., AND PANIGRAHY, R. 1997. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web. In Proceedings of the Annual ACM Symposium on Theory of Computing (STOC'97).
36. KEMPE, D., DOBRA, A., AND GEHRKE, J. 2003. Computing aggregate information using gossip. In Proceedings of the Annual IEEE Symposium on Foundations of Computer Science (FOCS).
37. KOLONIARI, G. AND PITOURA, E. 2004. Content-based routing of path queries in peer-to-peer systems. In Proceedings of the International Conference on Extending Database Technology (EDBT).
38. KRISHNAN, P. 1995. Online prediction algorithms for databases and operating systems. Ph.D. thesis, Brown University.
39. LIPTON, R. AND NAUGHTON, J. F. 1995. Query size estimation by adaptive sampling. J. Comput. Syst. Sci. 51, 1, 18-25.
40. LIPTON, R. J., NAUGHTON, J. F., AND SCHNEIDER, D. A. 1990. Practical selectivity estimation through adaptive sampling. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD).
41. MANKU, G. 2003. Routing networks for distributed hash tables. In Proceedings of the ACM Symposium on Principles of Distributed Computing (PODC).
42. MAYMOUKNOV, P. AND MAZIÈRES, D. 2002. Kademlia: A peer-to-peer information system based on the XOR metric. In Proceedings of the International Workshop on Peer-to-Peer Systems (IPTPS).
43. MICHEL, S., BENDER, M., NTARMOS, N., TRIANTAFILLOU, P., WEIKUM, G., AND ZIMMER, C. 2006. Discovering and exploiting keyword and attribute-value co-occurrences to improve P2P routing indices. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM).
44. MONTRESOR, A., MELING, H., AND BABAOǦLU, Ö. 2002. Messor: Load-Balancing through a swarm of autonomous agents. In Proceedings of the Workshop on Agent and Peer-to-Peer Systems.
45. NG, W. S., OOI, B. C., TAN, K. L., AND ZHOU, A. 2003. PeerDB: A P2P-based system for distributed data sharing. In Proceedings of the International Conference on Data Engineering (ICDE).
46. NTARMOS, N. AND TRIANTAFILLOU, P. 2004. AESOP: Altruism-Endowed self-organizing peers. In Proceedings of the International Workshop on Databases, Information Systems and Peer-to-Peer Computing (DBISP2P).
47. NTARMOS, N., TRIANTAFILLOU, P., AND WEIKUM, G. 2006. Counting at large: Efficient cardinality estimation in Internet-scale data networks. In Proceedings of the International Conference on Data Engineering (ICDE).
48. PALMER, C. R., SIGANOS, G., FALOUTSOS, M., FALOUTSOS, C., AND GIBBONS, P. B. 2001. The connectivity and fault-tolerance of the Internet topology. In Proceedings of the Workshop on Network-Related Data Management (NRDM).
49. PAPADIMOS, V., MAIER, D., AND TUFTE, K. 2003. Distributed query processing and catalogs for peer-to-peer systems. In Proceedings of the ACM SIGMOD / VLDB Biennial Conference on Innovative Data Systems Research (CIDR).
50. PITOURA, T., NTARMOS, N., AND TRIANTAFILLOU, P. 2006. Replication, load balancing, and efficient range query processing in DHT data networks. In Proceedings of the International Conference on Extending Database Technology (EDBT).
51. PITOURA, T. AND TRIANTAFILLOU, P. 2007. Load distribution fairness in p2p data management systems. In Proceedings of the International Conference on Data Engineering (ICDE).
52. RATNASAMY, S., FRANCIS, P., HANDLEY, M., KARP, R., AND SHENKER, S. 2001. A scalable contentaddressable network. In Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM).
53. RHEA, S., GEELS, D., ROSCOE, T., AND KUBIATOWICZ, J. 2004. Handling churn in a DHT. In Proceedings of the USENIX Annual Technical Conference (USENIX).
54. SAROIU, S., GUMMADI, P. K., AND GRIBBLE, S. D. 2002. A measurement study of peer-to-peer file sharing systems. In Proceedings of the Multimedia Computing and Networking Conference (MMCN).
55. STOICA, I., MORRIS, R., KARGER, D., KAASHOEK, M. F., AND BALAKRISHNAN, H. 2001. Chord: A scalable peer-to-peer lookup service for Internet applications. In Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM).
56. TRIANTAFILLOU, P. AND PITOURA, T. 2003. Towards a unifying framework for complex query processing over structured peer-to-peer data networks. In Proceedings of the International Workshop on Databases, Information Systems and Peer-to-Peer Computing (DBISP2P).
57. VAN RENESSE, R., BIRMAN, K. P., AND VOGELS, W. 2003. Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst. 21, 2, 164-206.
58. YALAGANDULA, P. AND DAHLIN, M. 2004. A scalable distributed information management system. In Proceedings of the ACM Symposium on Communications Architectures and Protocols (SIGCOMM).
59. YANG, B. AND GARCIA-MOLINA, H. 2001. Comparing hybrid peer-to-peer systems. In Proceedings of the International Conference on Very Large Data Bases (VLDB).
60. ZHAO, B. Y., KUBIATOWICZ, J. D., AND JOSEPH, A. D. 2001. Tapestry: An infrastructure for fault-tolerant wide-area location and routing. Tech. rep. UCB/CSD-01-1141, University of California at Berkeley, Computer Science Department.