Efficient key grouping for near-optimal load balancing in stream processing systems

DEBS 2015 - Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

Volumn , Issue , 2015, Pages 80-91

  Rivetti, Nicoló ^a,b   Querzoni, Leonardo ^b   Anceaume, Emmanuelle ^c   Busnel, Yann ^d   Sericola, Bruno ^e  

^a UNIVERSITÉ DE NANTES   (France)

^b SAPIENZA UNIVERSITY OF ROME   (Italy)

^c CAMPUS DE BEAULIEU   (France)

^d CREST   (France)

^e INRIA   (France)

Author keywords

Data streaming; Key grouping; Load balancing; Stream processing

Indexed keywords

HASH FUNCTIONS; RESOURCE ALLOCATION; SOFTWARE ARCHITECTURE;

DATA STREAMING; FREQUENT VALUES; KEY GROUPING; LOAD IMBALANCE; STREAM PROCESSING; STREAM PROCESSING SYSTEMS; THEORETICAL APPROXIMATIONS; VALUE DISTRIBUTION;

NETWORK MANAGEMENT;

EID: 84960952735     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2675743.2771827     Document Type: Conference Paper

References (17)

1. N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. In Proceedings of the 28th annual ACM Symposium on Theory of computing (STOC), 1996.
2. E. Anceaume, Y. Busnel, and B. Sericola. Uniform node sampling service robust against collusions of malicious nodes. In Proceedings of the 43rd IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2013.
3. Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting distinct elements in a data stream. In Proceedings of the 6th International Workshop on Randomization and Approximation Techniques (RANDOM), 2002.
4. L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web caching and zipf-like distributions: Evidence and implications. In Proceedings of the 18th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM), Volume 1, 1999.
5. M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. Theoretical Computer Science, 312(1), 2004.
6. DEBS 2013. Grand Challenge. http://www.orgs.ttu.edu/debs2013/index.php?goto=cfchallengedetails.
7. P. Flajolet and G. N. Martin. Probabilistic counting algorithms for data base applications. Journal of Computer and System Sciences, 31(2), 1985.
8. B. Gedik. Partitioning functions for stateful data parallelism in stream processing. The VLDB Journal, 23(4), 2014.
9. R. L. Graham. Bounds on multiprocessing timing anomalies. SIAM Journal on Applied Mathematics, 17(2), 1969.
10. S. Guha and A. McGregor. Quantile estimation in random-order streams. SIAM Journal on Computing, 38(5), 2009.
11. D. M. Kane, J. Nelson, and D. P. Woodruff. An optimal algorithm for the distinct element problem. In Proceedings of the Symposium on Principles of Databases (PODS), 2010.
12. R. Kumar, J. Novak, P. Raghavan, and A. Tomkins. On the bursty evolution of blogspace. World Wide Web, 8(2), 2005.
13. A. Metwally, D. Agrawal, and A. El Abbadi. Efficient computation of frequent and top-k elements in data streams. In Proceedings of the 10th International Conference on Database Theory (ICDT), 2005.
14. Muthukrishnan. Data Streams: Algorithms and Applications. Now Publishers Inc., 2005.
15. M. A. U. Nasir, G. D. F. Morales, D. G. Soriano, N. Kourtellis, and M. Serafini. The power of both choices: Practical load balancing for distributed stream processing engines. In Proceedings of the 31st IEEE International Conference on Data Engineering (ICDE), 2015.
16. O. Pearce, T. Gamblin, B. R. de Supinski, M. Schulz, and N. M. Amato. Quantifying the effectiveness of load balance algorithms. In Proceedings of the 26th ACM International Conference on Supercomputing (ICS), 2012.
17. The Apache Software Foundation. Apache Storm. http://storm.apache.org.