Fast heuristics for near-optimal task allocation in data stream processing over clusters

CIKM 2014 - Proceedings of the 2014 ACM International Conference on Information and Knowledge Management

Volumn , Issue , 2014, Pages 1579-1588

  Chatzistergiou, Andreas ^a   Viglas, Stratis D ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

KNOWLEDGE MANAGEMENT; OPTIMIZATION;

COMPUTATIONAL LOADS; DATA STREAM PROCESSING; DATA-INTENSIVE APPLICATION; DISTRIBUTED DATA STREAM PROCESSING; EXECUTION ENVIRONMENTS; NEAR-OPTIMAL ALLOCATION; REAL-WORLD SCENARIO; TASK ALLOCATION ALGORITHM;

DATA STREAMS;

EID: 84937604963     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2661829.2661882     Document Type: Conference Paper

References (39)

1. D. J. Abadi et al. Aurora: a new model and architecture for data stream management. The VLDB Journal, 2003.
2. D. J. Abadi et al. The design of the Borealis stream processing engine. In CIDR, 2005.
3. L. Amini et al. SPC: A distributed, scalable platform for data mining. In DMSSP, 2006.
4. L. Aniello et al. Adaptive online scheduling in Storm. In DEBS, 2013.
5. B. Babcock et al. Operator scheduling in data stream systems. The VLDB Journal, 13(4):333-353, 2004.
6. S. Bokhari. A shortest tree algorithm for optimal assignments across space and time in a distributed processor system. Software Engineering, IEEE Transactions on, SE-7(6), 1981.
7. S. H. Bokhari. Dual processor scheduling with dynamic reassignment. IEEE Trans. Softw. Eng., 5(4), 1979.
8. A. Burns. Scheduling hard real-time systems: a review. Softw. Eng. J., 6(3), 1991.
9. D. Carney et al. Operator scheduling in a data stream manager. In VLDB, 2003.
10. M.-S. Chern et al. An 1c branch-and-bound algorithm for the module assignment problem. Inform. Process. Lett., 1989.
11. M. Cherniack et al. Scalable distributed stream processing. In CIDR, 2003.
12. T. C. K. Chou and J. A. Abraham. Load balancing in distributed systems. IEEE Trans. Softw. Eng., 1982.
13. W. Chu. Optimal file allocation in a multiple computer system. Computers, IEEE Transactions on, C-18(10), 1969.
14. W. Chu et al. Task allocation in distributed data processing. Computer, 13(11), 1980.
15. W. W. Chu et al. Task allocation and precedence relations for distributed real-time systems. IEEE Trans. Comput., 1987.
16. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. In OSDI, 2004.
17. K. Efe. Heuristic models of task assignment scheduling in distributed systems. Computer, 15(6), 1982.
18. O. I. El-Dessouki et al. Distributed enumeration on between computers. Computers, IEEE Transactions on, 1980.
19. V. B. Gylys and J. A. Edwards. Optimal partitioning of workload for distributed systems. In COMPCON, 1976.
20. K. Haessig and C. J. Jenny. Partitioning and allocation computational objects in distributed computing systems. In Proc. IFIP Congr., 1980.
21. C. J. Jenny. Process partitioning in distributed systems. In Proc. NTC, 1977.
22. M. Kafil et al. Optimal task assignment in heterogeneous distributed computing systems. Concurrency, IEEE, 1998.
23. J. Kreps et al. Kafka: A distributed messaging system for log processing. In NetDB, 2011.
24. P.-Y. R. Ma et al. A task allocation model for distributed computing systems. Comp., IEEE Trans. on, 1982.
25. L. Neumeyer et al. S4: Distributed stream computing platform. In ICDM Workshops, 2010.
26. P. Pietzuch et al. Network-aware operator placement for stream-processing systems. In ICDE '06, 2006.
27. G. Rao et al. Assignment of tasks in a distributed processor system with limited memory. Comp., IEEE Trans. on, 1979.
28. A. Sarje and G. Sagar. Heuristic model for task allocation in distributed computer systems. Computers and Digital Techniques, IEEE Proceedings, 138(5), 1991.
29. M. A. Shah et al. Flux: An adaptive partitioning operator for continuous query systems. In ICDE, 2002.
30. J. Sheild. Partitioning concurrent VLSI simulation programs onto a multiprocessor by simulated annealing. Computers and Digital Techniques, IEEE Proceedings, 134(1), 1987.
31. C.-C. Shen et al. A graph matching approach to optimal task assignment in distributed computing systems using a minimax criterion. Computers, IEEE Transactions on, 1985.
32. H. S. Stone. Multiprocessor scheduling with the aid of network flow algorithms. IEEE Trans. Softw. Eng., 1977.
33. Storm. http://github.com/nathanmarz/storm.
34. J. Wolf et al. SODA: An optimizing scheduler for large-scale stream-based distributed computer systems. Lecture Notes in Computer Science, 2008.
35. Y. Xing. Dynamic load distribution in the Borealis stream processor. In ICDE, 2005.
36. Y. Xing. Load distribution for distributed stream processing. In EDBT Workshops, 2005.
37. Y. Xing et al. Providing resiliency to load variations in distributed stream processing. In VLDB, 2006.
38. X. Yang and X. Zhang. A general heuristic algorithm of task allocation in distributed systems. In Comp. and Appl., 1987.
39. Apache Zookeeper. http://zookeeper.apache.org/.