Workload scheduling in distributed stream processors using graph partitioning

Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

Volumn , Issue , 2015, Pages 124-133

  Fischer, Lorenz ^a   Bernstein, Abraham ^a  

^a UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPLICATION PROGRAMS; CLUSTERING ALGORITHMS; DISTRIBUTED PARAMETER CONTROL SYSTEMS; GRAPH THEORY; SCHEDULING; STORMS; TOPOLOGY;

BANDWIDTH CONSUMPTION; COMMUNICATION BEHAVIOR; DISTRIBUTED STREAM PROCESSING; GRAPH PARTITIONING ALGORITHMS; RESOURCE UTILIZATIONS; SCHEDULER IMPLEMENTATION; SCHEDULING STRATEGIES; STREAM PROCESSING SYSTEMS;

BIG DATA;

EID: 84963763803     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2015.7363749     Document Type: Conference Paper

References (27)

1. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters, " in Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation-OSDI '04.
2. B. T. Rao and L. Reddy, "Survey on improved scheduling in hadoop mapreduce in cloud environments, " International Journal of Computer Applications.
3. T. Akidau, A. Balikov, K. Bekiroglu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and W. Sam, "Millwheel : Faulttolerant stream processing at internet scale, " Proceedings of the VLDB Endowment.
4. D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi, "Naiad: A timely dataflow system, " in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles-SOSP '13.
5. Z. Qian, Y. He, C. Su, Z. Wu, H. Zhu, T. Zhang, L. Zhou, Y. Yu, and Z. Zhang, "TimeStream: Reliable stream computation in the cloud, " in Proceedings of the 8th ACM European Conference on Computer Systems-EuroSys '13.
6. S. Schneider, M. Hirzel, B. Gedik, and K.-L. Wu, "Auto-parallelizing stateful distributed streaming applications, " in Proceedings of the 21st international conference on Parallel architectures and compilation techniques-PACT '12.
7. L. Fischer, T. Scharrenbach, and A. Bernstein, "Scalable linked data stream processing via network-aware workload scheduling, " in 9th International Workshop on Scalable Semantic Web Knowledge Base Systems (2013).
8. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica, "Sparrow: Distributed, low latency scheduling, " in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles-SOSP '13.
9. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster computing with working sets, " in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing-HotCloud '10.
10. D. J. Abadi, Y. Ahmad, M. Balazinska, J.-h. Hwang, W. Lindner, A. S. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S. Zdonik, "The design of the borealis stream processing engine, " in Proceedings of the 2005 CIDR Conference. Citeseer.
11. P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, and M. Seltzer, "Network-aware operator placement for stream-processing systems, " in 22nd International Conference on Data Engineering-ICDE '06.
12. Y. Xing, S. Zdonik, and J.-H. Hwang, "Dynamic load distribution in the borealis stream processor, " in 21st International Conference on Data Engineering-ICDE '05.
13. Y. Xing, J.-H. Hwang, U. Ç etintemel, and S. Zdonik, "Providing resiliency to load variations in distributed stream processing, " in Proceedings of the 32nd International Conference on Very Large Data Bases, ser. VLDB '06.
14. T. Heinze, Z. Jerzak, G. Hackenbroich, and C. Fetzer, "Latencyaware elastic scaling for distributed data stream processing systems, " in Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems-DEBS '14.
15. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed data-parallel programs from sequential building blocks, " in Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007-EuroSys '07.
16. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: Fair scheduling for distributed computing clusters, " in Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles-SOSP '09.
17. J. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, K.-L. Wu, and L. Fleischer, "Soda: An optimizing scheduler for large-scale stream-based distributed computer systems, " in Middleware '08.
18. L. Aniello, R. Baldoni, and L. Querzoni, "Adaptive online scheduling in storm, " in Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems-DEBS '13.
19. V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez, "StreamCloud: An elastic and scalable data streaming system, " IEEE Transactions on Parallel and Distributed Systems.
20. E. Kalyvianaki, W. Wiesemann, Q. H. Vu, D. Kuhn, and P. Pietzuch, "Sqpr: Stream query planning with reuse, " Proceedings-International Conference on Data Engineering.
21. G. Karypis and V. Kumar, "A fast and high quality multilevel scheme for partitioning irregular graphs, " SIAM Journal on Scientific Computing.
22. A. Aletà, J. M. Codina, J. Sanchez, and A. González, "Graphpartitioning based instruction scheduling for clustered processors, " in Proceedings of the 34th Annual ACM/IEEE International Symposium on Microarchitecture.
23. C. Curino, E. Jones, Y. Zhang, and S. Madden, "Schism, " Proc. VLDB Endow.
24. B. L. Chamberlain, "Graph partitioning algorithms for distributing workloads of parallel computations, " Tech. Rep.
25. R. Khandekar, K. Hildrum, S. Parekh, D. Rajan, J. Wolf, K.-L. Wu, H. Andrade, and B. Gedik, "Cola: Optimizing stream processing applications via graph partitioning, " in Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware.
26. M. Al-Fares, A. Loukissas, and A. Vahdat, "A scalable, commodity data center network architecture, " in Proceedings of the ACM SIGCOMM 2008 conference on Data communication-SIGCOMM '08.
27. S. Gao, T. Scharrenbach, and A. Bernstein, "The CLOCK data-aware eviction approach: Towards processing linked data streams with limited resources, " in Proceedings of the 11th Extended Semantic Web Conference-ESWC '14.