Model-driven scheduling for distributed stream processing systems

Journal of Parallel and Distributed Computing

Volumn 117, Issue , 2018, Pages 98-114

  Shukla, Anshu ^a   Simmhan, Yogesh ^a  

^a INDIAN INSTITUTE OF SCIENCE   (India)

Author keywords

Big data; Cloud computing; Distributed systems; Performance models; Scheduling algorithms; Stream processing

Indexed keywords

BENCHMARKING; BIG DATA; CLOUD COMPUTING; DIRECTED GRAPHS; DISTRIBUTED PARAMETER CONTROL SYSTEMS; MAPPING; SCHEDULING; SCHEDULING ALGORITHMS;

DIRECTED ACYCLIC GRAPH (DAG); DISTRIBUTED RESOURCES; DISTRIBUTED STREAM PROCESSING; DISTRIBUTED SYSTEMS; MODEL-DRIVEN TECHNIQUES; PERFORMANCE MODEL; STREAM PROCESSING; STREAMING APPLICATIONS;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 85044845984     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2018.02.003     Document Type: Article

References (40)

1. Abadi, D.J., Ahmad, Y., Balazinska, M., Cetintemel, U., Cherniack, M., Hwang, J.-H., Lindner, W., Maskey, A., Rasin, A., Ryvkina, E., et al. The design of the borealis stream processing engine. Conference on Innovative Data Systems Research, CIDR, 2005, ACM, 277–289.
2. Aniello, L., Baldoni, R., Querzoni, L., Adaptive online scheduling in storm. International Conference on Distributed Event-Based Systems, DEBS, 2013, ACM, 207–218.
3. D.H. Bailey, A. Snavely, Performance modeling: Understanding the past and predicting the future, in: European Conference on Parallel Processing, Euro-Par, 2005, pp. 185–195.
4. Biem, A., Bouillet, E., Feng, H., Ranganathan, A., Riabov, A., Verscheure, O., Koutsopoulos, H., Moran, C., IBM infosphere streams for scalable, real-time, intelligent transportation services. International Conference on Management of Data, SIGMOD, 2010, ACM, 1093–1104.
5. Carbone, P., Ewen, S., Haridi, S., Katsifodimos, A., Markl, V., Tzoumas, K., Apache Flink: Stream and batch processing in a single engine. Data Eng. Bull., 2015, 28.
6. Cardellini, V., Nardelli, M., Luzi, D., Elastic stateful stream processing in storm. High Performance Computing & Simulation, HPCS, 2016, IEEE, 583–590.
7. Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S.R., Reiss, F., Shah, M.A., Telegraphcq: Continuous dataflow processing. International Conference on Management of Data, SIGMOD, 2003, ACM pp. 668–668.
8. Chen, G.J., Wiener, J.L., Iyer, S., Jaiswal, A., Lei, R., Simha, N., Wang, W., Wilfong, K., Williamson, T., Yilmaz, S., Realtime data processing at Facebook. International Conference on Management of Data, SIGMOD, 2016, ACM, 1087–1098.
9. De Roure, D., Goble, C., Stevens, R., The design and realisation of the virtual research environment for social sharing of workflows. Future Gener. Comput. Syst. 25:5 (2009), 561–567.
10. B. Donovan, D.B. Work, Using coarse GPS data to quantify city-scale transportation system resilience to extreme events, in: Transportation Research Board 94th Annual Meeting, 2014, pp. 1–1.
11. Eskandari, L., Huang, Z., Eyers, D., P-scheduler: Adaptive hierarchical scheduling in apache storm. Australasian Computer Science Week Multiconference, ACSW, 2016, ACM pp. 26:1–26:10.
12. Feitelson, D.G., Rudolph, L., Gang scheduling performance benefits for fine-grain synchronization. J. Parallel Distrib. Comput. 16:4 (1992), 306–318.
13. Fu, T.Z., Ding, J., Ma, R.T., Winslett, M., Yang, Y., Zhang, Z., Drs: dynamic resource scheduling for real-time analytics over fast streams. International Conference on Distributed Computing Systems, ICDCS, 2015, IEEE, 411–420.
14. Gedik, B., Andrade, H., Wu, K.-L., Yu, P.S., Doo, M., Spade: The system s declarative stream processing engine. International Conference on Management of Data, SIGMOD, 2008, ACM, 1123–1134.
15. Gedik, B., Schneider, S., Hirzel, M., Wu, K.-L., Elastic scaling for data stream processing. Trans. on Parallel and Distr. Proc. Sys., TPDS 25:6 (2014), 1447–1463.
16. Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M., Internet of Things (IoT): A vision, architectural elements, and future directions. Future Gener. Comput. Syst. 29:7 (2013), 1645–1660.
17. Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., Valduriez, P., Streamcloud: An elastic and scalable data streaming system. Trans. on Parallel and Distr. Proc. Sys., TPDS 23:12 (2012), 2351–2365.
18. Jha, S., Luckow, D.S.K.A., Rana, O., Simmhan, Y., Hong, N.C., Introducing distributed dynamic data-intensive (D3) science: Understanding applications and infrastructure. Concurr. Comput.: Pract. Exper., 2016, 38.
19. Joseph B. Dence, T.P.D., A rapidly converging recursive approach to Pi. Math. Teacher 86:2 (1993), 121–124.
20. Khandekar, R., Hildrum, K., Parekh, S., Rajan, D., Wolf, J., Wu, K.-L., Andrade, H., Gedik, B., Cola: Optimizing stream processing applications via graph partitioning. International Conference on Middleware, Middleware, 2009, Springer, 16:1–16:20.
21. Kulkarni, S., Bhagat, N., Fu, M., Kedigehalli, V., Kellogg, C., Mittal, S., Patel, J.M., Ramasamy, K., Taneja, S., Twitter heron: Stream processing at scale. International Conference on Management of Data, SIGMOD, 2015, ACM, 239–250.
22. Kumbhare, A., Simmhan, Y., Prasanna, V.K., Exploiting application dynamism and cloud elasticity for continuous dataflows. International Conference on High Performance Computing, Networking, Storage and Analysis, SC, 2013, IEEE, 1–12.
23. Laney, D., 3D data management: Controlling data volume, velocity and variety. META Group Res. Note, 6, 2001, 70.
24. Li, B., Diao, Y., Shenoy, P., Supporting scalable analytics with latency constraints. Proc. VLDB Endowment 8:11 (2015), 1166–1177.
25. Lu, R., Wu, G., Xie, B., Hu, J., Stream bench: Towards benchmarking modern distributed stream computing frameworks. International Conference on Utility and Cloud Computing, UCC, 2014, IEEE, 69–78.
26. Ousterhout, J.K., Scheduling techniques for concurrent systems. International Conference on Distributed Computing Systems, ICDCS, 1982, 22–30.
27. Peng, B., Hosseini, M., Hong, Z., Farivar, R., Campbell, R., R-storm: Resource-aware scheduling in storm. International Conference on Middleware, Middleware, 2015, ACM, 149–161.
28. Rivetti, N., Anceaume, E., Busnel, Y., Querzoni, L., Sericola, B., Online scheduling for shuffle grouping in distributed stream processing systems. International Conference on Middleware, Middleware, 2016, ACM.
29. Rychly, M., et al. Scheduling decisions in stream processing on heterogeneous clusters. Complex, Intelligent and Software Intensive Systems, CISIS, 2014, IEEE, 614–619.
30. Satzger, B., Hummer, W., Leitner, P., Dustdar, S., Esc: Towards an elastic stream computing platform for the cloud. International Conference on Cloud Computing, CLOUD, 2011, IEEE, 348–355.
31. Schneider, S., Andrade, H., Gedik, B., Biem, A., Wu, K.-L., Elastic scaling of data parallel operators in stream processing. International Parallel & Distributed Processing Symposium, IPDPS, 2009, IEEE, 1–12.
32. Shukla, A., Simmhan, Y., Benchmarking distributed stream processing platforms for IoT applications. Technology Conference on Performance Evaluation and Benchmarking, TPCTC Lecture Notes in Computer Science, vol. 10080, 2016, Springer.
33. A. Shukla, Y. Simmhan, Model-driven scheduling for distributed stream processing systems, 2017, p. 54. ArXiv preprint arXiv:1702.01785.
34. A. Shukla, Y. Simmhan, Toward reliable and rapid elasticity for streaming dataflows on clouds, 2017. ArXiv preprint arXiv:1712.00605.
35. Simmhan, Y., Aman, S., Kumbhare, A., Liu, R., Stevens, S., Zhou, Q., Prasanna, V., Cloud-based software platform for data-driven smart grid management. IEEE/AIP Comput. Sci. Eng., CiSE, 2013, 1–11.
36. Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., et al. Storm@ twitter. International Conference on Management of Data, SIGMOD, 2014, ACM, 147–156.
37. Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., et al. Bigdatabench: A big data benchmark suite from internet services. High Performance Computer Architecture, HPCA, 2014, IEEE, 488–499.
38. Wu, Y., Tan, K.-L., Chronostream: Elastic stateful stream computation in the cloud. International Conference on Data Engineering, ICDE, 2015, IEEE, 723–734.
39. Xu, J., Chen, Z., Tang, J., Su, S., T-storm: Traffic-aware online scheduling in storm. International Conference on Distributed Computing Systems, ICDCS, 2014, IEEE, 535–544.
40. Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I., Discretized streams: An efficient and fault-tolerant model for stream processing on large clusters. Workshop on Hot Topics in Cloud Computing, HotCloud, 2012, USENIX.