Maximum Sustainable throughput Prediction for Data Stream Processing over Public Clouds

Proceedings - 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2017

Volumn , Issue , 2017, Pages 504-513

  Imai, Shigeru ^a   Patterson, Stacy ^a   Varela, Carlos A ^a  

^a RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

Auto scaling; Cloud computing; Performance prediction; Resource management

Indexed keywords

BENCHMARKING; CLOUD COMPUTING; CLUSTER COMPUTING; COSTS; DATA COMMUNICATION SYSTEMS; DATA HANDLING; FORECASTING; GRID COMPUTING; SUSTAINABLE DEVELOPMENT; THROUGHPUT;

AUTO-SCALING; AVERAGE PREDICTION ERROR; DATA STREAM PROCESSING; PERFORMANCE PREDICTION; RESOURCE MANAGEMENT; RESOURCE USAGE PATTERNS; SERVICE LEVEL AGREEMENTS; STREAMING APPLICATIONS;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 85027442774     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CCGRID.2017.105     Document Type: Conference Paper

References (34)

1. S. Imai, R. Klockowski, and C. A. Varela, "Self-healing spatio-Temporal data streams using error signatures, " in IEEE 2nd International Conference on Big Data Science and Engineering, 2013, pp. 957-964.
2. M. Solaimani, M. Iftekhar, L. Khan, B. Thuraisingham, and J. B. Ingram, "Spark-based anomaly detection over multisource VMware performance data in real-Time, " in IEEE Symposium on Computational Intelligence in Cyber Security, 2014, pp. 1-8.
3. A. Biem, E. Bouillet, H. Feng, A. Ranganathan, A. Riabov, O. Verscheure, H. N. Koutsopoulos, M. Rahmani, and B. Gü, "Real-Time traffic information management using stream computing, " IEEE Data Engineering Bulletin, vol. 33, pp. 64-68, June 2010.
4. S. Imai, S. Patterson, and C. A. Varela, "Elastic virtual machine scheduling for continuous air traffic optimization, " in 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 2016, pp. 183-186.
5. C. Hochreiner, M. Vögler, S. Schulte, and S. Dustdar, in IEEE 9th International Conference on Cloud Computing.
6. C. Hochreiner, S. Schulte, S. Dustdar, and F. Lecue, "Elastic stream processing for distributed environments, " IEEE Internet Computing, vol. 19, no. 6, pp. 54-59, Nov 2015.
7. A. Shukla and Y. Simmhan, "Benchmarking distributed stream processing platforms for IoT applications, " arXiv preprint arXiv:1606.07621, 2016.
8. The Apache Software Foundation, "Apache Storm, " http://storm.apache.org/, Accessed: 2017-02-15.
9. A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, N. Bhagat, S. Mittal, and D. Ryaboy, "Storm@twitter, " in Proceedings of the 2014 SIGMOD International Conference on Management of Data, 2014, pp. 147-156.
10. The Apache Software Foundation, "Apache Flink, " http://spark.apache.org/, Accessed: 2017-02-15.
11. P. Carbone, S. Ewen, S. Haridi, A. Katsifodimos, V. Markl, and K. Tzoumas, "Apache Flink: Stream and batch processing in a single engine, " IEEE Data Engineering Bulletin, in the special issue on Next-gen Stream Processing, vol. 38, no. 4, Dec 2015.
12. The Apache Software Foundation, "Apache Samza, " http://samza.apache.org/, Accessed: 2017-02-15.
13. "Apache Spark, " http://spark.apache.org/, Accessed: 2017-02-15.
14. M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica, "Discretized streams: Fault-Tolerant streaming computation at scale, " in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, 2013, pp. 423-438.
15. T. Lorido-Botran, J. Miguel-Alonso, and J. A. Lozano, "A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments, " Journal of Grid Computing, vol. 12, no. 4, pp. 559-592, Dec 2014.
16. T. Li, J. Tang, and J. Xu, "A predictive scheduling framework for fast and distributed stream data processing, " in IEEE International Conference on Big Data, 2015, pp. 333-338.
17. T. Heinze, Z. Jerzak, G. Hackenbroich, and C. Fetzer, "Latency-Aware elastic scaling for distributed data stream processing systems, " in Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, 2014, pp. 13-22.
18. B. Lohrmann, P. Janacik, and O. Kao, "Elastic stream processing with latency guarantees, " in IEEE 35th International Conference on Distributed Computing Systems, 2015, pp. 399-410.
19. T. Z. J. Fu, J. Ding, R. T. B. Ma, M. Winslett, Y. Yang, and Z. Zhang, "DRS: Dynamic resource scheduling for realtime analytics over fast streams, " in IEEE 35th International Conference on Distributed Computing Systems, 2015, pp. 411-420.
20. A. Ishii and T. Suzumura, "Elastic stream computing with clouds, " in IEEE 4th International Conference on Cloud Computing, 2011, pp. 195-202.
21. Intel Corporation, "Storm benchmark, " https://github.com/intel-hadoop/storm-benchmark, Accessed: 2017-02-15.
22. B. Peng, M. Hosseini, Z. Hong, R. Farivar, and R. Campbell, "R-storm: Resource-Aware scheduling in storm, " in Proceedings of the 16th ACM Annual Middleware Conference, 2015, pp. 149-161.
23. L. Fischer and A. Bernstein, "Workload scheduling in distributed stream processors using graph partitioning, " in IEEE International Conference on Big Data, 2015, pp. 124-133.
24. B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom, "Models and issues in data stream systems, " in Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems. ACM, 2002, pp. 1-16.
25. S. Kulkarni, N. Bhagat, M. Fu, V. Kedigehalli, C. Kellogg, S. Mittal, J. M. Patel, K. Ramasamy, and S. Taneja, "Twitter Heron: Stream processing at scale, " in Proceedings of the ACM SIGMOD International Conference on Management of Data, 2015, pp. 239-250.
26. Microsoft, "Measuring maximum sustainable engine throughput, " https://msdn.microsoft.com/en-us/library/cc296884(v= bts.10).aspx, Accessed: 2017-02-15.
27. J. Kreps and L. Corp, "Kafka : A distributed messaging system for log processing, " ACM SIGMOD Workshop on Networking Meets Databases, p. 6, 2011.
28. O. Etzion and P. Niblett, Event processing in action. Manning Publications Co., 2010.
29. Yahoo! Inc., "Yahoo streaming benchmarks, " https://github. com/yahoo/streaming-benchmarks, Accessed: 2017-02-15.
30. S. Venkataraman, Z. Yang, M. Franklin, B. Recht, and I. Stoica, "Ernest: Efficient performance prediction for large-scale advanced analytics, " in Proceedings of the 13th Usenix Conference on Networked Systems Design and Implementation, 2016, pp. 363-378.
31. Mark Twain, "The project gutenberg ebook of the adventures of tom sawyer, " https://www.gutenberg.org/files/74/74-h/74-h.htm, Accessed: 2017-02-15.
32. G. M. Amdahl, "Validity of the single processor approach to achieving large scale computing capabilities, " in Proceedings of the Spring Joint Computer Conference, 1967, pp. 483-485.
33. M. Arlitt and T. Jin, "A workload characterization study of the 1998 world cup web site, " IEEE Network, vol. 14, no. 3, pp. 30-37, May 2000.
34. R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. De Rose, and R. Buyya, "Cloudsim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, " Software: Practice and Experience, vol. 41, no. 1, pp. 23-50, Jan 2011.