Online parameter optimization for elastic data stream processing

ACM SoCC 2015 - Proceedings of the 6th ACM Symposium on Cloud Computing

Volumn , Issue , 2015, Pages 276-287

  Heinze, Thomas ^a   Roediger, Lars ^a   Meister, Andreas ^b   Ji, Yuanzhen ^a   Jerzak, Zbigniew ^a   Fetzer, Christof ^c  

^a SAP RESEARCH   (Germany)

^b OTTO VON GUERICKE UNIVERSITY   (Germany)

^c DRESDEN UNIVERSITY OF TECHNOLOGY   (Germany)

Author keywords

Distributed data stream processing; Elasticity; Load balancing; Parameter optimization

Indexed keywords

CLOUD COMPUTING; COSTS; DATA COMMUNICATION SYSTEMS; ECONOMIC AND SOCIAL EFFECTS; ELASTICITY; NETWORK MANAGEMENT; QUALITY OF SERVICE; RESOURCE ALLOCATION;

DATA STREAM PROCESSING; DISTRIBUTED DATA STREAM PROCESSING; ONLINE PARAMETER OPTIMIZATIONS; OVER PROVISIONING; PARAMETER OPTIMIZATION; UNDERLYING SYSTEMS; WORKLOAD CHARACTERISTICS; WORKLOAD PATTERNS;

DATA HANDLING;

EID: 84959036559     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2806777.2806847     Document Type: Conference Paper

References (28)

1. D. J. Abadi, Y. Ahmad, M. Balazinska, U. Cetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, et al. The Design of the Borealis Stream Processing Engine. In Proceedings of the Second Biennial Conference on Innovative Data Systems Research, CIDR 2005, pages 277-289, 2005.
2. Amazon. Amazon Auto Scaling. http://aws.amazon.com/autoscaling/.
3. Amazon. Amazon Kinesis. http://aws.amazon.com/de/kinesis/.
4. A. Bifet and R. Gavalda. Learning from time-changing data with adaptive windowing. In Proceedings of the Seventh SIAM International Conference on Data Mining, SDM 2007, pages 443-448, 2007.
5. E. G. Coffman Jr, M. R. Garey, and D. S. Johnson. Approximation algorithms for bin packing: A survey. In Approximation algorithms for NP-hard problems, pages 46-93. PWS Publishing Co., 1996.
6. C. Delimitrou and C. Kozyrakis. Quasar: resource-efficient and qosaware cluster management. In Proceedings of the 19th international conference on Architectural support for programming languages and operating systems, ASPLOS 2014, pages 127-144. ACM, 2014.
7. M. Ead, H. Herodotou, A. Aboulnaga, and S. Babu. PStorM: Profile Storage and Matching for Feedback-Based Tuning of MapReduce Jobs. In Proceedings of the 17th International Conference on Extending Database Technology, EDBT 2014, pages 1-12, 2014.
8. R. C. Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the SIGMOD International Conference on Management of Data, SIGMOD 2013, pages 725-736. ACM, 2013.
9. B. Gedik, S. Schneider, M. Hirzel, and K.-L. Wu. Elastic scaling for data stream processing. IEEE Transactions on Parallel and Distributed Systems (TPDS), 25(6):1447-1463, 2014.
10. R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella. Multi-resource packing for cluster schedulers. In Proceedings of the 2014 ACM conference on SIGCOMM, pages 455-466. ACM, 2014.
11. V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez. Streamcloud: An elastic and scalable data streaming system. IEEE Transactions on Parallel and Distributed Systems (TPDS), 23(12):2351-2365, 2012.
12. T. Heinze, Z. Jerzak, G. Hackenbroich, and C. Fetzer. Latency-aware elastic scaling for distributed data stream processing systems. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, DEBS 2014, pages 13-22. ACM, 2014.
13. T. Heinze, V. Pappalardo, Z. Jerzak, and C. Fetzer. Auto-scaling techniques for elastic data stream processing. In Workshops Proceedings of the 30th International Conference on Data Engineering Workshops, ICDEW 2014, pages 296-302. IEEE, 2014.
14. N. R. Herbst, S. Kounev, and R. Reussner. Elasticity in cloud computing: What it is, and what it is not. In Proceedings of the 10th International Conference on Autonomic Computing, ICAC 2013, pages 23-27, 2013.
15. H. Herodotou and S. Babu. Profiling, what-if analysis, and costbased optimization of mapreduce programs. Proceedings of the VLDB Endowment, 4(11):1111-1122, 2011.
16. E. Kalyvianaki, W. Wiesemann, Q. H. Vu, D. Kuhn, and P. Pietzuch. SQPR: Stream query planning with reuse. In Proceedings of the 27th International Conference on Data Engineering, ICDE 2011, pages 840-851. IEEE, 2011.
17. U. Lampe, R. Hans, M. Seliger, and M. Pauly. Pricing in infrastructure clouds-an analytical and empirical examination. In Proceedings of the 20th Americas Conference on Information Systems, AMCIS 2014, 2014.
18. T. Lorido-Botŕan, J. Miguel-Alonso, and J. A. Lozano. Auto-scaling techniques for elastic applications in cloud environments. Department of Computer Architecture and Technology, University of Basque Country, Tech. Rep. EHU-KAT-IK-09, 12, 2012.
19. S. Martello and P. Toth. Algorithms for knapsack problems. Surveys in Combinatorial Optimization, 31:213-258, 1987.
20. X. Meng, V. Pappas, and L. Zhang. Improving the scalability of data center networks with traffic-aware virtual machine placement. In Proceedings of 2010 IEEE INFOCOM, pages 1-9. IEEE, 2010.
21. C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, and M. A. Kozuch. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC 2012, page 7. ACM, 2012.
22. S. Schneider, H. Andrade, B. Gedik, A. Biem, and K.-L. Wu. Elastic scaling of data parallel operators in stream processing. In IPDPS 2009: Proceedings of the 23rd IEEE International Symposium on Parallel & Distributed Processing, IPDPS 2009, pages 1-12. IEEE, 2009.
23. Z. Shen, S. Subbiah, X. Gu, and J.Wilkes. Cloudscale: elastic resource scaling for multi-tenant cloud systems. In Proceedings of the second ACM Annual Symposium on Cloud Computing, SoCC 2011, pages 1-14. ACM, 2011.
24. W. Song, Z. Xiao, Q. Chen, and H. Luo. Adaptive resource provisioning for the cloud using online bin packing. IEEE Transactions on Computers, 63(11):2647-2660, 2014.
25. E. Wu, Y. Diao, and S. Rizvi. High-performance complex event processing over streams. In Proceedings of the SIGMOD International Conference on Management of Data, SIGMOD 2006, pages 407-418. ACM, 2006.
26. Y. Xing, S. Zdonik, and J.-H. Hwang. Dynamic load distribution in the borealis stream processor. In Proceedings of the 21st International Conference on Data Engineering, ICDE 2005, pages 791-802. IEEE, 2005.
27. T. Ye and S. Kalyanaraman. A recursive random search algorithm for large-scale network parameter configuration. In Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, SIGMETRICS 2003, pages 196-205. ACM, 2003.
28. H. Ziekow and Z. Jerzak. The DEBS 2014 Grand Challenge. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, DEBS 2014, pages 266-269. ACM, 2014.