Self-adaptive processing graph with operator fission for elastic stream processing

Journal of Systems and Software

Volumn 127, Issue , 2017, Pages 205-216

  Hidalgo, Nicolas ^a   Wladdimiro, Daniel ^a   Rosas, Erika ^a  

^a UNIVERSIDAD DIEGO PORTALES   (Chile)

Author keywords

Elastic processing; S4; Scalable processing; Self adaptable graph; Stream processing

Indexed keywords

DATA STREAMS; FLOW GRAPHS; GRAPH ALGORITHMS; GRAPHIC METHODS;

ADAPTIVE PROCESSING; DISTRIBUTED PROCESSING; EXTRACTING INFORMATION; IMPROVE PERFORMANCE; PROCESSING TOOLS; RESOURCE UTILIZATIONS; SELF-ADAPTABLE; STREAM PROCESSING;

TOPOLOGY;

EID: 85006516228     PISSN: 01641212     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jss.2016.06.010     Document Type: Article

References (42)

1. Appel, S., Frischbier, S., Freudenreich, T., Buchmann, A.P., Eventlets: Components for the integration of event streams with SOA. Proceedings of the 2012 Fifth IEEE International Conference on Service-Oriented Computing and Applications (SOCA), Taipei, Taiwan, 2012, IEEE, 1–9.
2. Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A.S., Ryvkina, E., Stonebraker, M., Tibbetts, R., I road: A stream data management benchmark. Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30, 2004, VLDB Endowment, 480–491.
3. Ching, W.-K., Ng, M.K., Markov Chains: Models, Algorithms and Applications, 2006, Springer US, New York.
4. Fernandez, R.C., Weidlich, M., Pietzuch, P., Gal, A., Scalable stateful stream processing for smart grids. Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems (DEBS), 2014, ACM, 276–281.
5. Gedik, B., Partitioning functions for stateful data parallelism in stream processing. Very Large Data Bases (VLDB) J. 23:4 (2014), 517–539.
6. Gedik, B., Schneider, S., Hirzel, M., Wu, K.-L., Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25:6 (2014), 1447–1463.
7. Gong, Z., Gu, X., Wilkes, J., Press: Predictive elastic resource scaling for cloud systems. Proceedings of the 2010 International Conference on Network and Service Management (CNSM), 2010, 9–16.
8. Gulisano, V., Jiménez-Peris, R., Patiño-Martínez, M., Valduriez, P., Streamcloud: A large scale data streaming system. Proceedings of the 2010 International Conference on Distributed Computing Systems (ICDCS), Genova, Italy, 2010, IEEE Computer Society, 126–137.
9. Heinze, T., Jerzak, Z., Hackenbroich, G., Fetzer, C., Latency-aware elastic scaling for distributed data stream processing systems. Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems (DEBS), 2014, ACM, 13–22.
10. Heinze, T., Ji, Y., Pan, Y., Grueneberger, F.J., Jerzak, Z., Fetzer, C., Elastic complex event processing under varying query load. Proceedings of the First International Workshop on Big Dynamic Distributed Data (BD3), 2013, CEUR-WS, 25–30.
11. Heinze, T., Pappalardo, V., Jerzak, Z., Fetzer, C., Auto-scaling techniques for elastic data stream processing. Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems (DEBS), 2014, ACM, 318–321.
12. Heinze, T., Roediger, L., Meister, A., Ji, Y., Jerzak, Z., Fetzer, C., Online parameter optimization for elastic data stream processing. Proceedings of the 6th ACM Symposium on Cloud Computing (SoCC), 2015, ACM, 276–287.
13. Heinze, T., Zia, M., Krahn, R., Jerzak, Z., Fetzer, C., An adaptive replication scheme for elastic data stream processing systems. Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS), 2015, ACM, 150–161.
14. Hidalgo, J.M.G., Almeida, T.A., Yamakami, A., On the validity of a new SMS spam collection. Proceedings of the 11th International Conference on Machine Learning and Applications (ICMLA), FL, USA, 2012, IEEE, 240–245.
15. Hirzel, M., Soulé, R., Schneider, S., Gedik, B., Grimm, R., A catalog of stream processing optimizations. ACM Comput. Surv. 46:4 (2014), 46:1–46:34.
16. Hunt, P., Konar, M., Junqueira, F.P., Reed, B., Zookeeper: Wait-free coordination for internet-scale systems. Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, 2010, USENIX Association 11–11.
17. Kephart, J.O., Chess, D.M., The vision of autonomic computing. Computer 36:1 (2003), 41–50.
18. Leskovec, J., Backstrom, L., Kleinberg, J.M., Meme-tracking and the dynamics of the news cycle. Proceedings of the15th ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), Paris, France, 2009, 497–506.
19. Lin, J., Dyer, C., Data-Intensive Text Processing with MapReduce, 2010, Morgan and Claypool Publishers.
20. Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A., A review of auto-scaling techniques for elastic applications in cloud environments. J. Grid Comput. 12:4 (2014), 559–592.
21. Madsen, K.G.S., Thyssen, P., Zhou, Y., Integrating fault-tolerance and elasticity in a distributed data stream processing system. Proceedings of the 26th International Conference on Scientific and Statistical Database Management (SSDBM), 2014, ACM, 48:1–48:4.
22. Madsen, K.G.S., Zhou, Y., Dynamic resource management in a massively parallel stream processing engine. Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (CIKM), 2015, ACM, 13–22.
23. Mone, G., Beyond hadoop. Commun. ACM 56:1 (2013), 22–24.
24. Neumeyer, L., Robbins, B., Nair, A., Kesari, A., S4: distributed stream computing platform. 2010 IEEE International Conference on Data Mining Workshops (ICDMW), Sydney, Australia, 2010, IEEE, 170–177.
25. Nikravesh, A., Ajila, S., Lung, C.-H., Cloud resource auto-scaling system based on hidden markov model (hmm). Proceedings of the 2014 IEEE International Conference on Semantic Computing (ICSC), 2014, IEEE, 124–127.
26. Oberhelman, D.D., Coming to terms with web 2.0. Ref. Rev. 21:7 (2007), 5–6.
27. Papoulis, A., Probability, Random Variables, and Stochastic Processes, 1984, Mc-Graw Hill, New York.
28. Pollner, N., Steudtner, C., Meyer-Wegener, K., Operator fission for load balancing in distributed heterogeneous data stream processing systems. Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS), 2015, ACM, 332–335.
29. Rivetti, N., Querzoni, L., Anceaume, E., Busnel, Y., Sericola, B., Efficient key grouping for near-optimal load balancing in stream processing systems. Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS), 2015, ACM, 80–91.
30. Satzger, B., Hummer, W., Leitner, P., Dustdar, S., Esc: Towards an elastic stream computing platform for the cloud. Proceedings of the 2011 IEEE International Conference on Cloud Computing (CLOUD), 2011, IEEE, 348–355.
31. Schneider, S., Hirzel, M., Gedik, B., Wu, K.-L., Auto-parallelizing stateful distributed streaming applications. Proceedings of the 21st International Conference on Parallel Architectures and Compilation Techniques (PACT), 2012, ACM, 53–64.
32. Shah, M., Hellerstein, J., Chandrasekaran, S., Franklin, M., Flux: an adaptive partitioning operator for continuous query systems. Proceedings of the IEEE 19th International Conference on Data Engineering (ICDE), 2003, IEEE, 25–36.
33. Tagg, C., A corpus linguistics study of SMS text messaging, 2009, The University of Birmingham Ph.D. thesis.
34. Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., Kappas, A., Sentiment in short strength detection informal text. J. Am. Soc. Inf. Sci. Technol. 61:12 (2010), 2544–2558.
35. van der Veen, J., Van Der Waaij, B., Lazovik, E., Wijbrandi, W., Meijer, R., Dynamically scaling apache storm for the analysis of streaming data. Proceedings of the 2015 IEEE First International Conference on Big Data Computing Service and Applications (BigDataService), 2015, IEEE, 154–161.
36. Wu, X., Liu, Y., Optimization of load adaptive distributed stream processing services. Proceedings of the 2014 IEEE International Conference on Services Computing (SCC), 2014, IEEE Computer Society Press, 504–511.
37. Xu, J., Chen, Z., Tang, J., Su, S., T-storm: Traffic-aware online scheduling in storm. Proceedings of the 2014 IEEE 34th International Conference on Distributed Computing Systems (ICDCS), 2014, IEEE, 535–544.
38. Yu, G., Hu, J., Zhang, C., Zhuang, L., Song, J., Short-term traffic flow forecasting based on markov chain model. Proceedings of the IEEE Intelligent Vehicles Symposium, 2003, IEEE, 208–212.
39. Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I., Discretized streams: Fault-tolerant streaming computation at scale. Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (SOSP), 2013, ACM, 423–438.
40. Zeitler, E., Risch, T., Scalable splitting of massive data streams. Database Systems for Advanced Applications Lecture Notes in Computer Science, Vol. 5982, 2010, Springer Berlin Heidelberg, 184–198.
41. Zhou, Y., Aberer, K., Tan, K.-L., Toward massive query optimization in large-scale distributed stream systems. Issarny, V., Schantz, R., (eds.) Middleware 2008 Lecture Notes in Computer Science, Vol. 5346, 2008, Springer, 326–345.
42. Zhou, Y., Ooi, B., Tan, K.-L., Wu, J., Efficient dynamic operator placement in a locally distributed continuous query system. On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE Lecture Notes in Computer Science, Vol. 4275, 2006, Springer, 54–71.