Dynamic resource management in a Massively Parallel Stream Processing Engine

International Conference on Information and Knowledge Management, Proceedings

Volumn 19-23-Oct-2015, Issue , 2015, Pages 13-22

  Madsen, Kasper Grud Skat ^a   Zhou, Yongluan ^a  

^a UNIVERSITY OF SOUTHERN DENMARK   (Denmark)

Author keywords

Elasticity; Fault tolerance; Resource management

Indexed keywords

DYNAMIC LOADS; EFFICIENCY; ELASTICITY; ENGINES; FAULT TOLERANCE; KNOWLEDGE MANAGEMENT; NATURAL RESOURCES MANAGEMENT; RESOURCE ALLOCATION; WASTE MANAGEMENT;

ALLOCATION PROBLEMS; DYNAMIC RESOURCE MANAGEMENT; DYNAMIC RESOURCES; FAULT TOLERANCE MECHANISMS; LARGE-SCALE CLUSTERS; MASSIVELY PARALLELS; RESOURCE MANAGEMENT; WASTE OF RESOURCES;

INFORMATION MANAGEMENT;

EID: 84958257272     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2806416.2806449     Document Type: Conference Paper

References (24)

1. T. Akidau, A. Balikov, Bekiroʇlu, et al. Millwheel: fault-tolerant stream processing at internet scale. VLDB, 2013.
2. D. Alves, P. Bizarro, and P. Marques. Flood: elastic streaming mapreduce. In DEBS '10, 2010.
3. M. Balazinska, H. Balakrishnan, S. R. Madden, and M. Stonebraker. Fault-tolerance in the borealis distributed stream processing system. ACM Trans. Database Syst., 33(1), Mar. 2008.
4. R. Castro Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Integrating scale out and fault tolerance in stream processing using operator state management. In SIGMOD, 2013.
5. B. Gedik, S. Schneider, M. Hirzel, and K.-L. Wu. Elastic scaling for data stream processing. TPDS, 2013.
6. Y. Gu et al. An empirical study of high availability in stream processing systems. In Middleware, 2009.
7. V. Gulisano, R. Jimenez-Peris, M. Patino-Martinez, C. Soriente, and P. Valduriez. Streamcloud: An elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst., 2012.
8. P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. Zookeeper: wait-free coordination for internet-scale systems. In USENIXATC'10, 2010.
9. J.-H. Hwang, M. Balazinska, A. Rasin, U. Cetintemel, M. Stonebraker, and S. Zdonik. High-availability algorithms for distributed stream processing. In ICDE '05, 2005.
10. J. hyon Hwang, Y. Xing, and S. Zdonik. A cooperative, self-configuring high-availability solution for stream processing. In In ICDE, 2007.
11. W. Lam, L. Liu, S. Prasad, A. Rajaraman, Z. Vacheri, and A. Doan. Muppet: Mapreduce-style processing of fast data. PVLDB, 5(12), 2012.
12. J. Li, K. Tufte, V. Shkapenyuk, V. Papadimos, T. Johnson, and D. Maier. Out-of-order processing: A new architecture for high-performance stream systems. Proc. VLDB Endow., 1(1), Aug. 2008.
13. L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream computing platform. In ICDMW '10, 2010.
14. B. Satzger, W. Hummer, P. Leitner, and S. Dustdar. Esc: Towards an Elastic Stream Computing Platform for the Cloud. In CLOUD'11, 2011.
15. S. Schneider, H. Andrade, B. Gedik, A. Biem, and K.-L. Wu. Elastic scaling of data parallel operators in stream processing. In IPDPS '09, 2009.
16. Z. Sebepou and K. Magoutis. Cec: Continuous eventual checkpointing for data stream processing operators. In DSN, 2011.
17. M. A. Shah, J. M. Hellerstein, and E. Brewer. Highly available, fault-tolerant, parallel dataflows. In SIGMOD, 2004.
18. M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin. Flux: An adaptive partitioning operator for continuous query systems. In ICDE, 2003.
19. A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, N. Bhagat, S. Mittal, and D. Ryaboy. Storm@twitter. In SIGMOD '14, 2014.
20. Y. Xing, S. Zdonik, and J.-H. Hwang. Dynamic load distribution in the borealis stream processor. In ICDE, 2005.
21. M. Yagiura, S. Iwasaki, T. Ibaraki, and F. Glover. A very large-scale neighborhood search algorithm for the multi-resource generalized assignment problem. Discrete Optimization, 1(1), 2004.
22. M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica. Discretized streams: An efficient and fault-tolerant model for stream processing on large clusters. In HotCloud, 2012.
23. Y. Zhou, K. Aberer, and K.-L. Tan. Toward massive query optimization in large-scale distributed stream systems. In Middleware, 2008.
24. Y. Zhou, B. C. Ooi, K. Tan, and J. Wu. Efficient dynamic operator placement in a locally distributed continuous query system. In OTM, 2006.