Economical and fault-tolerant load balancing in distributed stream processing systems

IEICE Transactions on Information and Systems

Volumn E95-D, Issue 4, 2012, Pages 1062-1073

  Xiao, Fuyuan ^a   Kitasuka, Teruaki ^a   Aritsugi, Masayoshi ^a  

^a KUMAMOTO UNIVERSITY   (Japan)

Author keywords

Distributed data stream processing systems (DDSPSs); Economical operator distribution (EOD); Fault tolerance; Load balancing strategy

Indexed keywords

BINS; DATA COMMUNICATION SYSTEMS; DATA HANDLING; DISTRIBUTED PARAMETER CONTROL SYSTEMS; ELECTRIC POWER PLANT LOADS; FAULT TOLERANCE; QUALITY OF SERVICE; WIDE AREA NETWORKS;

DATA STREAM PROCESSING; DISTRIBUTED DATA STREAM PROCESSING; DISTRIBUTED STREAM PROCESSING; DYNAMIC OPERATORS; LOAD BALANCING STRATEGY; NETWORK RESOURCE; OPERATOR DISTRIBUTIONS; REPLICATION MECHANISM;

ELECTRIC LOAD SHEDDING;

EID: 84859329136     PISSN: 09168532     EISSN: 17451361     Source Type: Journal    
DOI: 10.1587/transinf.E95.D.1062     Document Type: Conference Paper

References (25)

1. D.J. Abadi, Y. Ahrnad, M. Balazinska, U. Cetiineie, M. Cherniack, J.H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S.B. Zdonik, "The design of the Borealis stream processing engine,' Proc. Second Biennial Conference on Innova tive Data Systems Research, CTDR, pp.277 289, Jan. 2005. Ava1iable at http://www.cidrdb.org/cidr2005/papers/P23.pdf
2. D.J. Abadi, D. Camey, U. cetiritemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik, "Aurora: a new model and architecture for data stream management," The VLDB Journal, vol.12, no.2, pp.120 139, 2003. DOT l0.1007/s00778 003 0095 z. (Pubitemid 37017924)
3. M. Balazinska, H. Balakrishnan, SR. Madden, and M. Stonebraker, "Fault tolerance in the borealis distributed stream processing system' ACM Trans. Database Syst., vol.33, no.1, pp.1 44, 2008. DOT 10.1145/1331904.1331907. (Pubitemid 351462733)
4. S. Chandrasekaran. O Cooper. K Deshpande. Mt Franklin. JM Hellerstein, W. Hong, S. Krishnarnurthy, S. Madden, V. Raman, F Reiss, and M. Shah, "TelegraphCQ: Continuous dataflow processing for an uncertain world," Proc. First Biennial Conference on Innovative Data Systems Research, CIDR, Jan. 2003. Ava1iable at http://www.cidrdb.org/cidr2003/prograra/p24.pdf
5. J.H. Hwang, Y. Xing, U. cetiisteinel, and SB. Zdonik, "A coop erative, self-conguring high-ava11ab11ity solution for stream processing," Proc. 23rd International Conference on Data Engineering, ICDE, pp.17185, 2007. DOl 10.1 109/ICDE.2007.3783.
6. flu M.A. Shah, J.M. Hellerstein, and E. Brewer, "Highly ava11able, fault-tolerant, parallel dataows," Proc. 2004 ACM SIGMOD international conference on Management of data, SIGMOD, New York, NY, USA, pp.827 838, ACM, 2004. DOl 10.1145/ I 00758. 1 0072.
7. N. Tatbul, Y. Ahmad, U. cetintenel, J.H. Hwang, Y Xing, and SB. Zdonik, "Load toanagement and high ava11ab11ity in the borealis dis tributed stream processing engine:' in GeoSensor Networks, ed. S. NiEel, A. Labrinidis, and A. Stetanidis, Lecture Notes in Computer Science, vol.4540, pp.85, Springer Berlin / Heidelberg, 2008. DOl 10.1007/978 3 540 79992.
8. M. Cherniack, H. Balakrishnan, M. Balazinska, D. Camey, U. cetintensel, Y. Xing, and SB. Zdonik, "Scalable distributed stream processing," Proc. First Biennial Conference on Innovative Data Systems Research, CIDR, 2003. Ava1iable at http://www.cidrdb.org/cidr2003/program/p23.pdf
9. M. Balazinska, H. Balakrishnan, and M. Stonebraker, "Contract based load management in federated distributed systems. /' Proceedings of the First Symposium on Networked Systems Design and Im plementation, NSDI, pp.197 210, 2004. http://www.usenix.org/exents/nsdi04/tech/balazinska.html
10. Y. Xing, S. Zdonik, and J.H. Hwang, "Dynamic load distribu tion in the borealis stream processor:' Proceedings of the 21st In temational Conference on Data Engineering, ICDE, Los Alami tos, CA, USA, pp.791 802, IEEE Computer Society, 2005. DOl 10.1 109/ICDE.2005.53. (Pubitemid 41731202)
11. Y. Xing, J.H. Hwang U. cetintemel and S. Zdonik "Providing res11iency to load variations in distributed stream processing:' Proc. 32nd international conference on Very large data bases VLDB pp.715 78VLDB Endowment 200. Ava1iable at http://www.vldb.org/cont/200/p775 xing.pdf
12. W. Lang, J.M. Patel, and J.F. Naughton, "On energy management, load balancing and replication:' SIGMOD Rec., vol.38, no.4, pp.35 42, Dec. 2009. DOl 10.1145/1815948.18 1595.
13. N. Tatbul, U. cetinteiel, and SB. Zdonik, "Staying FIT: Efcient load shedding techniques for distributed stream processing:' Proc. 33rd international conference on Very large data bases, VLDB, pp.159 170, 2007. Ava1iable at http://www.vldb.org/cont/2007/papers/research/pl59 tatbul.pdf
14. N. Tatbul, U. cetintemel, SB. Zdonik, M. Cherniack, and M. Stonebraker, "Load shedding in a data stream manager," Proc. 29th Intemational Conference on Very Large Data Bases, VLDB, pp.309 320, 2003. Ava1iable at http://www.vldb.org/cont/2003/papers/Sl0P03.pdf
15. N. Tatbul and S. Zdonik, "Dealing with overload in distributed stream processing systeiris:' Proc. 22nd International Confer ence on Data Engineering Workshops, ICDE Workshops, Los Alamitos, CA, USA, p.24, IEEE Computer Society, 200. DOl 10.11 09/ICDEW.200.45.
16. J.H. Hwang U. cetinteiel and SB. Zdonik "Fast and highly ava11able stream processing over wide area networks:' Proc. 24th International Conference on Data Engineering ICDE pp.804 813 2008 DOl 10.1 109/ICDE.2008.4497489
17. M. Castro and B. Liskov, "Practical byzantine fault tolerance:' Proc. Third Symposium on Operating Systems Design and Implementa tion, OSDI, Berkeley, CA, USA, pp.173 1 8, USENIX Association, 1999. http://portal.acm.org/ citation.cfmlid=2980.29824
18. F. Dabek, R. Cox, F. Kaashoek, and R. Morris. "Vivaldi: A decentralized network coordinate system[ SIGCOMM Com put. Commun. Rev., vol.34, no.4, pp.15 2, Aug. 2004. DOl 10.1145/1030194.1015471. (Pubitemid 40954866)
19. F. Xiao, K. Nagano, T. Itokawa, T. Kitasuka, and M. Aritsugi, "A self recovery technique for highly ava11able stream processing over local area networks.:' Proc. 2010 IEEE Region 10 Conference, TENCON, pp.2402411, Nov. 2010. DOl 10.1109/ TENCON.201 0.585904.
20. E.G. Coffman, Jr., M.R. Garey, and D.S. Johnson, Approximaon algorithms for bin packing: a survey, pp.493, PWS Publishing Co., Boston, MA, USA, 1997. http://portal.acm.org/citation.cfm7id=24 1938.241940
21. J. Hromkovic, Algorithmics for hard problems, Springer, 1998.
22. V.V. Vazirani, Approximation Algorithms, Springer, 2001.
23. The ns 3 network simulator. http://www.nsnam.org/
24. J.H. Hwang, U. cetintemel, and SB. Zdonik, "Fast and reliable stream processing over wide area networks," Proc. 23rd Interna tional Conterence on Data Engineering Workshop, ICDEW, pp.0114 013, 2007. DOl 10.11 09/ICDEW.2007.440 1047.
25. J.H. Hwang, M. Balazinska, A. Rasin, U. Cetintemel, M. Stonebraker, and S. Zdonik, "High ava11ab11ity algorithms for dis tributed stream processing:' Proc. 21st Intemational Conference on Data Engineering, ICDE, Los Alamitos, CA, USA, pp.779 790, IEEE Computer Society, 2005. DOl 10.1 109/ICDE.2005.72. (Pubitemid 41731201)