A General Communication Cost Optimization Framework for Big Data Stream Processing in Geo-Distributed Data Centers

IEEE Transactions on Computers

Volumn 65, Issue 1, 2016, Pages 19-29

  Gu, Lin ^a   Zeng, Deze ^c   Guo, Song ^b   Xiang, Yong ^d   Hu, Jiankun ^e  

^a HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

^b UNIVERSITY OF AIZU   (Japan)

^c CHINA UNIVERSITY OF GEOSCIENCES   (China)

^d DEAKIN UNIVERSITY   (Australia)

^e UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

big data; geo distributed data centers; network cost minimization; stream processing; VM placement

Indexed keywords

AIRCRAFT ENGINE MANUFACTURE; COSTS; DATA COMMUNICATION SYSTEMS; DATA HANDLING; DIGITAL STORAGE; INTEGER PROGRAMMING; INTERNET; INTERNET SERVICE PROVIDERS; NETWORK MANAGEMENT; OPTIMIZATION; SEMANTICS;

DISTRIBUTED DATA; MIXED-INTEGER LINEAR PROGRAMMING; NETWORK COSTS; OPERATIONAL EXPENDITURES; QUALITY OF INFORMATION; SERVICE LEVEL AGREEMENTS; STREAM PROCESSING; VM PLACEMENTS;

BIG DATA;

EID: 84961753660     PISSN: 00189340     EISSN: None     Source Type: Journal    
DOI: 10.1109/TC.2015.2417566     Document Type: Article

References (28)

1. G. Lee, J. Lin, C. Liu, A. Lorek, and D. Ryaboy, "The unified logging infrastructure for data analytics at Twitter," Proc. VLDB Endowment, vol. 5, no. 12, pp. 1771-1780, 2012.
2. G. Mishne, J. Dalton, Z. Li, A. Sharma, and J. Lin, "Fast data in the era of big data: Twitter's real-time related query suggestion architecture," in Proc. Int. Conf. Manage. Data, 2013, pp. 1147-1158.
3. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proc. 9th USENIX Conf. Netw. Syst. Des. Implementation, 2012, p. 2.
4. Z. Zhang, M. Zhang, A. G. Greenberg, Y. C. Hu, R. Mahajan, and B. Christian, "Optimizing cost and performance in online service provider networks," in Proc. USENIX Netw. Syst. Des. Implementation, 2010, pp. 33-48.
5. P. Bodk, I. Menache, M. Chowdhury, P. Mani, D. A. Maltz, and I. Stoica, "Surviving failures in bandwidth-constrained datacenters," in Proc. Conf. Appl., Technol., Archit. Protocols Comput. Commun., 2012, pp. 431-442.
6. K. Y. Chen, Y. Xu, K. Xi, and H. Chao, "Intelligent virtual machine placement for cost efficiency in geo-distributed cloud systems," in Proc. IEEE Int. Conf. Commun., 2013, pp. 3498-3503.
7. A. Greenberg, J. Hamilton, D. A. Maltz, and P. Patel,"The cost of a cloud: Research problems in data center networks," SIGCOMM Comput. Commun. Rev., vol. 39, no. 1, pp. 68-73, 2008.
8. Y. Chen, S. Jain, V. Adhikari, Z.-L. Zhang, and K. Xu, "A first look at inter-datacenter traffic characteristics via Yahoo! datasets," in Proc. Int. Conf. Comput. Commun., 2011, pp. 1620-1628.
9. M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Cetintemel, Y. Xing, and S. B. Zdonik, "Scalable distributed stream processing," in Proc. 2nd Biennial Conf. Innovative Data Syst. Res., 2003, vol. 3, pp. 257-268.
10. L. Tian and K. M. Chandy, "Resource allocation in streaming environments," in Proc. 7th Int. Conf. Grid Comput., 2006, pp. 270-277.
11. J. Jiang, T. Lan, S. Ha, M. Chen, and M. Chiang, "Joint VM placement and routing for data center traffic engineering," in Proc. Int. Conf. Comput. Commun., 2012, pp. 2876-2880.
12. K. You, B. Tang, and F. Ding, "Near-optimal virtual machine placement with product traffic pattern in data centers," in Proc. IEEE Int. Conf. Commun., 2013, pp. 3705-3709.
13. H. Ballani, K. Jang, T. Karagiannis, C. Kim, D. Gunawardena, and G. OShea, "Chatty tenants and the cloud network sharing problem," in Proc. 10th Conf. Netw. Syst. Des. Implementation, 2013, pp. 171-184.
14. W. Fang, X. Liang, S. Li, L. Chiaraviglio, and N. Xiong, "VMPlanner: Optimizing virtual machine placement and traffic flow routing to reduce network power costs in cloud data centers," Comput. Netw., vol. 57, no. 1, pp. 179-196, 2013.
15. X. Li, J. Wu, S. Tang, and S. Lu, "Let's stay together: Towards traffic aware virtual machine placement in data centers," in Proc. IEEE Int. Conf. Comput. Commun., 2014, pp. 1-9.
16. L. Wang, F. Zhang, J. ArjonaAroca, A. Vasilakos, K. Zheng, C. Hou, D. Li, and Z. Liu, "GreenDCN: A general framework for achieving energy efficiency in data center networks," J. Select. Areas Commun., vol. 32, no. 1, pp. 4-15, 2014.
17. C.-G. Lee and Z. Ma, "The generalized quadratic assignment problem," Dept. Mech. Ind. Eng., Univ. Toronto, Canada, Res. Rep., 2004.
18. H. C. Zhao, C. H. Xia, Z. Liu, and D. Towsley, "A unified modeling framework for distributed resource allocation of general fork and join processing networks," in Proc. Int. Conf. Meas. Modeling Comput. Syst., 2010, pp. 299-310.
19. K. LaCurts, S. Deng, A. Goyal, and H. Balakrishnan, "Choreo: Network-aware task placement for cloud applications," in Proc. Internet Meas. Conf., 2013, pp. 191-204.
20. B. Chinoy and H.-W. Braun, "The National Science Foundation network," Tech. Rep. SDSC Report GA-A21029, San Diego Supercomputer Center, Sep. 1992.
21. N. Laoutaris, M. Sirivianos, X. Yang, and P. Rodriguez, "Interdatacenter bulk transfers with netstitcher," in Proc. Conf. Special Interest Group Data Commun., 2011, pp. 74-85.
22. Y. Feng, B. Li, and B. Li, "Postcard: Minimizing costs on interdatacenter traffic with store-and-forward," in Proc. Int. Conf. Distrib. Comput. Syst. Workshops, 2012, pp. 43-50.
23. L. Liu, H. Wang, X. Liu, X. Jin, W. B. He, Q. B. Wang, and Y. Chen, "GreenCloud: A new architecture for green data center," in Proc. 6th Int. Conf. Ind. Session Auton. Comput. Commun. Ind. Session, 2009, pp. 29-38.
24. R. Cohen, L. Lewin-Eytan, J. Naor, and D. Raz, "Almost optimal virtual machine placement for traffic intense data centers," in Proc. Int. Conf. Comput. Commun., 2013, pp. 355-359.
25. X. Meng, V. Pappas, and L. Zhang, "Improving the scalability of data center networks with traffic-aware virtual machine placement," in Proc. IEEE Int. Conf. Comput. Commun., 2010, pp. 1-9.
26. Y. Ajiro and A. Tanaka, "Improving packing algorithms for server consolidation," in Proc. Int. Comput, Meas. Group Conf., 2007, pp. 399-406.
27. B. He, M. Yang, Z. Guo, R. Chen, B. Su, W. Lin, and L. Zhou, "Comet: Batched stream processing for data intensive distributed computing," in Proc. ACM 1st Symp. Cloud Comput., 2010, pp. 63-74.
28. G. T. Lakshmanan, Y. Li, and R. Strom, "Placement of replicated tasks for distributed stream processing systems," in Proc. ACM Int. Conf. Distrib. Event-Based Syst., 2010, pp. 128-139.