Moving big data to the cloud: An online cost-minimizing approach

IEEE Journal on Selected Areas in Communications

Volumn 31, Issue 12, 2013, Pages 2710-2721

  Zhang, Linquan ^a   Wu, Chuan ^b   Li, Zongpeng ^a   Guo, Chuanxiong ^c   Chen, Minghua ^d   Lau, Francis C M ^b  

^a UNIVERSITY OF CALGARY   (Canada)

^b UNIVERSITY OF HONG KONG   (Hong Kong)

^c MICROSOFT   (United States)

^d CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Big Data; Cloud Computing; Online Algorithms

Indexed keywords

BIG DATUM; COMPUTATION PARADIGMS; DATA AGGREGATION; GEOGRAPHICAL LOCATIONS; OFF-LINE ALGORITHM; ON-LINE ALGORITHMS; RESOURCE ACCESS; TRANSMITTING DATA;

ALGORITHMS; CLOUD COMPUTING; COSTS;

DATA HANDLING;

EID: 84890546248     PISSN: 07338716     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSAC.2013.131211     Document Type: Article

References (37)

1. Rackspace, http://www.rackspace.com/.
2. M. Armbrust, A. Fox, R. Grifth, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. P. A. Rabkin, I. Stoica, and M. Zaharia, "Above the Clouds: A Berkeley View of Cloud Computing," EECS, University of California, Berkeley, Tech. Rep., 2009.
3. S. Pandey, L. Wu, S. Guru, and R. Buyya, "A Particle Swarm Optimization (PSO)-based Heuristic for Scheduling Workflow Applications in Cloud Computing Environment," in Proc. IEEE AINA, 2010.
4. Human Genome Project, http://www.ornl.gov/hgmis/home.shtml.
5. Hadoop - Facebook, http://www.facebook.com/note.php?note id=1612 1578919.
6. Hadoop at Twitter, http://www.slideshare.net/kevinweil/hadoop-attwitter- hadoop-summit-201.
7. AWS Import/Export, http://aws.amazon.com/importexport/.
8. E. E. Schadt, M. D. Linderman, J. Sorenson, L. Lee, and G. P. Nolan, "Computational Solutions to Large-scale Data Management and Analysis," Nat Rev Genet, vol. 11, no. 9, pp. 647-657, 09 2010.
9. Moving an Elephant: Large Scale Hadoop Data Migration at Facebook, http://www.facebook.com/notes/paul-yang/moving-an-elephant-largescale- hadoop-data-migration-at-facebook/10150246275318920.
10. R. J. Brunner, S. G. Djorgovski, T. A. Prince, and A. S. Szalay, "Handbook of Massive Data Sets," J. Abello, P. M. Pardalos, and M. G. C. Resende, Eds. Norwell, MA, USA: Kluwer Academic Publishers, 2002, ch. Massive Datasets in Astronomy, pp. 931-979.
11. M. Cardosa, C. Wang, A. Nangia, A. Chandra, and J. Weissman, "Exploring MapReduce Efficiency with Highly-Ditributed Data," in Proc. ACM MapReduce, 2011.
12. Amazon Elastic MapReduce, http://aws.amazon.com/elasticmapreduce/.
13. M. Hajjat, X. Sun, Y. E. Sung, D. Maltz, and S. Rao, "Cloudward Bound: Planning for Beneficial Migration of Enterprise Applications to the Cloud," in Proc. ACM SIGCOMM, August 2010.
14. X. Cheng and J. Liu, "Load-Balanced Migration of Social Media to Content Clouds," in Proc. ACM NOSSDAV, June 2011.
15. Y. Wu, C. Wu, B. Li, L. Zhang, Z. Li, and F. Lau, "Scaling Social Media Applications into Geo-Distributed Clouds," in Proc. IEEE INFOCOM, Mar. 2012.
16. B. Cho and I. Gupta, "New Algorithms for Planning Bulk Transfer via Internet and Shipping Networks," in Proc. IEEE ICDCS, 2010.
17. B. Cho and I. Gupta, "Budget-Constrained Bulk Data Transfer via Internet and Shipping Networks," in Proc. ACM ICAC, 2011.
18. M. Lin, A. Wierman, L. L. Andrew, and E. Thereska, "Dynamic Rightsizing for Power-proportional Data Centers," in Proc. IEEE INFOCOM, April 2011.
19. V. Mathew, R. Sitaraman, and P. Shenoy, "Energy-aware Load Balancing in Content Delivery Networks," in Proc. IEEE INFOCOM, Mar. 2012.
20. T. Lu and M. Chen, "Simple and Effective Dynamic Provisioning for Power-Proportional Data Centers," in Proc. IEEE CISS, Mar. 2012.
21. M. Lin, Z. Liu, A. Wierman, and L. Andrew, "Online Algorithms for Geographical Load Balancing," in Proc. IEEE IGCC, 2012.
22. AWS Direct Connect, http://aws.amazon.com/directconnect/.
23. Windows Azure Virtual Network, http://www.windowsazure.com/enus/ home/features/networking/.
24. GoGrid, http://www.gogrid.com/.
25. D. Logothetis, C. Olston, B. Reed, K. C. Webb, and K. Yocum, "Stateful Bulk Processing for Incremental Analytics," in Proc. ACM SoCC, 2010.
26. A. Borodin and R. El-Yaniv, Online Computation and Competitive Analysis. Cambridge University Press, 1998, vol. 2.
27. K. M. Anstreicher, "Linear Programming in O([n3/ ln n]L) Operations," SIAM Journal on Optimization, vol. 9, no. 4, pp. 803-812, 1999.
28. G. E. P. Box, G. M. Jenkins, and G. C. Reinsel, Time Series Analysis: Forecasting and Control, 4th ed. Wiley, 2008.
29. Square Kilometre Array, http://www.skatelescope.org/.
30. Amazon Elastic Compute Cloud, http://aws.amazon.com/ec2/.
31. Linode, https://www.linode.com/speedtest/.
32. DigitalOcean, https://www.digitalocean.com/.
33. Sky Quality Group, The Instituto de Astrofísica de Canarias (IAC), http://www.iac.es/proyecto/site-testing/index.php.
34. H. V. Ramío, C. M. noz Tüńon, A. M. Varela, and J. J. Fuensalida, "Complementary note: DIMM (ELT-DS) calibration," The Instituto de Astrofísica de Canarias (IAC), Tech. Rep., 2008.
35. A. Qureshi, "Power-Demand Routing in Massive Geo-Distributed Systems," PhD Thesis submitted to MIT, 2010.
36. Amazon Simple Storage Service, http://aws.amazon.com/s3/.
37. TERAPIX, http://terapix.iap.fr/. Canada