An algorithm for cost-effectively storing scientific datasets with multiple service providers in the cloud

Proceedings - IEEE 9th International Conference on e-Science, e-Science 2013

Volumn , Issue , 2013, Pages 285-292

  Yuan, Dong ^a   Liu, Xiao ^c   Cui, Lizhen ^b   Zhang, Tiantian ^b   Li, Wenhao ^a   Cao, Dahai ^a   Yang, Yun ^a,d  

^a SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

^b SHANDONG UNIVERSITY   (China)

^c EAST CHINA NORMAL UNIVERSITY   (China)

^d ANHUI UNIVERSITY   (China)

Author keywords

Cloud computing; Datasets storage; Scientific application

Indexed keywords

ALGORITHMS; BANDWIDTH; CLOUD COMPUTING; COMPUTER SIMULATION; DISTRIBUTED DATABASE SYSTEMS; ECONOMICS; INVESTMENTS;

APPLICATION COST; BANDWIDTH RESOURCE; CLOUD SERVICE PROVIDERS; DATA-INTENSIVE APPLICATION; INFRASTRUCTURE INVESTMENT; MULTIPLE SERVICES; NOVEL ALGORITHM; SCIENTIFIC APPLICATIONS;

COSTS;

EID: 84893504807     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/eScience.2013.34     Document Type: Conference Paper

References (36)

1. "Amazon Cloud Services", http://aws.amazon.com/.
2. I. Adams, D. D. E. Long, E. L. Miller, S. Pasupathy, and M. W. Storer, "Maximizing Efficiency by Trading Storage for Computation, " in Workshop on Hot Topics in Cloud Computing (HotCloud'09), pp. 1-5, 2009.
3. S. Agarwala, D. Jadav, and L. A. Bathen, "iCostale: Adaptive Cost Optimization for Storage Clouds, " in IEEE International Conference on Cloud Computing (CLOUD2011), pp. 436-443, 2011.
4. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, "A View of Cloud Computing, " Communication of the ACM, vol. 53, pp. 50-58, 2010.
5. M. S. Avila-Garcia, X. Xiong, A. E. Trefethen, C. Crichton, A. Tsui, and P. Hu, "A Virtual Research Environment for Cancer Imaging Research, " in 7th International Conference on E-Science (e-Science2011), pp. 1-6, 2011.
6. J. Baliga, R. W. Ayre, K. Hinton, and R. S. Tucker, "Green cloud computing: Balancing energy in processing, storage, and transport, " Proceedings of the IEEE, vol. 99, pp. 149-167, 2011.
7. R. Bose and J. Frew, "Lineage Retrieval for Scientific Data Processing: A Survey, " ACM Computing Surveys, vol. 37, pp. 1-28, 2005.
8. A. Burton and A. Treloar, "Publish My Data: A Composition of Services from ANDS and ARCS, " in 5th International Conference on E-Science (e- Science'09), pp. 164-170, 2009.
9. P. Chen, B. Plale, and M. S. Aktas, "Temporal representation for scientific data provenance, " in 8th International Conference on E-Science (e- Science2012), pp. 1-8, 2012.
10. Y. Cui, H. Wang, and X. Cheng, "Channel Allocation in Wireless Data Center Networks, " in IEEE INFOCOM 2011 pp. 1395-1403, 2011.
11. E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good, "The Cost of Doing Science on the Cloud: The Montage Example, " in ACM/IEEE Conference on Supercomputing (SC'08), pp. 1-12, 2008.
12. I. Foster, J. Vockler, M. Wilde, and Z. Yong, "Chimera: A Virtual Data System for Representing, Querying, and Automating Data Derivation, " in 14th International Conference on Scientific and Statistical Database Management, (SSDBM'02), pp. 37-46, 2002.
13. I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared, " in Grid Computing Environments Workshop (GCE'08), pp. 1-10, 2008.
14. S. K. Garg, R. Buyya, and H. J. Siegel, "Time and Cost Trade-Off Management for Scheduling Parallel Applications on Utility Grids, " Future Generation Computer Systems, vol. 26, pp. 1344-1355, 2010.
15. P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, and L. Zhuang, "Nectar: Automatic Management of Data and Computation in Datacenters, " in 9th Symposium on Operating Systems Design and Implementation (OSDI'2010), pp. 1-14, 2010.
16. X. Huang, Z. Luo, and B. Yan, "Cyberinfrastructure and e-Science Application Practices in Chinese Academy of Sciences, " in 7th International Conference on E-Science (e-Science2011), pp. 348-354, 2011.
17. M. Humphrey, N. Beekwilder, J. L. Goodall, and M. B. Ercan, "Calibration of watershed models using cloud computing, " in 8th International Conference on E-Science (e-Science2012), pp. 1-8, 2012.
18. G. Juve, E. Deelman, K. Vahi, and G. Mehta, "Data Sharing Options for Scientific Workflows on Amazon EC2, " in ACM/IEEE Conference on Supercomputing (SC'10), pp. 1-9, 2010.
19. D. Kondo, B. Javadi, P. Malecot, F. Cappello, and D. P. Anderson, "Cost- Benefit Analysis of Cloud Computing versus Desktop Grids, " in 23th IEEE International Parallel & Distributed Processing Symposium (IPDPS'09), pp. 1-12, 2009.
20. X. Liu, Z. Ni, D. Yuan, Y. Jiang, Z. Wu, J. Chen, and Y. Yang, "A Novel Statistical Time-Series Pattern based Interval Forecasting Strategy for Activity Durations in Workflow Systems, " Journal of Systems and Software vol. 84, pp. 354-376, 2011.
21. B. Ludascher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, and E. A. Lee, "Scientific Workflow Management and the Kepler System, " Concurrency and Computation: Practice and Experience, pp. 1039-1065, 2005.
22. K.-K. Muniswamy-Reddy, P. Macko, and M. Seltzer, "Provenance for the Cloud, " in 8th USENIX Conference on File and Storage Technology (FAST'10), pp. 197-210, 2010.
23. H. Nguyen and D. Abramson, "WorkWays: Interactive Workflow-based Science Gateways, " in 8th International Conference on E-Science (e- Science2012), pp. 1-8, 2012.
24. L. J. Osterweil, L. A. Clarke, A. M. Ellison, R. Podorozhny, A. Wise, E. Boose, and J. Hadley, "Experience in Using A Process Language to Define Scientific Workflow and Generate Dataset Provenance, " in 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 319-329, 2008.
25. J. Qiu, J. Ekanayake, T. Gunarathne, J. Y. Choi, S.-H. Bae, H. Li, B. Zhang, Y. Ryan, S. Ekanayake, T.-L. Wu, A. Hughes, and G. Fox, "Hybrid Cloud and Cluster Computing Paradigms for Life Science Applications " Journal of BMC Bioinformatics, vol. 11, 2010.
26. X. Su, Y. Ma, H. Yang, X. Chang, K. Nan, J. Xu, and K. Ning, "An Open- Source Collaboration Environment for Metagenomics Research, " in 7th International Conference on E-Science (e-Science2011), pp. 7-14, 2011.
27. A. S. Szalay and J. Gray, "Science in an Exponential World, " Nature, vol. 440, pp. 23-24, 2006.
28. S. Toor, M. Sabesan, S. Holmgren, and T. Risch, "A Scalable Architecture for e-Science Data Management, " in 7th International Conference on EScience (e-Science2011), pp. 210-217, 2011.
29. D. Warneke and O. Kao, "Exploiting Dynamic Resource Allocation for Efficient Parallel Data Processing in the Cloud, " IEEE Transactions on Parallel and Distributed Systems, vol. 22, pp. 985-997, 2011.
30. Y. Yang, K. Liu, J. Chen, X. Liu, D. Yuan, and H. Jin, "An Algorithm in SwinDeW-C for Scheduling Transaction-Intensive Cost-Constrained Cloud Workflows, " in 4th International Conference on E-Science (e- Science2008), pp. 374-375, 2008.
31. L. Young Choon and A. Y. Zomaya, "Energy Conscious Scheduling for Distributed Computing Systems under Different Operating Conditions, " IEEE Transactions on Parallel and Distributed Systems, vol. 22, pp. 1374- 1381, 2011.
32. D. Yuan, Y. Yang, X. Liu, and J. Chen, "A Cost-Effective Strategy for Intermediate Data Storage in Scientific Cloud Workflows, " in 24th IEEE International Parallel & Distributed Processing Symposium (IPDPS'10), pp. 1-12, 2010.
33. D. Yuan, Y. Yang, X. Liu, and J. Chen, "On-demand Minimum Cost Benchmarking for Intermediate Datasets Storage in Scientific Cloud Workflow Systems, " Journal of Parallel and Distributed Computing, vol. 71, pp. 316-332, 2011.
34. D. Yuan, Y. Yang, X. Liu, W. Li, L. Cui, M. Xu, and J. Chen, "A Highly Practical Approach towards Achieving Minimum Datasets Storage Cost in the Cloud, " IEEE Transactions on Parallel and Distributed Systems, vol. 24, pp. 1234-1244, 2012.
35. D. Yuan, Y. Yang, X. Liu, G. Zhang, and J. Chen, "A Data Dependency Based Strategy for Intermediate Data Storage in Scientific Cloud Workflow Systems, " Concurrency and Computation: Practice and Experience, vol. 24, pp. 956-976, 2012.
36. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving MapReduce Performance in Heterogeneous Environments, " in 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI'2008), pp. 29-42, 2008.