Cloud data management for scientific workflows: Research issues, methodologies, and state-of-the-art

Proceedings - 2014 10th International Conference on Semantics, Knowledge and Grids, SKG 2014

Volumn , Issue , 2014, Pages 21-28

  Yuan, Dong ^a   Cui, Lizhen ^b   Liu, Xiao ^c  

^a SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

^b SHANDONG UNIVERSITY   (China)

^c EAST CHINA NORMAL UNIVERSITY   (China)

Author keywords

Cloud Computing; Data Management; Research Issues; Scientific Workflow

Indexed keywords

AUTOMATION; CLOUD COMPUTING; DATA SHARING; DIGITAL STORAGE; SEMANTICS;

CLOUD COMPUTING TECHNOLOGIES; DISTRIBUTED COMPUTING SYSTEMS; HIGH PERFORMANCE COMPUTING SYSTEMS; HIGH-PERFORMANCE COMPUTING RESOURCES; RESEARCH ISSUES; RESEARCH METHODOLOGIES; SCIENTIFIC APPLICATIONS; SCIENTIFIC WORKFLOWS;

INFORMATION MANAGEMENT;

EID: 84918491741     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SKG.2014.37     Document Type: Conference Paper

References (68)

1. E. Deelman and A. Chervenak, "Data Management Challenges of Data-Intensive Scientific Workflows, " in IEEE International Symposium on Cluster Computing and the Grid (CCGrid'08), Lyon, France, 2008, pp. 687-692.
2. A. S. Szalay and J. Gray, "Science in an Exponential World, " Nature, vol. 440, pp. 23-24, 2006.
3. B. Ludascher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, et al., "Scientific Workflow Management and the Kepler System, " Concurrency and Computation: Practice and Experience, pp. 1039-1065, 2005.
4. A. Weiss, "Computing in the Cloud, " ACM Networker, vol. 11, pp. 18-25, 2007.
5. R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, "Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility, " Future Generation Computer Systems, vol. in press, pp. 1-18, 2009.
6. I. Foster, Z. Yong, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared, " in Grid Computing Environments Workshop (GCE'08), Austin, Texas, USA, 2008, pp. 1-10.
7. C. Moretti, J. Bulosan, D. Thain, and P. J. Flynn, "All-Pairs: An Abstraction for Data-Intensive Cloud Computing, " in 22nd IEEE International Parallel & Distributed Processing Symposium (IPDPS'08), Miami, Florida, USA, 2008.
8. K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, and M. Tsugawa, "Science Clouds: Early Experiences in Cloud Computing for Scientific Applications, " in First Workshop on Cloud Computing and its Applications (CCA'08), Chicago, Illinois, USA, 2008, pp. 1-6.
9. L. Wang, J. Tao, M. Kunze, A. C. Castellanos, D. Kramer, and W. Karl, "Scientific Cloud Computing: Early Definition and Experience, " in 10th IEEE International Conference on High Performance Computing and Communications, HPCC '08., Dalin, China, 2008, pp. 825-830.
10. E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good, "The Cost of Doing Science on the Cloud: the Montage Example, " in ACM/IEEE Conference on Supercomputing (SC'08), Austin, Texas, 2008, pp. 1-12.
11. C. Hoffa, G. Mehta, T. Freeman, E. Deelman, K. Keahey, B. Berriman, et al., "On the Use of Cloud Computing for Scientific Workflows, " in 4th IEEE International Conference on e-Science, Indianapolis, Indiana, USA, 2008, pp. 640-645.
12. M. A. Vouk, "Cloud Computing -Issues, Research and Implementations, " in 30th Int. Conf. on Information Technology Interfaces, Cavtat, Croatia, 2008, pp. 31-40.
13. A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, and S. Tuecke, "The data grid: Towards an architecture for the distributed management and analysis of large scientific datasets, " Journal of Network and Computer Applications, vol. 23, pp. 187-200, 2000.
14. S. Venugopal, R. Buyya, and K. Ramamohanarao, "A Taxonomy of Data Grids for Distributed Data Sharing, Management, and Processing, " ACM Computing Surveys, vol. 38, pp. 1-53, 2006.
15. O. Tatebe, Y. Morita, S. Matsuoka, N. Soda, and S. Sekiguchi, "Grid Datafarm Architecture for Petascale Data Intensive Computing, " in 2nd IEEE/ACM International Symposium on Cluster Computing and the (CCGrid2002), Berlin, Germany, 2002, pp. 102-110.
16. H. Stockinger, A. Samar, K. Holtman, B. Allcock, I. Foster, and B. Tierney, "File and Object Replication in Data Grids " Cluster Computing, vol. 5, pp. 305-314, 2002.
17. D. T. Liu and M. J. Franklin, "GridDB: A Data-Centric Overlay for Scientific Grids, " in 30th VLDB Conference, Toronto, Canada, 2004, pp. 600-611.
18. C. Baru, R. Moore, A. Rajasekar, and M. Wan, "The SDSC Storage Resource Broker, " in IBM Centre for Advanced Studies Conference, Toronto, Canada 1998, pp. 1-12.
19. A. Chervenak, E. Deelman, I. Foster, L. Guy, W. Hoschek, A. Iamnitchi, et al., "Giggle: A Framework for Constructing Scalable Replica Location Services, " in ACM/IEEE conference on Supercomputing (SC'02), Baltimore, Maryland, 2002, pp. 1-17.
20. M. Cai, A. Chervenak, and M. Frank, "A Peer-to-Peer Replica Location Service Based on A Distributed Hash Table, " in ACM/IEEE Conference on Supercomputing, Pittsburgh, USA, 2004.
21. S. Venugopal, R. Buyya, and L. Winton, "A Grid Service Broker for Scheduling Distributed Data-Oriented Applications on Global Grids, " in 2nd Workshop on Middleware in Grid Computing, Toronto, Canada, 2004, pp. 75-80.
22. S. Jablonski, O. Curé, M. A. Rehman, and B. Volz, "DaltOn: An Infrastructure for Scientific Data Management, " in Proceedings of the 8th international conference on Computational Science (ICCS2008), Kraków, Poland, 2008, pp. 520-529.
23. B. Allcock, J. Bester, J. Bresnahan, A. L. Chervenak, I. Foster, C. Kesselman, et al., "Data management and transfer in highperformance computational grid environments, " Parallel Computing, vol. 28, pp. 749-771, 2002.
24. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, S. Patil, et al., "Pegasus: Mapping Scientific Workflows onto the Grid, " in European Across Grids Conference, Nicosia, Cyprus, 2004, pp. 11-20.
25. R. Buyya and S. Venugopal, "The Gridbus Toolkit for Service Oriented Grid and Utility Computing: An Overview and Status Report, " in IEEE International Workshop on Grid Economics and Business Models, Seoul, Korea, 2004, pp. 19-66.
26. T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, et al., "Taverna: A tool for the composition and enactment of bioinformatics workflows, " Bioinformatics, vol. 20, pp. 3045-3054, 2004.
27. T. Glatard, J. Montagnat, X. Pennec, D. Emsellem, and D. Lingrand, "MOTEUR: A Data-Intensive Service-Based Workflow Manager, " Research Report I3S, pp. 1-39, 2006.
28. M. Wieczorek, R. Prodan, and T. Fahringer, "Scheduling of Scientific Workflows in the ASKALON Grid Environment, " SIGMOD Record, vol. 34, pp. 56-62, 2005.
29. D. Churches, G. Gombas, A. Harrison, J. Maassen, C. Robinson, M. Shields, et al., "Programming scientific and distributed workflow with Triana services, " Concurrency and Computation: Practice and Experience, vol. 18, pp. 1021-1037, 2006.
30. C. Junwei, S. A. Jarvis, S. Saini, and G. R. Nudd, "GridFlow: Workflow Management for Grid Computing, " in 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid2003), Tokyo, Japan, 2003, pp. 198-205.
31. N. Russell, A. ter Hofstede, D. Edmond, and W. van der Aalst, "Workflow Data Patterns, " in 24th International Conference on Conceptual Modeling (ER'05), Klagenfurt, Austria, 2005, pp. 353-368.
32. W. M. Johnston, J. R. P. Hanna, and R. J. Millar, "Advances in dataflow programming languages, " ACM Computing Surveys, vol. 36, pp. 1-34, 2004.
33. A. Chervenak, E. Deelman, M. Livny, M.-H. Su, R. Schuler, S. Bharathi, et al., "Data Placement for Scientific Applications in Distributed Environments, " in 8th Grid Computing Conference, Austin, Texas, USA, 2007, pp. 267-274.
34. G. Singh, K. Vahi, A. Ramakrishnan, G. Mehta, E. Deelman, H. Zhao, et al., "Optimizing Workflow Data Footprint, " Scientific Programming, vol. 15, pp. 249-268, 2007.
35. OpenNebula. Available: http://www.opennebula.org/, accessed on 1st Sep. 2014.
36. Nimbus. Available: http://www.nimbusproject.org/, accessed on 1st Sep. 2014.
37. Eucalyptus. Available: http://open.eucalyptus.com/, accessed on 1st Sep. 2014.
38. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, et al., "A View of Cloud Computing, " Communication of the ACM, vol. 53, pp. 50-58, 2010.
39. M. D. d. Assuncao, A. d. Costanzo, and R. Buyya, "Evaluating the Cost-Benefit of Using Cloud Computing to Extend the Capacity of Clusters, " in 18th ACM International Symposium on High Performance Distributed Computing (HPDC'09), Garching, Germany, 2009, pp. 141-150.
40. D. Kondo, B. Javadi, P. Malecot, F. Cappello, and D. P. Anderson, "Cost-Benefit Analysis of Cloud Computing versus Desktop Grids, " in 23th IEEE International Parallel & Distributed Processing Symposium (IPDPS'09), Rome, Italy, 2009, pp. 1-12.
41. K. Tsakalozos, H. Kllapi, E. Sitaridi, M. Roussopoulos, D. Paparas, and A. Delis, "Flexible Use of Cloud Resources Through Profit Maximization and Price Discrimination, " in IEEE 27th International Conference on Data Engineering (ICDE2011), Hanover Germany, 2011, pp. 75-86.
42. B. Cho and I. Gupta, "New Algorithms for Planning Bulk Transfer via Internet and Shipping Networks, " in IEEE 30th International Conference on Distributed Computing Systems (ICDCS), 2010, pp. 305-314.
43. X. Fan, J. Cao, and W. Wu, "Contention-Aware Data Caching in Wireless Multi-hop ad hoc Networks, " Journal of Parallel and Distributed Computing, vol. 71, pp. 603-614, 2011.
44. Y. Huang, J. Cao, B. Jin, X. Tao, J. Lu, and Y. Feng, "Flexible Cache Consistency Maintenance over Wireless Ad Hoc Networks, " IEEE Transactions on Parallel and Distributed Systems, vol. 21, pp. 1150-1161, 2010.
45. X. Jia, D. Li, H. Du, and J. Cao, "On Optimal Replication of Data Object at Hierarchical and Transparent Web Proxies, " IEEE Transactions on Parallel and Distributed Systems, vol. 16, pp. 673-685, 2005.
46. D. Warneke and O. Kao, "Exploiting Dynamic Resource Allocation for Efficient Parallel Data Processing in the Cloud, " IEEE Transactions on Parallel and Distributed Systems, vol. 22, pp. 985-997, 2011.
47. L. Young Choon and A. Y. Zomaya, "Energy Conscious Scheduling for Distributed Computing Systems under Different Operating Conditions, " IEEE Transactions on Parallel and Distributed Systems, vol. 22, pp. 1374-1381, 2011.
48. S. K. Garg, R. Buyya, and H. J. Siegel, "Time and Cost Trade-Off Management for Scheduling Parallel Applications on Utility Grids, " Future Generation Computer Systems, vol. 26, pp. 1344-1355, 2010.
49. P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, and L. Zhuang, "Nectar: Automatic Management of Data and Computation in Datacenters, " in 9th Symposium on Operating Systems Design and Implementation (OSDI'2010), Vancouver, BC, Canada, 2010, pp. 1-14.
50. I. Adams, D. D. E. Long, E. L. Miller, S. Pasupathy, and M. W. Storer, "Maximizing Efficiency by Trading Storage for Computation, " in Workshop on Hot Topics in Cloud Computing (HotCloud'09), San Diego, CA, 2009, pp. 1-5.
51. D. Yuan, Y. Yang, X. Liu, and J. Chen, "Computation and Storage Trade-off for Cost-Effectively Storing Scientific Datasets in the Cloud, " in Handbook of Data Intensive Computing, ed: Springer New York, 2011, pp. 129-153.
52. D. Yuan, Y. Yang, X. Liu, and J. Chen, "On-demand Minimum Cost Benchmarking for Intermediate Datasets Storage in Scientific Cloud Workflow Systems, " Journal of Parallel and Distributed Computing, vol. 71, pp. 316-332, 2011.
53. D. Yuan, Y. Yang, X. Liu, and J. Chen, "A Cost-Effective Strategy for Intermediate Data Storage in Scientific Cloud Workflows, " in 24th IEEE International Parallel & Distributed Processing Symposium (IPDPS'10), Atlanta, Georgia, USA, 2010, pp. 1-12.
54. D. Yuan, Y. Yang, X. Liu, and J. Chen, "A Local-Optimisation based Strategy for Cost-Effective Datasets Storage of Scientific Applications in the Cloud, " in Proc. of 4th IEEE International Conference on Cloud Computing (Cloud2011), Washington DC, USA, 2011, pp. 179-186.
55. G. Zhang, Y. Yang, D. Yuan, and J. Chen, "A Trust-Based Noise Injection Strategy for Privacy Protection in Cloud, " Software: Practice and Experience, vol. 42, pp. 431-445, 2012.
56. D. Yuan, Y. Yang, X. Liu, G. Zhang, and J. Chen, "A Data Dependency Based Strategy for Intermediate Data Storage in Scientific Cloud Workflow Systems, " Concurrency and Computation: Practice and Experience, vol. 24, pp. 956-976, 2010.
57. D. Yuan, Y. Yang, X. Liu, W. Li, L. Cui, M. Xu, et al., "A Highly Practical Approach towards Achieving Minimum Datasets Storage Cost in the Cloud, " IEEE Transactions on Parallel and Distributed Systems, vol. 24, pp. 1234-1244, 2012.
58. R. Barga and D. Gannon, "Scientific versus Business Workflows, " in Workflows for e-Science, ed, 2007, pp. 9-16.
59. X. Liu, Y. Yang, D. Yuan, and J. Chen, "Do We Need to Handle Every Temporal Violation in Scientific Workflow Systems?, " ACM Trans. Softw. Eng. Methodol., vol. 23, pp. 1-34, 2014.
60. D. Yuan, Y. Yang, X. Liu, and J. Chen, "A Data Placement Strategy in Scientific Cloud Workflows, " Future Generation Computer Systems, vol. 26, pp. 1200-1214, 2010.
61. W. T. McCormick, P. J. Sehweitzer, and T. W. White, "Problem Decomposition and Data Reorganization by a Clustering Technique, " Operations Research, vol. 20, pp. 993-1009, 1972.
62. K. Deng, K. Ren, J. Song, D. Yuan, Y. Xiang, and J. Chen, "A Clustering based Coscheduling Strategy for Efficient Scientific Workflow Execution in Cloud Computing, " Concurrency and Computation: Practice and Experience, vol. 25, pp. 2523-2539, 2013.
63. K. Deng, J. Song, K. Ren, D. Yuan, and J. Chen, "Graph-Cut Based Coscheduling Strategy Towards Efficient Execution of Scientific Workflows in Collaborative Cloud Environments, " in Proceedings of the 2011 IEEE/ACM 12th International Conference on Grid Computing, 2011, pp. 34-41.
64. X. Liu, Y. Yang, D. Yuan, G. Zhang, W. Li, and D. Cao, "A generic QoS framework for cloud workflow systems, " in 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing (DASC) 2011, pp. 713-720.
65. X. Liu, Z. Ni, Z. Wu, D. Yuan, J. Chen, and Y. Yang, "A Novel General Framework for Automatic and Cost-Effective Handling of Recoverable Temporal Violations in Scientific Workflow Systems, " Journal of Systems and Software, vol. 84, pp. 492-509, 2011.
66. D. Yuan, X. Liu, L. Cui, T. Zhang, W. Li, D. Cao, et al., "An Algorithm for Cost-Effectively Storing Scientific Datasets with Multiple Service Providers in the Cloud, " in 9th International Conference on e-Science (e-Science2013), 2013, pp. 285-292.
67. W. Li, Y. Yang, and D. Yuan, "A Novel Cost-Effective Dynamic Data Replication Strategy for Reliability in Cloud Data Centres, " in 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing (DASC), 2011, pp. 496-502.
68. W. Li, Y. Yang, J. Chen, and D. Yuan, "A Cost-Effective Mechanism for Cloud Data Reliability Management based on Proactive Replica Checking, " in Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2012), 2012, pp. 564-571.