A data dependency based strategy for intermediate data storage in scientific cloud workflow systems

Concurrency and Computation: Practice and Experience

Volumn 24, Issue 9, 2012, Pages 956-976

  Yuan, Dong ^a   Yang, Yun ^a   Liu, Xiao ^a   Zhang, Gaofeng ^a   Chen, Jinjun ^a  

^a SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

cloud computing; data sets storage; scientific workflow

Indexed keywords

ASTROPHYSICS; CLOUD COMPUTING; COST EFFECTIVENESS; COST REDUCTION; ECONOMIC AND SOCIAL EFFECTS; PETROLEUM RESERVOIR EVALUATION; STORAGE AS A SERVICE (STAAS);

CLOUD SERVICE PROVIDERS; COMPUTATION COSTS; DATA DEPENDENCIES; DATA DEPENDENCY GRAPHS; DATA PROVENANCE; NOVEL INTERMEDIATES; SCIENTIFIC WORKFLOWS; STORAGE CAPACITY;

DIGITAL STORAGE;

EID: 84861131986     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.1636     Document Type: Article

References (33)

1. Deelman E, Blythe J, Gil Y, Kesselman C, Mehta G, Patil S, Su M-H, Vahi K, Livny M,. Pegasus: Mapping scientific workflows onto the grid. European Across Grids Conference, Nicosia, Cyprus, 2004; 11-20. (Pubitemid 39741759)
2. Ludascher B, Altintas I, Berkley C, Higgins D, Jaeger E, Jones M, Lee EA,. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience, 2005; 1039-1065.
3. Oinn T, Addis M, Ferris J, Marvin D, Senger M, Greenwood M, Carver T, Glover K, Pocock MR, Wipat A, Li P,. Taverna: A tool for the composition and enactment of bioinformatics workflows. Bioinformatics 2004; 20: 3045-3054. (Pubitemid 39619196)
4. Deelman E, Chervenak A,. Data management challenges of data-intensive scientific workflows. IEEE International Symposium on Cluster Computing and the Grid (CCGrid'08), Lyon, France, 2008; 687-692.
5. Adams I, Long EDD, Miller LE, Pasupathy S, Storer WM,. Maximizing efficiency by trading storage for computation. Workshop on Hot Topics in Cloud Computing (HotCloud'09), San Diego, CA, 2009; 1-5.
6. Yuan D, Yang Y, Liu X, Chen J,. A cost-effective strategy for intermediate data storage in scientific cloud workflows. The 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS'10), Atlanta, Georgia, U.S.A., 2010; 1-12.
7. Weiss A,. Computing in the cloud. ACM Networker 2007; 11: 18-25.
8. Buyya R, Yeo SC, Venugopal S, Broberg J, Brandic I,. Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Generation Computer Systems 2009; 25: 599-616.
9. Foster I, Yong Z, Raicu I, Lu S,. Cloud computing and grid computing 360-degree compared. Grid Computing Environments Workshop (GCE'08), Austin, TX, U.S.A., 2008; 1-10.
10. Moretti C, Bulosan J, Thain D, Flynn PJ,. All-pairs: An abstraction for data-intensive cloud computing. IEEE International Parallel and Distributed Processing Symposium, IPDPS'08, Miami, FL, U.S.A., 2008; 1-11.
11. Keahey K, Figueiredo R, Fortes J, Freeman T, Tsugawa M,. Science clouds: Early experiences in cloud computing for scientific applications. First Workshop on Cloud Computing and its Applications (CCA'08), Chicago, IL, U.S.A., 2008; 1-6.
12. Wang L, Tao J, Kunze M, Castellanos AC, Kramer D, Karl W,. Scientific cloud computing: Early definition and experience. The 10th IEEE International Conference on High Performance Computing and Communications, HPCC '08, Dalin, China, 2008; 825-830.
13. Deelman E, Singh G, Livny M, Berriman B, Good J,. The cost of doing science on the cloud: The Montage example. ACM/IEEE Conference on Supercomputing (SC'08), Austin, TX, U.S.A., 2008; 1-12.
14. Hoffa C, Mehta G, Freeman T, Deelman E, Keahey K, Berriman B, Good J,. On the use of cloud computing for scientific workflows. The 4th IEEE International Conference on e-Science, Indianapolis, IN, U.S.A., 2008; 640-645.
15. Chen J, Yang Y,. Activity completion duration based checkpoint selection for dynamic verification of temporal constraints in grid workflow systems. International Journal of High Performance Computing Applications 2008; 22: 319-329.
16. Chen J, Yang Y,. Temporal dependency based checkpoint selection for dynamic verification of temporal constraints in scientific workflow systems. ACM Transactions on Software Engineering and Methodology 2009. Available at: [accepted on 17 June 2009 ].
17. Simmhan YL, Plale B, Gannon D,. A survey of data provenance in e-science. SIGMOD Record 2005; 34: 31-36. (Pubitemid 43152529)
18. Bose R, Frew J,. Lineage retrieval for scientific data processing: A survey. ACM Computing Surveys 2005; 37: 1-28. (Pubitemid 41246320)
19. Amazon Cloud Services. Available at: [15 May 2010 [.
20. Liu X, Yuan D, Zhang G, Chen J, Yang Y,. SwinDeW-C: A Peer-to-peer based cloud workflow system for managing instance intensive applications. Handbook of Cloud Computing. Springer: Berlin. Available at: (to be published in July 2010).
21. Yang Y, Liu K, Chen J, Lignier J, Jin H,. Peer-to-peer based grid workflow runtime environment of SwinDeW-G. IEEE International Conference on E-science and Grid Computing, Bangalore, India, 2007; 51-58.
22. Yan J, Yang Y, Raikundalia GK,. SwinDeW-A P2P-based decentralized workflow management system. IEEE Transactions on Systems, Man and Cybernetics, Part A 2006; 36: 922-935.
23. Armbrust M, Fox A, Griffith R, Joseph DA, Katz HR, Konwinski A, Lee G, Patterson AD, Rabkin A, Stoica I, Zaharia M,. Above the clouds: A Berkeley view of cloud computing. Technical Report UCB/EECS-2009-28, University of California at Berkeley. Available at: (accessed on 15 May 2010).
24. Chervenak A, Deelman E, Foster I, Guy L, Hoschek W, Iamnitchi A, Kesselman C, Kunszt P, Ripeanu M, Schwartzkopf B, Stockinger H, Stockinger K, Tierney B,. Giggle: A framework for constructing scalable replica location services. ACM/IEEE Conference on Supercomputing (SC'02), Baltimore, MD, 2002; 1-17.
25. Chervenak A, Deelman E, Livny M, Su MH, Schuler R, Bharathi S, Mehta G, Vahi K,. Data placement for scientific applications in distributed environments. The 8th Grid Computing Conference, Austin, TX, U.S.A., 2007; 267-274.
26. Yuan D, Yang Y, Liu X, Chen J,. A data placement strategy in scientific cloud workflows. Future Generation Computer Systems, DOI: 10.1016/j.future.2010. 02.004.
27. Buyya R, Venugopal S,. The gridbus toolkit for service oriented grid and utility computing: An overview and status report. IEEE International Workshop on Grid Economics and Business Models, Seoul, 2004; 19-66.
28. Assuncao MDD, Costanzo AD, Buyya R,. Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters. The 18th ACM International Symposium on High Performance Distributed Computing (HPDC'09), Garching, Germany, 2009; 1-10.
29. Kondo D, Javadi B, Malecot P, Cappello F, Anderson DP,. Cost-benefit analysis of cloud computing versus desktop grids. IEEE International Symposium on Parallel and Distributed Processing (IPDPS'09), Rome, Italy, 2009; 1-12.
30. Bao Z, Cohen-Boulakia S, Davidson BS, Eyal A, Khanna S,. Differencing provenance in scientific workflows. The 25th IEEE International Conference on Data Engineering (ICDE'09), Shanghai, China, 2009; 808-819.
31. Groth P, Moreau L,. Recording process documentation for provenance. IEEE Transactions on Parallel and Distributed Systems 2009; 20: 1246-1259.
32. Altintas I, Barney O, Jaeger-Frank E,. Provenance collection support in the Kepler scientific workflow system. International Provenance and Annotation Workshop, Chicago, IL, U.S.A., 2006; 118-132. (Pubitemid 44577314)
33. Osterweil JL, Clarke AL, Ellison AM, Podorozhny R, Wise A, Boose E, Hadley J,. Experience in using a process language to define scientific workflow and generate dataset provenance. The 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, Atlanta, GA, 2008; 319-329.