On-demand minimum cost benchmarking for intermediate dataset storage in scientific cloud workflow systems

Journal of Parallel and Distributed Computing

Volumn 71, Issue 2, 2011, Pages 316-332

  Yuan, Dong ^a   Yang, Yun ^a   Liu, Xiao ^a   Chen, Jinjun ^a  

^a SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Cloud computing; Cost benchmarking; Dataset storage; Scientific workflow

Indexed keywords

CLOUD COMPUTING; CLOUD STORAGES; COMPUTATION COSTS; COST BENCHMARKING; COST MODELS; DATA INTENSIVE; DATA PROVENANCE; DATA SETS; INTERMEDIATE DATUM; MINIMUM COST; NOVEL ALGORITHM; PAY-AS-YOU-GO; SCIENTIFIC WORKFLOWS; STORAGE CAPACITY; STORAGE COSTS; WORK-FLOW SYSTEMS; WORK-FLOWS;

ALGORITHMS; ASTROPHYSICS; BENCHMARKING; CLOUDS; COMPUTER SYSTEMS; ECONOMIC AND SOCIAL EFFECTS; MANAGEMENT;

COSTS;

EID: 78650419636     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2010.09.003     Document Type: Article

References (38)

1. I. Adams, D.D.E. Long, E.L. Miller, S. Pasupathy, M.W. Storer, Maximizing efficiency by trading storage for computation, in: Workshop on Hot Topics in Cloud Computing, HotCloud'09, San Diego, CA, 2009, pp. 15.
2. I. Altintas, O. Barney, E. Jaeger-Frank, Provenance collection support in the Kepler scientific workflow system, in: International Provenance and Annotation Workshop, Chicago, Illinois, USA, 2006, pp. 118132. (Pubitemid 44577314)
3. Amazon Cloud Services. http://aws.amazon.com/ (accessed on 12.08.10).
4. M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R.H. Katz, A. Konwinski, G. Lee, D.A. Patterson, A. Rabkin, I. Stoica, M. Zaharia, Above the clouds: a Berkeley view of cloud computing, Technical Report UCB/EECS-2009-28, University of California at Berkeley. http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS- 2009-28.pdf (accessed on 12.08.10).
5. M.D.d. Assuncao, A.d. Costanzo, R. Buyya, Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters, in: 18th ACM International Symposium on High Performance Distributed Computing, HPDC'09, Garching, Germany, 2009, pp. 110.
6. ATNF Parkes Swinburne Recorder. http://astronomy.swin.edu.au/pulsar/? topic=apsr (accessed on 12.08.10).
7. Australia Telescope National Facility. http://www.atnf.csiro.au/ (accessed on 12.08.10).
8. Australia Telescope, Parkes Observatory. http://www.parkes.atnf.csiro.au/ (accessed on 12.08.10).
9. Z. Bao, S. Cohen-Boulakia, S.B. Davidson, A. Eyal, S. Khanna, Differencing provenance in scientific workflows, in: 25th IEEE International Conference on Data Engineering, ICDE'09, Shanghai, China, 2009, pp. 808819.
10. R. Bose, and J. Frew Lineage retrieval for scientific data processing: a survey ACM Computing Surveys 37 2005 1 28 (Pubitemid 41246320)
11. R. Buyya, C.S. Yeo, S. Venugopal, J. Broberg, and I. Brandic Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility Future Generation Computer Systems 25 2009 599 616
12. A. Chervenak, E. Deelman, I. Foster, L. Guy, W. Hoschek, A. Iamnitchi, C. Kesselman, P. Kunszt, M. Ripeanu, B. Schwartzkopf, H. Stockinger, K. Stockinger, B. Tierney, Giggle: a framework for constructing scalable replica location services, in: ACM/IEEE Conference on Supercomputing, SC'02, Baltimore, Maryland, 2002, pp. 117.
13. A. Chervenak, E. Deelman, M. Livny, M.-H. Su, R. Schuler, S. Bharathi, G. Mehta, K. Vahi, Data placement for scientific applications in distributed environments, in: 8th Grid Computing Conference, Austin, Texas, USA, 2007, pp. 267274.
14. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, S. Patil, M.-H. Su, K. Vahi, M. Livny, Pegasus: mapping scientific workflows onto the grid, in: European Across Grids Conference, Nicosia, Cyprus, 2004, pp. 1120. (Pubitemid 39741759)
15. E. Deelman, A. Chervenak, Data management challenges of data-intensive scientific workflows, in: IEEE International Symposium on Cluster Computing and the Grid, CCGrid'08, Lyon, France, 2008, pp. 687692.
16. E. Deelman, G. Singh, M. Livny, B. Berriman, J. Good, The cost of doing science on the cloud: the montage example, in: ACM/IEEE Conference on Supercomputing, SC'08, Austin, Texas, 2008, pp. 112.
17. I. Foster, Z. Yong, I. Raicu, S. Lu, Cloud computing and grid computing 360-degree compared, in: Grid Computing Environments Workshop, GCE'08, Austin, Texas, USA, 2008, pp. 110.
18. P. Groth, and L. Moreau Recording process documentation for provenance IEEE Transactions on Parallel and Distributed Systems 20 2009 1246 1259
19. Hadoop. http://hadoop.apache.org/ (accessed on 12.08.10).
20. C. Hoffa, G. Mehta, T. Freeman, E. Deelman, K. Keahey, B. Berriman, J. Good, On the use of cloud computing for scientific workflows, in: 4th IEEE International Conference on e-Science, Indianapolis, Indiana, USA, 2008, pp. 640645.
21. K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, M. Tsugawa, Science clouds: early experiences in cloud computing for scientific applications, in: First Workshop on Cloud Computing and its Applications, CCA'08, Chicago, Illinois, USA, 2008, pp. 16.
22. D. Kondo, B. Javadi, P. Malecot, F. Cappello, D.P. Anderson, Cost-benefit analysis of cloud computing versus desktop grids, in: IEEE International Symposium on Parallel & Distributed Processing, IPDPS'09, Rome, Italy, 2009, pp. 112.
23. X. Liu, D. Yuan, G. Zhang, J. Chen, and Y. Yang SwinDeW-C: a peer-to-peer based cloud workflow system for managing instance intensive applications Handbook of Cloud Computing 2010 Springer 309 332 http://www.ict.swin.edu.au/ personal/xliu/papers/SwinDeW-C.pdf
24. B. Ludascher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, and E.A. Lee Scientific workflow management and the Kepler system Concurrency and Computation: Practice and Experience 2005 1039 1065 (Pubitemid 46116684)
25. C. Moretti, J. Bulosan, D. Thain, P.J. Flynn, All-Pairs: an abstraction for data-intensive cloud computing, in: IEEE International Parallel & Distributed Processing Symposium, IPDPS'08, Miami, Florida, USA, 2008, pp. 111.
26. P. Odifreddi Classical Recursion Theory: The Theory of Functions and Sets of Natural Numbers vol. 125 1992 Elsevier pp. ii-xi, 1668
27. T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M.R. Pocock, A. Wipat, and P. Li Taverna: a tool for the composition and enactment of bioinformatics workflows Bioinformatics 20 2004 3045 3054 (Pubitemid 39619196)
28. L.J. Osterweil, L.A. Clarke, A.M. Ellison, R. Podorozhny, A. Wise, E. Boose, J. Hadley, Experience in using a process language to define scientific workflow and generate dataset provenance, in: 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, Atlanta, Georgia, 2008, pp. 319329.
29. Y.L. Simmhan, B. Plale, and D. Gannon A survey of data provenance in e-science SIGMOD Record 34 2005 31 36 (Pubitemid 43152529)
30. Swinburne high performance supercomputing facility. http://astronomy. swin.edu.au/supercomputing/ (accessed on 12.08.10).
31. L. Wang, J. Tao, M. Kunze, A.C. Castellanos, D. Kramer, W. Karl, Scientific cloud computing: early definition and experience, in: 10th IEEE International Conference on High Performance Computing and Communications, HPCC'08, Dalin, China, 2008, pp. 825830.
32. A. Weiss Computing in the cloud ACM Networker 11 2007 18 25
33. VMware. http://www.vmware.com/ (accessed on 12.08.10).
34. J. Yan, Y. Yang, and G.K. Raikundalia SwinDeWa P2P-based decentralized workflow management system IEEE Transactions on Systems, Man and Cybernetics, Part A 36 2006 922 935
35. Y. Yang, K. Liu, J. Chen, J. Lignier, H. Jin, Peer-to-Peer based grid workflow runtime environment of SwinDeW-G, in: IEEE International Conference on e-Science and Grid Computing, Bangalore, India, 2007, pp. 5158.
36. D. Yuan, Y. Yang, X. Liu, J. Chen, A cost-effective strategy for intermediate data storage in scientific cloud workflows, in: 24th IEEE International Parallel & Distributed Processing Symposium, IPDPS'10, Atlanta, Georgia, USA, 2010, pp. 112.
37. D. Yuan, Y. Yang, X. Liu, and J. Chen A data placement strategy in scientific cloud workflows Future Generation Computer Systems 26 2010 1200 1214
38. D. Yuan, Y. Yang, X. Liu, G. Zhang, J. Chen, A data dependency based strategy for intermediate data storage in scientific cloud workflow systems, Concurrency and Computation: Practice and Experience (2010) (http://dx.doi.org/ 10.1002/cpe.1636 ).