A data placement strategy in scientific cloud workflows

Future Generation Computer Systems

Volumn 26, Issue 8, 2010, Pages 1200-1214

  Yuan, Dong ^a   Yang, Yun ^a   Liu, Xiao ^a   Chen, Jinjun ^a  

^a SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Cloud computing; Data management; Scientific workflow

Indexed keywords

APPLICATION DATA; CLOUD COMPUTING; DATA CENTRES; DATA MANAGEMENT; DATA MOVEMENTS; DATA PLACEMENT; DATA SETS; DISTRIBUTED DATA; K-MEANS CLUSTERING; MATRIX; RUNTIMES; SCIENTIFIC WORKFLOWS; TASK-NEEDS; WORK-FLOWS;

DISTRIBUTED COMPUTER SYSTEMS;

INFORMATION MANAGEMENT;

EID: 77955511626     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2010.02.004     Document Type: Article

References (52)

1. E. Deelman, A. Chervenak, Data management challenges of data-intensive scientific workflows, in: IEEE International Symposium on Cluster Computing and the Grid, 2008, pp. 687692.
2. E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, S. Patil, M.-H. Su, K. Vahi, M. Livny, Pegasus: Mapping scientific workflows onto the grid, in: European Across Grids Conference, 2004, pp. 1120.
3. B. Ludascher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, and E.A. Lee Scientific workflow management and the Kepler system Concurrency and Computation: Practice and Experience 2005 1039 1065
4. T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M.R. Pocock, A. Wipat, and P. Li Taverna: A tool for the composition and enactment of bioinformatics workflows Bioinformatics 20 2004 3045 3054
5. A. Weiss Computing in the Cloud, vol. 11 2007 ACM Networker 1825
6. M. Brantner, D. Florescuy, D. Graf, D. Kossmann, T. Kraska, Building a Database on S3, in: SIGMOD, Vancouver, BC, Canada, 2008, pp. 251263.
7. R. Grossman, Y. Gu, Data Mining Using High Performance Data Clouds: Experimental Studies Using Sector and Sphere, in: SIGKDD, 2008, pp. 920927.
8. R. Buyya, C.S. Yeo, S. Venugopal, Market-oriented cloud computing: Vision, hype, and reality for delivering IT services as computing utilities, in: 10th IEEE International Conference on High Performance Computing and Communications, HPCC-08, Los Alamitos, CA, USA, 2008.
9. C. Moretti, J. Bulosan, D. Thain, P.J. Flynn, All-Pairs: An abstraction for data-intensive cloud computing, in: IEEE International Parallel & Distributed Processing Symposium, IPDPS'08, 2008, pp. 111.
10. R. Buyya, C.S. Yeo, S. Venugopal, J. Broberg, and I. Brandic Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility Future Generation Computer Systems 25 2009 599 616
11. I. Foster, Z. Yong, I. Raicu, S. Lu, Cloud computing and grid computing 360-degree compared, in: Grid Computing Environments Workshop, GCE'08, 2008, pp. 110.
12. K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, M. Tsugawa, Science clouds: Early experiences in cloud computing for scientific applications, in: First Workshop on Cloud Computing and its Applications, CCA'08, 2008, pp. 16.
13. L. Wang, J. Tao, M. Kunze, A.C. Castellanos, D. Kramer, W. Karl, Scientific cloud computing: Early definition and experience, in: 10th IEEE International Conference on High Performance Computing and Communications, HPCC'08, 2008, pp. 825830.
14. E. Deelman, G. Singh, M. Livny, B. Berriman, J. Good, The cost of doing science on the cloud: The montage example, in: ACM/IEEE Conference on Supercomputing, Austin, Texas, 2008, pp. 112.
15. C. Hoffa, G. Mehta, T. Freeman, E. Deelman, K. Keahey, B. Berriman, J. Good, On the use of cloud computing for scientific workflows, in: 4th IEEE International Conference on e-Science, 2008, pp. 640645.
16. R. Barga, and D. Gannon Scientific versus business workflows I.J. Taylor, E. Deelman, D.B. Gannon, M. Shields, Workflows for e-Science 2007 Springer London, UK 9 16
17. W.T. McCormick, P.J. Sehweitzer, and T.W. White Problem decomposition and data reorganization by a clustering technique Operations Research 20 1972 993 1009
18. T. Xie SEA: A striping-based energy-aware strategy for data placement in RAID-structured storage systems IEEE Transactions on Computers 57 2008 748 761
19. T. Kosar, M. Livny, Stork: Making data placement a first class citizen in the grid, in: Proceedings of 24th International Conference on Distributed Computing Systems, ICDCS 2004, 2004, pp. 342349.
20. J.M. Cope, N. Trebon, H.M. Tufo, P. Beckman, Robust data placement in urgent computing environments, in: IEEE International Symposium on Parallel & Distributed Processing, IPDPS 2009, 2009, pp. 113.
21. N. Hardavellas, M. Ferdman, B. Falsafi, A. Ailamaki, Reactive NUCA: Near-optimal block placement and replication in distributed caches, in: Proceedings of the 36th Annual International Symposium on Computer Architecture, ISCA '09, Austin, TX, USA, 2009, PP. 184195.
22. S. Ghemawat, H. Gobioff, and S.-T. Leung The Google File System, vol. 37 2003 SIGOPS Oper. Syst. Rev. 2943
23. Hadoop, http://hadoop.apache.org/ (accessed 25.11.09).
24. R. Prodan, and T. Fahringer Overhead analysis of scientific workflows in grid environments IEEE Transactions on Parallel and Distributed Systems 19 2008 378 393
25. M. Wieczorek, R. Prodan, and T. Fahringer Scheduling of scientific workflows in the ASKALON grid environment SIGMOD Record 34 2005 56 62 (Pubitemid 43152533)
26. C. Baru, R. Moore, A. Rajasekar, M. Wan, The SDSC storage resource broker, in: IBM Centre for Advanced Studies Conference, Toronto, Canada, 1998, pp. 112.
27. D. Churches, G. Gombas, A. Harrison, J. Maassen, C. Robinson, M. Shields, I. Taylor, and I. Wang Programming scientific and distributed workflow with Triana services Concurrency and Computation: Practice and Experience, vol. 18 2006 1021 1037
28. A. Chervenak, E. Deelman, I. Foster, L. Guy, W. Hoschek, A. Iamnitchi, C. Kesselman, P. Kunszt, M. Ripeanu, B. Schwartzkopf, H. Stockinger, K. Stockinger, B. Tierney, Giggle: A framework for constructing scalable replica location services, in: ACM/IEEE Conference on Supercomputing, Baltimore, Maryland, 2002, pp. 117.
29. R. Buyya, S. Venugopal, The gridbus toolkit for service oriented grid and utility computing: An overview and status report, in: IEEE International Workshop on Grid Economics and Business Models, Seoul, 2004, pp. 1966.
30. S. Venugopal, R. Buyya, L. Winton, A grid service broker for scheduling distributed data-oriented applications on global grids, in: 2nd Workshop on Middleware in Grid Computing, Toronto, Canada, 2004, pp. 7580.
31. S. Venugopal, R. Buyya, and K. Ramamohanarao A taxonomy of data grids for distributed data sharing, management, and processing ACM Computing Survey 38 2006 1 53
32. S. Doraimani, and A. Iamnitchi File grouping for scientific data management: Lessons from experimenting with real traces Proceedings of the 17th International Symposium on High Performance Distributed Computing 2008 ACM Boston, MA, USA 153 164
33. G. Fedak, H. He, F. Cappello, BitDew: A programmable environment for large-scale data management and distribution, in: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, Austin, Texas, 2008, pp. 112.
34. A. Chervenak, E. Deelman, M. Livny, M.-H. Su, R. Schuler, S. Bharathi, G. Mehta, K. Vahi, Data placement for scientific applications in distributed environments, in: 8th Grid Computing Conference, 2007, pp. 267274.
35. G. Singh, K. Vahi, A. Ramakrishnan, G. Mehta, E. Deelman, H. Zhao, R. Sakellariou, K. Blackburn, D. Brown, S. Fairhurst, D. Meyers, G.B. Berriman, J. Good, and D.S. Katz Optimizing workflow data footprint Scientific Programming 15 2007 249 268
36. T. Kosar, and M. Livny A framework for reliable and efficient data placement in distributed computing systems Journal of Parallel and Distributed Computing 65 2005 1146 1157
37. A.K. Jain, M.N. Murty, and P.J. Flynn Data clustering: A review ACM Computing Survey 31 1999 264 323
38. S. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. O'Callaghan Clustering data streams: Theory and practice IEEE Transactions on Knowledge and Data Engineering 15 2003 515 528
39. ATNF Parkes Swinburne Recorder, http://astronomy.swin.edu.au/pulsar/? topic=apsr (accessed 25.11.09).
40. M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R.H. Katz, A. Konwinski, G. Lee, D.A. Patterson, A. Rabkin, I. Stoica, M. Zaharia, Above the clouds: A Berkeley view of cloud computing, University of California at Berkeley, http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf, Technical Report UCB/EECS-2009-28 (accessed 25.11.09).
41. R. Grossman, Y. Gu, M. Sabala, and W. Zhang Compute and storage clouds using wide area high performance networks Future Generation Computer Systems 2008 179 183
42. Amazon elastic computing cloud, http://aws.amazon.com/ec2/ (accessed 25.11.09).
43. H. Liu, D. Orban, GridBatch: Cloud computing for large-scale data-intensive batch applications, in: Eighth IEEE International Symposium on Cluster Computing and the Grid, 2008, pp. 295305.
44. J. Dean, and S. Ghemawat MapReduce: Simplified data processing on large clusters Communications of the ACM 51 2008 107 113
45. A. Matsunaga, M. Tsugawa, J. Fortes, CloudBLAST: Combining MapReduce and virtualization on distributed resources for bioinformatics applications, in: 4th IEEE International Conference on e-Science, 2008, pp. 222229.
46. M.T. Ozsu, and P. Valduriez Principles of Distributed Database Systems 1991 Prentice-Hall, Inc. Upper Saddle River, NJ, USA
47. E. Deelman, D. Gannon, M. Shields, and I. Taylor Workflows and e-Science: An overview of workflow system features and capabilities Future Generation Computer Systems 25 2009 528 540
48. S. Venugopal, and R. Buyya An SCP-based heuristic approach for scheduling distributed data-intensive applications on global grids Journal of Parallel and Distributed Computing 68 2008 471 481
49. Y. Yang, K. Liu, J. Chen, X. Liu, D. Yuan, H. Jin, An algorithm in SwinDeW-C for scheduling transaction-intensive cost-constrained cloud workflows, in: 4th IEEE International Conference on e-Science, 2008, pp. 374375.
50. J. Yan, Y. Yang, and G.K. Raikundalia SwinDeW A P2P-based decentralized workflow management system IEEE Transactions on Systems, Man and Cybernetics 36 Part A 2006 922 935
51. Y. Yang, K. Liu, J. Chen, J. Lignier, H. Jin, Peer-to-peer based grid workflow runtime environment of SwinDeW-G, in: IEEE International Conference on e-Science and Grid Computing, 2007, pp. 5158.
52. VMware, http://www.vmware.com/ (accessed 25.11.09).