Rethinking data management for big data scientific workflows

Proceedings - 2013 IEEE International Conference on Big Data, Big Data 2013

Volumn , Issue , 2013, Pages 27-35

  Vahi, Karan ^a   Rynge, Mats ^a   Juve, Gideon ^a   Mayani, Rajiv ^a   Deelman, Ewa ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

cloud; data management; data staging site; object stores; Pegasus; Pegasus Lite; workflows

Indexed keywords

CLOUDS; CODES (SYMBOLS); DIGITAL STORAGE; FILE ORGANIZATION; LARGE DATASET; WORK SIMPLIFICATION;

DATA STAGING; EXECUTION ENVIRONMENTS; OBJECT STORE; PEGASUS; SCIENTIFIC WORKFLOWS; WORK-FLOWS; WORKFLOW MANAGEMENT SYSTEMS; WORKFLOW-BASED APPLICATIONS;

INFORMATION MANAGEMENT;

EID: 84893330457     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2013.6691724     Document Type: Conference Paper

References (46)

1. XSEDE-Extreme Science and Engineering Environment," 2012, Available: http://www.xsede.org.
2. Catlett Charles, "TeraGrid: Analysis of Organization, System Architecture, and Middleware Enabling New Types of Applications.," IOS Press, 2007.
3. Amazon Web Services," Available: http://aws.amazon.com/.
4. J. Diaz et al., "FutureGrid Image Repository: A Generic Catalog and Storage System for Heterogeneous Virtual Machine Images," in Cloud Computing Technology and Science (CloudCom), 2011 IEEE Third International Conference on, 2011, pp. 560-564.
5. FutureGrid: A distributed testbed, exploring possibilities with Clouds, Grids and High Performance Computing," 2012, Available: https://portal.futuregrid.org/.
6. M. Altunay et al., "A Science Driven Production Cyberinfrastructure\ -The Open Science Grid," J. Grid Comput., vol. 9, no. 2, pp. 201-218, Jun. 2011.
7. E. Deelman et al., "Pegasus: A framework for mapping complex scientific workflows onto distributed systems," Sci. Program., vol. 13, no. 3, pp. 219-237, Jul. 2005.
8. Amazon Simple Storage Service," Available: http://aws.amazon.com/s3/ .
9. W. Allcock et al., "The Globus Striped GridFTP Framework and Server," Washington, DC, USA, 2005, p. 54-.
10. Abadie Lana and Badin Paolo, "Storage Resource Manager version 2.2: design, implementation, and testing experience.," Proceedings of International Conference on Computing in High Energy and Nuclear Physics (CHEP 07), 2007.
11. D. Hünich and R. Müller-Pfefferkorn, "Managing large datasets with iRODS-A performance analysis," in Computer Science and Information Technology (IMCSIT), Proceedings of the 2010 International Multiconference on, 2010, pp. 647-654.
12. A. Rajasekar et al., "iRODS Primer: Integrated Rule-Oriented Data System," Synthesis Lectures on Information Concepts, Retrieval, and Services, vol. 2, no. 1, pp. 1-143, Jan. 2010.
13. J.C. Jacob et al., "Montage: A grid portal and software toolkit for science-grade astronomical image mosaicking," CoRR, vol. abs/1005.4454, 2010.
14. R. Graves et al., "CyberShake: A Physics-Based Seismic Hazard Model for Southern California," Pure and Applied Geophysics, vol. 168, no. 3, pp. 367-381, 2011.
15. G. Juve et al., "An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2," Journal of Grid Computing, vol. 10, no. 1, pp. 5-21, 2012.
16. Amazon S3 Pricing," Available: http://aws.amazon.com/s3/pricing/.
17. E. Deelman et al., "Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: The CyberShake Example," presented at the e-Science, 2006, p. 14.
18. S. Callaghan et al., "Metrics for heterogeneous scientific workflows: A case study of an earthquake science application," IJHPCA, vol. 25, no. 3, pp. 274-285, 2011.
19. D. Thain et al., "Distributed computing in practice: the Condor experience," Concurrency-Practice and Experience, vol. 17, no. 2-4, pp. 323-356, 2005.
20. I.T. Foster et al., "The Anatomy of the Grid: Enabling Scalable Virtual Organizations," IJHPCA, vol. 15, no. 3, pp. 200-222, 2001.
21. A.L. Chervenak et al., "Wide area data replication for scientific collaborations," IJHPCN, vol. 5, no. 3, pp. 124-134, 2008.
22. Condor DAGMan (Directed Acyclic Graph Manager)," Available: http://research.cs.wisc.edu/condor/dagman/.
23. G. Singh et al., "Optimizing workflow data footprint," Scientific Programming, vol. 15, no. 4, pp. 249-268, 2007.
24. A. Ramakrishnan et al., "Scheduling Data-IntensiveWorkflows onto Storage-Constrained Distributed Resources," presented at the CCGRID, 2007, pp. 401-409.
25. G. Juve et al., "Experiences with resource provisioning for scientific workflows using Corral," Sci. Program., vol. 18, no. 2, pp. 77-92, Apr. 2010.
26. I. Sfiligoi, "glideinWMS-A generic pilot-based workload management system," Journal of Physics: Conference Series, vol. 119, no. 6, p. 062044, Jul. 2008.
27. Kraken System Specifications," Available: http://www.nics.tennessee. edu/computing-resources/kraken/.
28. Amazon EC2 Instance Types," Available: http://aws.amazon.com/ec2/ instance-types/.
29. J.S. Voeckler et al., "Kickstarting remote applications," in 2nd International Workshop on Grid Computing Environments, 2006.
30. K. Kaufmann et al., "Practically Useful: What the Rosetta Protein Modeling Suite Can Do for You," Biochemistry, vol. 49, no. 14, pp. 2987-2998, Mar. 2010.
31. Y. Zhao et al., "Swift: Fast, Reliable, Loosely Coupled Parallel Computation," in Services, 2007 IEEE Congress on, 2007, pp. 199-206.
32. I. Altintas et al., "Kepler: An extensible system for design and execution of scientific workflows," in Scientific and Statistical Database Management, 2004. Proceedings. 16th International Conference on, 2004, pp. 423-424.
33. A. Harrison et al., "WS-RF Workflow in Triana," Int. J. High Perform. Comput. Appl., vol. 22, no. 3, pp. 268-283, Aug. 2008.
34. P. Missier et al., "Taverna, Reloaded.," in SSDBM, 2010, vol. 6187, pp. 471-481.
35. J. Wang et al., "Kepler + Hadoop," 2009, pp. 1-8.
36. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," Commun. ACM, vol. 51, no. 1, pp. 107-113, Jan. 2008.
37. Apache Hadoop," Available: http://hadoop.apache.org/.
38. P. Andrews et al., "Massive High-Performance Global File Systems for Grid computing," in Proceedings of the 2005 ACM/IEEE conference on Supercomputing, Washington, DC, USA, 2005, p. 53-.
39. D. Thain et al., "Chirp: A practical global file system for cluster and grid computing," Journal of Grid Computing.
40. D. Thain and M. Livny, "Parrot: An Application Environment for Data-Intensive Computing," Journal of Parallel and Distributed Computing Practices, pp. 9-18, 2005.
41. I. Raicu et al., "Accelerating large-scale data exploration through data diffusion," in Proceedings of the 2008 international workshop on Data-Aware distributed computing, New York, NY, USA, 2008, pp. 9-18.
42. I. Raicu et al., "Falkon: A Fast and Light-weight tasK executiON framework," in Proceedings of the 2007 ACM/IEEE conference on Supercomputing, New York, NY, USA, 2007, pp. 43:1-43:12.
43. A.L. Chervenak et al., "Data placement for scientific applications in distributed environments," in GRID2007, 2007, pp. 267-274.
44. Sara Alspaugh et al., "Policy-Driven Data Management for Distributed Scientific Collaborations Using a Rule Engine," Austin, 2008.
45. A.L. Chervenak et al., "The Globus Replica Location Service: Design and Experience," IEEE Trans. Parallel Distrib. Syst., vol. 20, no. 9, pp. 1260-1272, 2009.
46. E. Deelman et al., "GriPhyN and LIGO, Building a Virtual Data Grid for Gravitational Wave Scientists," presented at the HPDC, 2002, p. 225-.