A MapReduce workflow system for architecting scientific data intensive applications

Proceedings - International Conference on Software Engineering

Volumn , Issue , 2011, Pages 57-63

  Nguyen, Phuong ^a   Halem, Milton ^a  

^a University of Marylandw Baltimore County   (United States)

Author keywords

cloud computing; mapreduce applications; scheduling; workflow scheduling; workflow system

Indexed keywords

CLIMATE ANALYSIS; CLIMATE DATA; COMPLEX DATA; GRAPHIC USER INTERFACE; GRIDDING; JOB SCHEDULER; MAP-REDUCE; PARALLEL METHOD; PROGRAMMING MODELS; RUNTIME SUPPORT SYSTEMS; SATELLITE DATA; SCIENTIFIC DATA; SCIENTIFIC WORKFLOWS; WORK-FLOW SYSTEMS; WORK-FLOWS; WORKFLOW SCHEDULING;

CLOUD COMPUTING; COMPUTER PROGRAMMING; COMPUTER SYSTEMS; DATA REDUCTION; SCHEDULING; SOFTWARE ENGINEERING;

DATA PROCESSING;

EID: 79960610562     PISSN: 02705257     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1985500.1985510     Document Type: Conference Paper

References (22)

1. Kouzes, Richard T. Anderson, Gordon A. Elbert, Stephen T. Gorton, Ian Gracio. The Changing Paradigm of Data-Intensive Computing. IEEE Computer Society Press Volume 42 , Issue 1, January 2009.
2. Gideon Juve, Ewa Deelman, et al. Scientific Workflow Applications on Amazon EC2. Workshop on Cloud-based Services and Applications in conjunction with 5th IEEE International Conference on e-Science (e-Science 2009), Oxford UK, 11, 2009.
3. Wei Lu, Jared Jackson, and Roger Barga. AzureBlast: A Case Study of Developing Science Applications on the Cloud. ScienceCloud 2010.
4. Michael C. Schatz et al. CloudBurst: highly sensitive read mapping with MapReduce. Bioinformatics. June 1; 25(11): 1363-1369, 2009.
5. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, pages 137-150, 2004.
6. Ewa Deelman, Gurmeet Singh, Mei-Hui Su et al. Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems. Scientific Programming Journal, Vol 13(3), pages 219-237, 2005.
7. Trident: Scientific Workflow Workbench for Oceanography. http://www.microsoft.com/mscorp/tc/trident.mspx
8. Oozie http://yahoo.github.com/oozie/
9. Jianwu Wang, Daniel Crawl, Ilkay Altintas. Kepler + Hadoop : A General Architecture Facilitating Data-Intensive Applications in Scientific Workflow Systems. WORKS09 ACM 2009, ISBN 978-1-60558-717-2.
10. X. Fei, S. Lu, and C. Lin. A mapreduce-enabled scientific workflow composition framework. WebServices, IEEE International Conference on, vol. 0, pp. 663-670, 2009.
11. Q. Chen, L. Wang, and Z. Shang. MRGIS: A MapReduce-Enabled high performance workflow system for GIS. In the 3rd International Workshop on Scientific Workflows and Business Workflow Standards in e-Science (SWBES). IEEE Press, December, 2008.
12. http://hadoop.apache.org/
13. Yunhong Gu, Robert Grossman. Sector and Sphere: The Design and Implementation of a High Performance Data Cloud. Theme Issue of the Philosophical Transactions of the Royal Society A: Crossing Boundaries: Computational Science, E-Science and Global E-Infrastructure, vol. 367 no. 1897 2429-2445, 28 June 2009.
14. Yunhong Gu and Robert Grossman. Lessons Learned From a Year's Worth of Benchmarks of Large Data Clouds. 2nd Workshop on Many-Task Computing on Grids and Supercomputers Portland, Oregon 11, 2009.
15. Ann Chervenak, Ewa Deelman, Miron Livny, Mei-Hui Su, Rob Schuler, Shishir Bharathi, Gaurang Mehta, Karan Vahi. Data Placement for Scientific Applications in Distributed Environments. Proceedings of Grid Conference, Austin, Texas, September 2007.
16. M. Halem, -D. Chapman-, P. Nguyen, C.Tilmes, Y. Yelena, N. Most, K. Stewart. Service Oriented Atmospheric Radiances (SOAR): Services for Gridding and Analysis of Multi-Sensor Satellite Radiance Data for Climate Studies. IEEE Trans. of Geoscience and Remote Sensing Journal. Vol.47, No. 1 Jan. P.114-122, 2009.
17. P.Nguyen, D. Chapman, M.Halem. Towards Producing a 40 Year Earth Science Data Record of IR Radiances. American Geophysical Union, San Francisco, San Francisco, Dec.14-18, 2009.
18. G. Leptoukh. NASA remote sensing data in earth sciences: processing, archiving, distribution, applications at the GES DISCISRSE. http://www.isprs. org/publications/related/ISRSE/html/papers/217.pdf .
19. D. J. Goodman. Introduction and evaluation of Martlet: a scientific workflow language for abstracted parallelization. In WWW, pages 983-992, 2007.
20. Mihcael Wilde, Ian Foster, Kamil Iskra, Pete Beckman, Zhao Zhang, Allan Espinosa, Mihael Hategan, Ben Clifford, Ioan Raicu. Parallel Scripting for Applications at the Petascale and Beyond Computer. Vol. 42, No. 11 2009.
21. N. Golpayegani, M. Halem. Cloud Computing for Satellite Data Processing on High End Compute Clusters. IEEE International Conference on Cloud Computing, 2009.
22. Jie Li, Deb Agarwal, Marty Humphrey, Catharine van Ingen, Keith Jackson, and Youngryel Ryu. eScience in the Cloud: A MODIS Satellite Data Reprojection and Reduction Pipeline in the Windows Azure Platform. In IPDPS, IEEE, 2010.