A MapReduce-enabled scientific workflow framework with optimization scheduling algorithm

Parallel and Distributed Computing, Applications and Technologies, PDCAT Proceedings

Volumn , Issue , 2012, Pages 599-604

  Tang, Zhuo ^a   Liu, Min ^a   Li, Kenli ^a   Xu, Yuming ^a  

^a HUNAN UNIVERSITY   (China)

Author keywords

Hadoop; MapReduce; MRWS; scheduling; workflow

Indexed keywords

HADOOP; HETEROGENEOUS CLUSTERS; HETEROGENEOUS ENVIRONMENTS; MAP-REDUCE; MRWS; OPTIMIZATION SCHEDULING; SCIENTIFIC WORKFLOWS; WORKFLOW;

ENTERPRISE RESOURCE MANAGEMENT; SCHEDULING;

SCHEDULING ALGORITHMS;

EID: 84884639365     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/PDCAT.2012.22     Document Type: Conference Paper

References (12)

1. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters," in Proceedings of the 6th Symposium on Operating Systems Design and Implementation( OSDI), 2004, p. 137C150.
2. A. Barker and J. Van Hemert, "Scientific workflow: a survey and research directions," in Proceedings of the 7th international conference on Parallel processing and applied mathematics. Springer-Verlag, 2007, pp. 746-753.
3. Q. Chen, L. Wang, and Z. Shang, "Mrgis: A mapreduceenabled high performance workflow system for gis," in e-Science, 2008. eScience'08. IEEE Fourth International Conference on. Ieee, 2008, pp. 646-651.
4. P. Nguyen and M. Halem, "A mapreduce workflow system for architecting scientific data intensive applications," in Proceeding of the 2nd international workshop on Software engineering for cloud computing. ACM, 2011, pp. 57-63.
5. J. Polo, D. Carrera, Y. Becerra, V. Beltran, J. Torres, and E. Ayguadé, "Performance management of accelerated mapreduce workloads in heterogeneous clusters," in 39th International Conference on Parallel Processing (ICPP2010), 2010.
6. J. Polo, D. Carrera, Y. Becerra, M. Steinder, and I. Whalley, "Performance-driven task co-scheduling for mapreduce environments," in Network Operations and Management Symposium (NOMS), 2010 IEEE. IEEE, 2010, pp. 373-380.
7. H. Topcuouglu, S. Hariri, and M.-y. Wu, "Performanceeffective and low-complexity task scheduling for heterogeneous computing," IEEE Trans. Parallel Distrib. Syst., vol. 13, no. 3, pp. 260-274, 2002.
8. J. Wang, D. Crawl, and I. Altintas, "Kepler+ hadoop: A general architecture facilitating data-intensive applications in scientific workflow systems," in Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science. ACM, 2009, p. 12.
9. "Yahoo, oozie. http://yahoo. github. com/oozie/. "
10. T. White, Hadoop: The Definitive Guide. O'Reilly Media, Inc., 2009.
11. A. Barker, J. B. Weissman, and J. I. Hemert, "The circulate architecture: avoiding workflow bottlenecks caused by centralised orchestration," Cluster Computing, vol. 12, no. 2, pp. 221-235, 2009.
12. J. C. Jacob, D. S. Katz, T. Prince, G. B. Berriman, J. C. Good, and A. C. Laity, "The montage architecture for gridenabled science processing of large, distributed datasets," in in Proceedings of the Earth Science Technology Conference (ESTC), 2004.