Makeflow: A portable abstraction for data intensive computing on clusters, clouds, and grids

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages

  Albrecht, Michael ^a   Donnelly, Patrick ^a   Bui, Peter ^b   Thain, Douglas ^a  

^a University of Notre Dame   (United States)

^b UNIVERSITY OF WISCONSIN EAU CLAIRE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA-INTENSIVE APPLICATION; DATA-INTENSIVE COMPUTING; DESCRIPTION LANGUAGES; HIGH-PERFORMANCE COMPUTING CLUSTERS; LANGUAGES AND SYSTEMS; LARGE SCALE DISTRIBUTED COMPUTING; PERFORMANCE CHARACTERISTICS; PHYSICAL ARCHITECTURE;

DIGITAL STORAGE;

BENCHMARKING;

EID: 84875179844     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2443416.2443417     Document Type: Conference Paper

References (35)

1. Filesystem in user space. http://sourceforge.net/projects/fuse.
2. S. Ahuja, N. Carriero, and D. Gelernter. Linda and friends. IEEE Computer, 19(8):26-34, August 1986.
3. S. Altschul, W. Gish, W. Miller, E. Myers, and D. Lipman. Basic local alignment search tool. Journal of Molecular Biology, 3(215):403-410, Oct 1990.
4. Andrew Lih and Erez Zadok. PGMAKE: A Portable Distributed Make System. Technical Report CUCS-035-95, Computer Science Department, Columbia University, 1994.
5. E. H. Baalbergen. Design and implementation of parallel make. COMPUTING SYSTEMS, 1:135-158, 1988.
6. P. Bui, L. Yu, A. Thrasher, R. Carmichael, I. Lanc, P. Donnelly, and D. Thain. Scripting distributed scientific workflows using Weaver. Concurrency and Computation: Practice and Experience, 2011.
7. R. Carmichael, P. Braga-Henebry, D. Thain, and S. Emrich. Biocompute 2.0: An Improved Collaborative Workspace for Data Intensive Bio-Science. Concurrency and Computation: Practice and Experience, 23(17):2305-2314, 2011.
8. Cascading. http://www.cascading.org/, 2010.
9. Condor Team. The directed acyclic graph manager. http://www.cs.wisc.edu/ condor/dagman 2002.
10. A. E. Darling, L. Carey, and W. chun Feng. The design, implementation, and evaluation of mpiblast. In In Proceedings of ClusterWorld 2003, 2003.
11. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large cluster. In Operating Systems Design and Implementation, 2004.
12. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, B. Berriman, J. Good, A. Laity, J. Jacob, and D. Katz. Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming Journal, 13(3), 2005.
13. S. Feldman. Make - A Program for Maintaining Computer Programs. Software: Practice and Experience, 9:255-265, November 1978.
14. W. Gentzsch. Sun grid engine: Towards creating a compute power grid. In CCGRID '01: Proceedings of the 1st International Symposium on Cluster Computing and the Grid, 2001.
15. S. Ghemawat, H. Gobioff, and S. Leung. The Google filesystem. In ACM Symposium on Operating Systems Principles, 2003.
16. Y. Gu and R. L. Grossman. 1 Sector and Sphere: The Design and Implementation of a High Performance Data Cloud.
17. Hadoop. http://hadoop.apache.org/2007.
18. B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang. Mars: a mapreduce framework on graphics processors. In Proceedings of the 17th international conference on Parallel architectures and compilation techniques, PACT '08, pages 260-269, New York, NY, USA, 2008. ACM.
19. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data parallel programs from sequential building blocks. In Proceedings of EuroSys, March 2007.
20. W. Lu, J. Jackson, and R. Barga. Azureblast: a case study of developing science applications on the cloud. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10, pages 413-420, New York, NY, USA, 2010. ACM.
21. S. Mullender, G. van Rossum, A. Tanenbaum, R. van Renesse, and H. van Staveren. Amoeba: A distributed operating system for the 1990s. IEEE Computer, 23(5):44-53, 1990. (Pubitemid 20712678)
22. T. Oinn, M. Greenwood, M. Addis, M. N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M. R. Pocock, M. Senger, R. Stevens, A. Wipat, and C. Wroe. Taverna: lessons in creating a workflow environment for the life sciences: Research articles. Concurr. Comput. : Pract. Exper., 18:1067-1100, August 2006. (Pubitemid 44258498)
23. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1099-1110, New York, NY, USA, 2008. ACM.
24. A. Polze. Using the object space: A distributed parallel make. 4th IEEE Workshop on Future Trends of Distributed Computing Systems, 1993.
25. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis. Evaluating mapreduce for multi-core and multiprocessor systems. In In HPCA âǍŹ07: Proceedings of the 13th International Symposium on High-Performance Computer Architecture, pages 13-24. IEEEComputer Society, 2007. (Pubitemid 47208148)
26. A. Regier. A Flexible Comparative Genomics Framework for Integrating Heterogeneous Sequence Data. PhDthesis, 2011.
27. K. Taura, T. Matsuzaki, M. Miwa, Y. Kamoshida, D. Yokoyama, N. Dun, T. Shibata, C. S. Jun, and J. Tsujii. Design and implementation of gxp make - a workflow system based on make. IEEE Conference on eScience, 2010.
28. D. Thain and M. Livny. Parrot: An Application Environment for Data-Intensive Computing. Scalable Computing: Practice and Experience, 6(3):9-18, 2005.
29. D. Thain and C. Moretti. Abstractions for Cloud Computing with Condor. In S. Ahson and M. Ilyas, editors, Cloud Computing and Software Services: Theory and Techniques, pages 153-171. CRC Press, 2010.
30. D. Thain, T. Tannenbaum, and M. Livny. Condor and the grid. In F. Berman, G. Fox, and T. Hey, editors, Grid Computing: Making the Global Infrastructure a Reality. John Wiley, 2003.
31. A. Thrasher, R. Carmichael, P. Bui, L. Yu, D. Thain, and S. Emrich. Taming Complex Bioinformatics Workflows with Weaver, Makeflow, and Starch. In Workshop on Workflows in Support of Large Scale Science, pages 1-6, 2010.
32. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive - a petabyte scale data warehouse using hadoop. In International Conference on Data Engineering, pages 996-1005, 2010.
33. A. Vahdat and T. Anderson. Transparent result caching. Proceedings of the 1998 USENIX Technical Conference, 1998.
34. L. Yu, C. Moretti, A. Thrasher, S. Emrich, K. Judd, and D. Thain. Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions. Journal of Cluster Computing, 13(3):243-256, 2010.
35. Y. Zhao, J. Dobson, L. Moreau, I. Foster, and M. Wilde. A notation and system for expressing and executing cleanly typed workflows on messy scientific data. In SIGMOD, 2005.