Pilot-MapReduce: An extensible and flexible MapReduce implementation for distributed data

MapReduce '12 - 3rd International Workshop on MapReduce and Its Applications

Volumn , Issue , 2012, Pages 17-24

  Mantha, Pradeep Kumar ^a   Luckow, Andre ^a   Jha, Shantenu ^b  

^a LOUISIANA STATE UNIVERSITY   (United States)

^b RUTGERS UNIVERSITY   (United States)

Author keywords

BWA; Distributed computing; Genome Sequence Alignment; MapReduce; Pilot job and data; Simple API for Grid Applications (SAGA)

Indexed keywords

BWA; GENOME SEQUENCES; GRID APPLICATIONS; MAP-REDUCE; PILOT JOB AND DATA;

ABSTRACTING; COMPUTER PROGRAMMING; DATA TRANSFER; DISTRIBUTED COMPUTER SYSTEMS; INFORMATION MANAGEMENT;

DATA REDUCTION;

EID: 84863971193     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2287016.2287020     Document Type: Conference Paper

References (22)

1. Apache Hadoop. http://hadoop.apache.org/, 2012.
2. D. Bernholdt, S. Bharathi, and D. B. et al. The earth system grid. Proc. of the IEEE, 93(3):485-495, 2005.
3. D. Borthakur. The hadoop distributed file system: Architecture and design. Technical report, Apache Software Foundation, 2007.
4. M. Cardosa, C. Wang, A. Nangia, A. Chandra, and J. Weissman. Exploring mapreduce efficiency with highly-distributed data. In Proceedings of 2nd international workshop on MapReduce and its applications, pages 27-34, New York, NY, USA, 2011. ACM.
5. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. In Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6, pages 10-10, Berkeley, CA, USA, 2004. USENIX Association.
6. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox. Twister: a runtime for iterative mapreduce. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10, pages 810-818, New York, NY, USA, 2010. ACM.
7. I. Foster. Globus online: Accelerating and democratizing science through cloud-based services. IEEE Internet Computing, 15:70-73, 2011.
8. FutureGrid: An Experimental, High-Performance Grid Test-bed. https://portal.futuregrid.org/, 2012.
9. S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system. SIGOPS Oper. Syst. Rev., 37:29-43, 2003.
10. Y. Gu and R. L. Grossman. Sector and sphere: the design and implementation of a high-performance data cloud. Philosophical Transactions of the Royal Society -Series A: Mathematical, Physical and Engineering Sciences, 367(1897):2429-2445, 2009.
11. T. Hey, S. Tansley, and K. Tolle, editors. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, Redmond, Washington, 2009.
12. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fet-terly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev., 41:59-72, March 2007.
13. S. Jha, J. D. Blower, N. C. Hong, S. Dobson, D. S. Katz, A. Luckow, O. Rana, Y. Simmhan, and J. Weiss-man. 3dpas: Distributed dynamic data-intensive programming abstractions and systems. 2011.
14. H. Li and R. Durbin. Fast and accurate long-read alignment with burrows wheeler transform. Bioinformatics, 26:589-595, March 2010.
15. A. Luckow, M. Santcroos, O. Weider, A. Merzky, S. Maddineni, and S. Jha. Towards a common model for pilot-jobs. In Proceedings of The International ACM Symposium on High-Performance Parallel and Distributed Computing, 2012.
16. Y. Luo, Z. Guo, Y. Sun, B. Plale, J. Qiu, and W. W. Li. A hierarchical framework for cross-domain mapreduce execution. In Proceedings of the 2nd international workshop on emerging computational methods for the life sciences, pages 15-22, NY, NY, USA, 2011. ACM.
17. A. Merzky. SAGA API Extension: Advert API. OGF Document Series 177, http://www.gridforum.org/documents/GFD.177.pdf, 2011.
18. Picard. http://picard.sourceforge.net, 2012.
19. L. Pireddu, S. Leo, and G. Zanetti. SEAL: a distributed short read mapping and duplicate removal tool. Bioin-formatics (Oxford, England), 27(15):2159-2160, 2011.
20. SAGA BigJob. https://github.com/saga-project/BigJob/wiki, 2012.
21. S. Sehgal, M. Erdelyi, A. Merzky, and S. Jha. Understanding application-level interoperability: Scaling-out mapreduce over high-performance grids and clouds. Future Gener. Comput. Syst., 27:590-599, May 2011.
22. WebHDFS REST API. http://hadoop.apache.org/common/docs/r1.0.0/webhdfs. html, 2012.