Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers

IEEE Transactions on Parallel and Distributed Systems

Volumn 27, Issue 1, 2016, Pages 185-196

  Xu, Cong ^a   Goldstone, Robin ^b   Liu, Zhuo ^a   Chen, Hui ^a   Neitzel, Bryon ^c   Yu, Weikuan ^a  

^a AUBURN UNIVERSITY   (United States)

^b LAWRENCE LIVERMORE NATIONAL LABORATORY   (United States)

^c INTEL CORPORATION   (United States)

Author keywords

Analytics Shipping; Hadoop; HPC; Lustre; MapReduce

Indexed keywords

SHIPS;

DATA DISTRIBUTION; HADOOP; HIGH PERFORMANCE COMPUTING SYSTEMS; LONG-TERM STORAGE; LUSTRE; MAP-REDUCE; PERSISTENT STORAGE; SCIENTIFIC APPLICATIONS;

DIGITAL STORAGE;

EID: 84961732848     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2015.2389262     Document Type: Article

References (31)

1. Apache Hadoop NextGen MapReduce (YARN). [Online]. Available: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/ hadoop-yarn-site/YARN.html, 2014.
2. Apache Hadoop Project. [Online]. Available: http://hadoop. apache.org/, 2014.
3. Center for Plasma Edge Simulation. [Online]. Available: http:// cims.nyu.edu/cpes/, 2014.
4. Chroot. [Online]. Available: https://wiki.archlinux.org/index. php/Change-Root, 2015.
5. Gordon: Data-Intensive Supercomputing. [Online]. Available: http://www.sdsc.edu/supercomputing/gordon/, 2015.
6. Kernel Based Virtual Machine. [Online]. Available: http://www. linux-kvm.org/page/Main-Page, 2014.
7. KMeans over MapReduce. [Online]. Available: http://cmj4.web. rice.edu/MapRedKMeans.html, 2014.
8. Lustre 2.0 operations manual. [Online]. Available: http://wiki. lustre.org/images/3/35/821-2076-10.pdf, 2011.
9. The Combustion Research Facility at Sandia National Laboratories. [Online]. Available: http://crf.sandia.gov/, 2014.
10. The Xen Project. [Online]. Available: http://wiki.xen.org/wiki/ Main-Page, 2015.
11. VMware, Inc. [Online]. Available: http://vmware.com/, 2015.
12. R. Ananthanarayanan, K. Gupta, P. Pandey, H. Pucha, P. Sarkar, M. Shah, and R. Tewari, "Cloud analytics: Do we really need to reinvent the storage stack," presented at the Proc. 1st USENIX Workshop Hot Topics Cloud Comput., San Diego, CA, USA, 2009.
13. D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, H. D. Simon, V. Venkatakrishnan, and S. K. Weeratunga, "The nas parallel benchmarks," Int. J. High Perform. Comput. Appl., vol. 5, no. 3, pp. 63-73, 1991.
14. J. C. Bennett, H. Abbasi, P.-T. Bremer, R. Grout, A. Gyulassy, T. Jin, S. Klasky, H. Kolla, M. Parashar, V. Pascucci, P. Pebay, D. Thompson, H. Yu, F. Zhang, and J. Chen, "Combining in-situ and in-transit processing to enable extreme-scale scientific analysis," in Proc. High Perform. Comput., Netw., Storage Anal. Int. Conf., Nov. 2012, pp. 1-9.
15. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters," in Proc. 6th Conf. Symp. Opear. Syst. Des. Implementation-Volume 6, Berkeley, CA, USA, 2004. USENIX Association, pp. 10-10.
16. D. Luan, S. Huang, and G. Gong, "Using Lustre with Apache Hadoop," Sun Microsystems Inc., 2009.
17. C. Maltzahn, E. Molina-Estolano, A. Khurana, A. J. Nelson, S. A. Brandt, and S. Weil, "Ceph as a scalable alternative to the hadoop distributed file system," login: USENIX Mag., 2010.
18. J. Piernas, J. Nieplocha, and E. J. Felix, "Evaluation of active storage strategies for the lustre parallel file system," IN Proceedings of the 2007 ACM/IEEE Conference on Supercomputing. New York, NY, USA: ACM, pp. 28:1-28:10, 2007.
19. K. Ren, Q. Zheng, S. Patil, and G. Gibson, "Indexfs: Scaling file system metadata performance with stateless caching and bulk insertion," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Piscataway, NJ, USA: IEEE Press, pp. 237-248, 2014.
20. H. Shan and J. Shalf, "Using IOR to analyze the I/O performance for HPC platforms," In Cray Users Group Meeting (CUG) 2007, Seattle, Washington, USA, 2007.
21. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The hadoop distributed file system," in Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies. Washington, DC, USA: IEEE Computer Society, pp. 1-10, 2010.
22. W. Tantisiriroj, S. Patil, and G. Gibson, "The crossing the chasm: Sneaking a parallel file system into hadoop," in Proc. Petascale Data Stroage Workshop, 2008.
23. W. Tantisiriroj, S. W. Son, S. Patil, S. J. Lang, G. Gibson, and R. B. Ross, "On the duality of data-intensive file system design: Reconciling hdfs and pvfs," in Proc. Int. Conf. High Perform. Comput., Netw., Storage Anal., 2011, pp. 67. .
24. D. Tiwari, S. Boboila, S. S. Vazhkudai, Y. Kim, X. Ma, P. J. Desnoyers, and Y. Solihin, "Active flash: Towards energy-efficient, in-situ data analytics on extreme-scale machines," in Proc. 11th USENIX Conf. File Storage Technol., Berkeley, CA, USA, 2013, pp. 119-132.
25. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B. Reed, and E. Baldeschwieler, "Apache hadoop yarn: Yet another resource negotiator," in Proc. 4th Annu. Symp. Cloud Comput., New York, NY, USA, 2013, pp. 5:1-5:16.
26. Y. Wang, X. Que, W. Yu, D. Goldenberg, and D. Sehgal, "Hadoop acceleration through network levitated merge," in Proc. 2011 Int. Conf. High Perform. Comput., Netw., Storage Anal., 2011, pp. 57.
27. B. Whitlock, J. M. Favre, and J. S. Meredith, "Parallel in situ coupling of simulation with a fully featured visualization system," in Proc. 11th Eurographics Conf. Parallel Graph. Vis., Aire-la-Ville, Switzerland, 2011, pp. 101-109.
28. E. H. Wilson, M. T. Kandemir, and G. Gibson, "Will they blend: Exploring big data computation atop traditional hpc nas storage," in Proc. Int. Conf. Distrib. Comput. Syst., 2014, pp. 524-534.
29. W. Yu, J. S. Vetter, R. S. Canon, and S. Jiang, "Exploiting lustre file joining for effective collective I/O," presented at the 7th Int. Conf. Cluster Comput. Grid, Rio de Janeiro, Brazil, May 2007.
30. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster computing with working sets," in Proc. 2Nd USENIX Conf. Hot Topics Cloud Comput., Berkeley, CA, USA, 2010, pp. 10-10.
31. F. Zhang, S. Lasluisa, T. Jin, I. Rodero, H. Bui, and M. Parashar, "In-situ feature-based objects tracking for large-scale scientific simulations," in Proc. SC Companion, IEEE Computer Society, 2012, pp. 736-740.