Supporting HPC analytics applications with access patterns using data restructuring and data-centric scheduling techniques in mapreduce

IEEE Transactions on Parallel and Distributed Systems

Volumn 24, Issue 1, 2013, Pages 158-169

  Sehrish, Saba ^a   MacKey, Grant ^b   Shang, Pengju ^c,d   Wang, Jun ^c   Bent, John ^d  

^a NORTHWESTERN UNIVERSITY   (United States)

^b UNIVERSITY OF CENTRAL FLORIDA   (United States)

^c University of Central Florida   (United States)

^d GNSS Technology for Rail Transportation   (United States)

Author keywords

data intensive systems; HPC analytics framework; MapReduce; scheduling

Indexed keywords

ACCESS PATTERNS; DATA ANALYTICS; DATA CENTRIC; DATA LOCALITY; DATA RESTRUCTURING; DATA SEMANTICS; DATA SETS; DATA-INTENSIVE COMPUTING; DATA-INTENSIVE SYSTEMS; EXPLOSIVE GROWTH; FILE SYSTEMS; HIGH PERFORMANCE COMPUTING (HPC); HPC ANALYTICS FRAMEWORK; I/O OPERATIONS; LOGICAL SEMANTICS; MAP-REDUCE; MULTIPLE DATA; NUMBER OF DATUM; REAL APPLICATIONS; SCHEDULING TECHNIQUES; SCIENTIFIC APPLICATIONS; STORAGE SYSTEMS; THROUGHPUT IMPROVEMENT; WRITE OPERATIONS;

SCHEDULING; SEMANTICS;

DIGITAL STORAGE;

EID: 84870939669     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2012.88     Document Type: Article

References (42)

1. http://hadoop. apache. org/core/, 2012.
2. https://issues. apache. org/jira/browse/hadoop-1687, 2012.
3. http://usqcd. jlab. org/usqcd-docs/adat/, 2012.
4. http://www. bnl. gov/rhic, 2012.
5. http://www. cisl. ucar. edu/dir/09seminars/roskies20090130. ppt, 2011.
6. http://www. hdfgroup. org/hdf5/, 2011.
7. http://www. hdfgroup. org/hdf5/phdf5/, 2012.
8. http://www. soe. ucsc. edu/share/technical-reports/2011/ucscsoe-11-04. pdf, 2012.
9. http://www. unidata. ucar. edu/software/netcdf/, 2012.
10. MPI-2: Extensions to the message-passing interface, http://parallel. ru/docs/parallel/mpi2, July 1997.
11. J. Bent, G. Gibson, G. Grider, B. McClelland, P. Nowoczynski, J. Nunez, M. Polte, and M. Wingate, "PLFS: A Checkpoint Filesystem for Parallel Applications, " Proc. ACM/IEEE Conf. Supercomputing, Nov. 2009.
12. D. Borthaku, "The Hadoop Distributed File System: Architecture and Design.
13. T. H. Cormen, C. Stein, R. L. Rivest, and C. E. Leiserson, Introduction to Algorithms. McGraw-Hill Higher Education, 2001.
14. J. Dean, "Experiences with MapReduce, An Abstraction for Large-Scale Computation, " Proc. 15th Int'l Conf. Parallel Architectures and Compilation Techniques (PACT '06), p. 1, 2006. (Pubitemid 46601075)
15. J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters, " Proc. Sixth Conf. Symp. Operating Systems Design and Implementation (OSDI '04), pp. 10-10, 2004.
16. J. Ekanayake, S. Pallickara, and G. Fox, "Mapreduce for Data Intensive Scientific Analyses, " Proc. IEEE Fourth Int'l Conf. eScience (eScience '08), pp. 277-284, 2008.
17. J. Ekanayake, T. Gunarathne, G. Fox, A. S. Balkir, C. Poulain, N. Araujo, and R. Barga, "Dryadlinq for Scientific Analyses, " Proc. IEEE Fifth Int'l Conf. e-Science (E-SCIENCE '09), pp. 329-336, 2009.
18. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google File System, " Proc. 19th ACM Symp. Operating Systems Principles (SOSP '03), pp. 29-43, 2003. (Pubitemid 40929686)
19. W. Gropp, R. Thakur, and E. Lusk, Using MPI-2: Advanced Features of the Message Passing Interface. MIT Press, 1999.
20. C. He, Y. Lu, and D. Swanson, "Matchmaking: A New MapReduce Scheduling Technique, " Proc. IEEE Third Int'l Conf. Cloud Computing Technology and Science (CloudCom), pp. 40-47, 2011.
21. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks, " Proc. ACM SIGOPS/EuroSys European Conf. Computer Systems (EuroSys '07), pp. 59-72, 2007. (Pubitemid 47281575)
22. M. Isard and Y. Yu, "Distributed Data-Parallel Computing Using a High-Level Programming Language, " Proc. 35th SIGMOD Int'l Conf. Management of Data (SIGMOD '09), pp. 987-994, 2009.
23. Y. C. Kwon, D. Nunley, J. P. Gardner, M. Balazinska, B. Howe, and S. Loebman, "Scalable Clustering Algorithm for N-Body Simulations in a Shared-Nothing Cluster, " technical report, Univ. of Washington, 2009.
24. J. Li, W. K. Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale, "Parallel NetCDF: A High-Performance Scientific I/O Interface, " Proc. ACM/IEEE Conf. Supercomputing, p. 39, Nov. 2003.
25. X. Liu, J. Han, Y. Zhong, C. Han, and X. He, "Implementing WebGIS on Hadoop: A Case Study of Improving Small File I/O Performance on HDFS, " Proc. IEEE Int'l Conf. Cluster Computing and Workshops (CLUSTER '09), 2009.
26. A. Matsunaga, M. Tsugawa, and J. Fortes, "Cloudblast: Combining Mapreduce and Virtualization on Distributed Resources for Bioinformatics Applications, " Proc. IEEE Fourth Int'l Conf. eScience (ESCIENCE '08), pp. 222-229, 2008.
27. L. Meyer, J. Annis, M. Wilde, M. Mattoso, and I. Foster, "Planning Spatial Workflows to Optimize Grid Performance, " Proc. ACM Symp. Applied Computing (SAC '06), pp. 786-790, 2006. (Pubitemid 44758888)
28. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig Latin: A Not-So-Foreign Language for Data Processing, " Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '08), pp. 1099-1110, 2008.
29. I. Raicu, I. T. Foster, Y. Zhao, P. Little, C. M. Moretti, A. Chaudhary, and D. Thain, "The Quest for Scalable Support of Data-Intensive Workloads in Distributed Systems, " Proc. 18th ACM Int'l Symp. High Performance Distributed Computing (HPDC '09), pp. 207-216, 2009.
30. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, "Falkon: A Fast and Light-Weight Task Execution Framework, " Proc. ACM/IEEE Conf. Supercomputing (SC '07), pp. 1-12, 2007.
31. I. Raicu, Y. Zhao, I. T. Foster, and A. Szalay, "Accelerating Large-Scale Data Exploration through Data Diffusion, " Proc. Int'l Workshop Data-Aware Distributed Computing (DADC '08), pp. 9-18, 2008.
32. K. Ranganathan and I. Foster, "Decoupling Computation and Data Scheduling in Distributed Data-Intensive Applications, " Proc. IEEE 11th Int'l Symp. High Performance Distributed Computing (HPDC '02), p. 352, 2002.
33. E. Santos-Neto, W. Cirne, F. Brasileiro, A. Lima, R. Lima, and C. Grande, "Exploiting Replication and Data Reuse to Efficiently Schedule Data-Intensive Applications on Grids, " Proc. 10th Workshop Job Scheduling Strategies for Parallel Processing, pp. 210-232, 2004.
34. M. C. Schatz, "Cloudburst: Highly Sensitive Read Mapping with MapReduce, " Bioinformatics, vol. 25, no. 11, pp. 1363-1369, 2009.
35. X. Shen and A. Choudhary, "DPFS: A Distributed Parallel File System, " Proc. Int'l Conf. Parallel Processing, vol. 0:0533, 2001.
36. V. Springel, S. D. M. White, A. Jenkins, C. S. Frenk, N. Yoshida, L. Gao, J. Navarro, R. Thacker, D. Croton, J. Helly, J. A. Peacock, S. Cole, P. Thomas, H. Couchman, A. Evrard, J. Colberg, and F. Pearce, "Simulations of the Formation, Evolution and Clustering of Galaxies and Quasars, " Nature, vol. 435, no. 70422, pp. 629-636, June 2005. (Pubitemid 40825503)
37. R. Thakur, W. Gropp, and E. Lusk, "Data Sieving and Collective I/O in ROMIO, " Proc. Seventh Symp. Frontiers of Massively Parallel Computation (FRONTIERS '99), pp. 182-189, 1999.
38. S. Viswanathan, B. Veeravalli, D. Yu, and T. G. Robertazzi, "Design and Analysis of a Dynamic Scheduling Strategy with Resource Estimation for Large-Scale Grid Systems, " Proc. IEEE/ACM Fifth Int'l Workshop Grid Computing (GRID '04), pp. 163-170, 2004. (Pubitemid 40752138)
39. S. Yamagiwa and K. Wada, "Performance Impact on Resource Sharing among Multiple CPU-and GPU-Based Applications, " Int'l J. Parallel, Emergent and Distributed Systems, vol. 26, no. 4, pp. 313-329, 2011.
40. Y. Yu, M. Isard, D. Fetterly, M. Budiu, I. Erlingsson, P. K. Gunda, and J. Currey, "Dryadlinq: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language, " Proc. Eighth USENIX Conf. Operating Systems Design and Implementation (OSDI), pp. 1-14, 2008.
41. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica, "Improving Mapreduce Performance in Heterogeneous Environments, " Proc. Eighth USENIX Conf. Operating Systems Design and Implementation (OSDI '08), pp. 29-42, 2008.
42. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, and M. Wilde, "Swift: Fast, Reliable, Loosely Coupled Parallel Computation, " Proc. IEEE Congress Services, pp. 199-206, July 2007.