Improving I/O performance with adaptive data compression for big data applications

Proceedings - IEEE 28th International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2014

Volumn , Issue , 2014, Pages 1228-1237

  Zou, Hongbo ^a   Yu, Yongen ^b   Tang, Wei ^c   Chen, Hsuanwei Michelle ^d  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b ILLINOIS INSTITUTE OF TECHNOLOGY   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

^d SAN JOSE STATE UNIVERSITY   (United States)

Author keywords

Big Data; Compression; High end Computing; I O Bottlenecks; In situ Analytics

Indexed keywords

BANDWIDTH COMPRESSION; BIG DATA; COMPACTION; COST REDUCTION; MIDDLEWARE;

ADAPTIVE COMPRESSION; BIG DATA APPLICATIONS; COMPRESSION TECHNIQUES; COMPRESSION TECHNOLOGIES; DATA COMPRESSION ALGORITHMS; DATA TRANSITION; HIGH END COMPUTING; TRANSFER PERFORMANCE;

DATA COMPRESSION;

EID: 84918807309     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPSW.2014.138     Document Type: Conference Paper

References (39)

1. High-quality Data Compressor, bzip2, http://www.bzip.org.
2. Y. Chen, A. Ganapathi, and R. Katz, "To Compress or Not to Compress Compute vs. I/O Tradeoffs for MapReduce Energy Efficiency," Proc. The First ACM SIGCOMM Workshop on Green Networking, 2010, pp. 23-28.
3. Compression Utility, Gzip, http://www.gzip.org.
4. Message Passing Interface Forum. MPI-2: Extension to The Message-Passing Standard. 1997.
5. D. A. Lelewer, and D. S. Hirschberg, "Data Compression," Proc. ACM Computing Surverys (CSUR), 1987, pp. 261-296.
6. N. Fabian, K. Moreland, D. Thompson, A. C. Bauer, P. Marion, B. Geveci, M. Rasquin, and K. E. Jansen, "The ParaView Coprocessing Library: A Scalable, General Purpose In Situ Visualization Library," Proc. IEEE Symp. on Large-Scale Data Analysis and Visualization (LDAV2011), 2011, pp 89-96.
7. A. Gerndt, B. Hentschel, M. Wolter, T. Kuhlen, and C. Bischof, "VIRACOCHA: An Efficient Parallelization Framework for Large-Scale CFD Post-Processing in Virtual Environments," Proc. ACM/IEEE Conference on Supercomputing (SC04), 2004, pp. 50-61.
8. N. Hu, "Network Aware Data Transmission with Compression," Proc. The 4th Annual CMU Symposium on Computer Systems, 2001.
9. S. Hodson, S. Klasky, Q. Liu, J. Lofstead, N. Podhorszki, F. Zheng etc., "ADIOS 1.3.1 User's Manual," Oak Ridge National Laboratory, 2011, pp. 1-95.
10. I. Kontoyiannis, "Pattern Matching and Lossy Data Compression on Random Fields," Proc. IEEE Transactions on Information Theory, 2003, pp. 1047-1051.
11. E. Jeannot, B. Knutsson, M. Bjorkman, "Adaptive Online Data Compression," Proc. ACM Symp. on High-Performance Parallel and Distributed Computing (HPDC'02), 2002, pp. 379-388.
12. S. Laksh., N. Shah, S. Ethier, S. Klasky, R. Latham, R. Ross, and N. Samatova, "Compressing the Incompressible with ISABELA: In-situ Reduction of Spatio-temporal Data," Proc. International European Conference on Parallel and Distributed Computing (Euro-Par 2011), 2011, pp. 366-379.
13. M. Lin, S. S. Vazhkudai, A. R. Butt, F. Meng, X. Ma, Y. Kim, C. Engelmann, and G. Shipman, "Functional Partitioning to Optimize End-to-End Performance on Many-core Architectures," Proc. ACM/IEEE Conference on Supercomputing (SC10), 2010, pp. 1-12.
14. S. N. Laboratory, Cth shock physics, http://www.sandia.gov/CTH/. Jan. 2012.
15. J. F. Lofstead, F. Zheng, S. Klasky, "Adaptable, Metadata Rick I/O Methods for Portable High Performance I/O," Proc. IEEE International Parallel and Distributed Processing Symp (IPDPS'09), 2009, pp. 1-10.
16. B. Nicolae, "High Throughput Data-Compression for Cloud Storage," Proc. The 3rd International Conference on Data Management in Grid and Peer-to-Peer Systems (Globe'10), 2010, pp. 1-12.
17. R. Oldfield, S. Arunagiri, P. J. Teller, S. R. Seelam, M. R. Varela, R. Riesen, and P. C. Roth, "Modeling the Impact of Checkpoints on Next-generation Systems," Proc. IEEE Conference on Mass Storage System and Technologies (MSST 2007), 2007, pp. 30-46.
18. J. Piernas, J. Nieplocha, E. J. Felix, "Evaluation of Active Storage Strategies for the Lustre Parallel File System," Proc. ACM/IEEE Conference on Supercomputing (SC07), 2007, pp. 1-10.
19. K. Sayood, "Introduction to Data Compression," 3rd edition, Morgan Kaufmann, 2005.
20. A. Singh, P. Balaji, and W. Feng, "GePSeA: A General-Purpose Software Acceleration Framework for Lightweight Task Offloading," Proc. The 38th International Conference on Parallel Processing (ICPP), 2009, pp. 261-268.
21. M. Slawinska, T. Bode, H. Zou etc. "A Maya Use Case: Adaptable Scientific Workflows with ADIOS for General Relativistic Astrophysics," Proc. The Conference on Extreme Science and Engineering Discovery Environment, 2013.
22. XH Sun, C. Du, H. Zou, "V-mcs: A Configuration System for Virtual Machines," Proc. IEEE International Conference on Cluster Computing (Cluster'09), 2009.
23. W. Tang, J. Wilkening, N. Desai, W. Gerlach, A. Wilke, and F. Meyer, "A Scalable Data Analysis Platform for Metagenomics," in Proc. of IEEE Conference on Big Data, 2013.
24. W. Tang, Z. Lan, N. Desai, and Y. Yu etc., "Reducing Fragmentation on Torus-connected Supercomputers", Proc. IEEE International Parallel and Distributed Processing Symp (IPDPS'11), pp 828-839.
25. Paraview VisIt, OpenSource Scientific Visualization and Graphical Anaysis Tool, https://wci.llnl.gov/codes/visit
26. V. Vishwanath, M. Hereld, M. E. Papka, R. Hudson, G. Cal Jordan IV, and C. Daley, "In Situ Data Analysis and I/O Acceleration of FLASH Astrophysics Simulation on Leadership-Class System Using GLEAN," Proc. SciDAC, Journal of Physics: Conference Series, 2011.
27. B. Welton, D. Kimpe, J. Cope, C. Patrick, K. Iskra, and R. Ross, "Improving I/O Forwarding Throughput with Data Compression," Proc. IEEE International Conference on Cluster Computing (Cluster'11), 2011.
28. J. Wu, Z. Lan, and Y. Yu etc., "Performance Emulation of Cell-based AMR cosmology Simulations," Proc. IEEE International Conference on Cluster Computing (Cluster'11), 2011.
29. K. Wu, S. Ahern, E.W. Bethel, J. Chen, H. Childs, E. Cormier-Michel, C. Geddes, J. Gu, H. Hagen, B. Hamann, W. Koegler, J. Lauret, J. Meredith, P. Messmer, E. Otoo, V. Pere., A. Posk., P. O. Rubel, A. Sho., A. Sim, K. Stock., G. Weber, and W. Zhang, "FastBit: Interactively Searching Massive Data," Proc. SciDAC, Journal of Physics: Conference Series, 2009.
30. W. X. Wang, Z. Lin, W. M. Tang, W. W. Lee, S. Ethier, J. L. V. Lewandowski, G. Rewoldt, T. S. Hahm, and J. Manickam, "Gyrokinetic Simulation of Global Trubulent Tranport Properties in Tokamak Experiments," Proc. Physics of Plasmas, 2006, pp 59-64.
31. Yongen Yu, D. Rudd, Z. Lan, J. Wu etc., "Improving Parallel IO Performance of Cell-based AMR Cosmology Applications," Proc. IEEE International Parallel and Distributed Processing Symp (IPDPS'12), pp. 933-944, 2012.
32. Yongen Yu, J. Wu, and Z. Lan etc., "A Transparent Collective I/O Implementation," Proc. IEEE International Parallel and Distributed Processing Symp (IPDPS'13), 2013, pp 297-307, 2013.
33. F. Zheng, H. Zou, J. Cao, J. Dayal, T. Nugye, G. Eisenhauer, S. Klasky, "FlexIO: Location-flexible Execution of In-Situ Data Analytics for Large Scale Scientific Applications," Proc. IEEE International Parallel and Distributed Processing Symp (IPDPS'13), 2013, pp 320-331, 2013.
34. H. Zou, H. Jin, Z. Han etc. "A Virtual-Service-Domain based Bidding Algorithm for Resource Discovery in Computational Grid," Proc. IEEE/WIC/ACM International Conference on Web Intelligence, pp5-53, 2005.
35. H. Zou, H. Jin, Z. Han etc. "HRTC: Hybrid Resource Information Service Architecture based on GMA", Proc. IEEE International Conference on e-Business pp541-544, 2005.
36. H. Zou, K. Schwan, M. Slawinska, G. Eisenhauer, F. Zheng etc. "FlexQuery: An Online Query System for Interactive Remote Visual Data Exploration at Large Scale," Proc. IEEE International Conference on Cluster Computing (Cluster'13), 2013.
37. H. Zou, W. Wu, XH Sun etc. "An Evaluation of Parallel Optimization for OpenSolaris Network Stack," Proc. IEEE 35th Conference on Local Computer Networks (LCN), pp296-299, 2010.
38. H. Zou, XH Sun, etc. "A Source-aware Interrupt Scheduling for Modern Parallel I/O Systems," Proc. IEEE International Parallel and Distributed Processing Symp (IPDPS'12), pp. 156-166, 2012.
39. H. Zou, F. Zheng, K. Schwan etc. "Quality-Aware Data Management for Large Scale Scientific Applications," Proc. High Performance Computing, Networking, Storage and Analysis 2012 (SC'12), pp 816-820, 2012.