Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014

Volumn , Issue , 2014, Pages 231-240

  Zhao, Dongfang ^a   Yin, Jian ^b   Qiao, Kan ^a   Raicu, Ioan ^a  

^a ILLINOIS INSTITUTE OF TECHNOLOGY   (United States)

^b PACIFIC NORTHWEST NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BIG DATA; DATA COMPRESSION RATIO; DIGITAL STORAGE; FILE ORGANIZATION; MIDDLEWARE;

ARBITRARY POSITIONS; BLOCK-LEVEL COMPRESSIONS; COMPRESSED FILES; DISTRIBUTED FILE SYSTEMS; HIGH COMPRESSION RATIO; HIGH PERFORMANCE COMPUTING SYSTEMS; PARALLEL FILE SYSTEM; SCIENTIFIC APPLICATIONS;

DATA COMPRESSION;

EID: 84988273952     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2014.7004238     Document Type: Conference Paper

References (50)

1. I. Raicu, I. Foster, Y. Zhao, A. Szalay, P. Little, C. M. Moretti, A. Chaudhary and D. Thain, Towards Data Intensive Many-Task Computing T. Kosar, Ed. IGI Global, 2012.
2. LZO, "http: //www.oberhumer.com/opensource/lzo," Accessed September 5, 2014.
3. bzip2, "http: //www.bzip2.org," Accessed September 5, 2014.
4. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The hadoop distributed file system," in Proceedings of IEEE Symposium on Mass Storage Systems and Technologies, 2010.
5. D. Laney, S. Langer, C. Weber, P. Lindstrom, and A. Wegener, "Assessing the effects of data compression in simulations using physically motivated metrics," in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, 2013.
6. S. Lakshminarasimhan, J. Jenkins, I. Arkatkar, Z. Gong, H. Kolla S.-H. Ku, S. Ethier, J. Chen, C. S. Chang, S. Klasky, R. Latham R. Ross, and N. F. Samatova, "ISABELA-QA: Query-driven analytics with ISABELA-compressed extreme-scale scientific data," in Proceedings of 2011 International Conference for High Performance Computing Networking, Storage and Analysis (SC'11), 2011.
7. HDF5, "http: //www.hdfgroup.org/hdf5/doc/index.html," Accessed September 5, 2014.
8. NetCDF, "http: //www.unidata.ucar.edu/software/netcdf," Accessed September 5, 2014.
9. D. Zhao, J. Yin, and I. Raicu, "Improving the i/o throughput for dataintensive scientific applications with efficient compression mechanisms," in International Conference for High Performance Computing, Networking Storage and Analysis (SC '13), poster session, 2013.
10. F. Schmuck and R. Haskin, "GPFS: A shared-disk file system for large computing clusters," in Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST), 2002.
11. D. Zhao, Z. Zhang, X. Zhou, T. Li, K. Wang, D. Kimpe, P. Carns R. Ross, and I. Raicu, "FusionFS: Towards supporting data-intensive scientific applications on extreme-scale distributed systems," in Proceedings of IEEE International Conference on Big Data, 2014.
12. D. Zhao and I. Raicu, "Distributed file systems for exascale computing," in International Conference for High Performance Computing, Networking Storage and Analysis (SC '12), doctoral showcase, 2012.
13. GCRM, "http: //kiwi.atmos.colostate.edu/gcrm/," Accessed September 5 2014.
14. SDSS Query, "http: //cas.sdss.org/astrodr6/en/help/docs/ realquery.asp," Accessed September 5, 2014.
15. Intrepid, "https: //www.alcf.anl.gov/user-guides/intrepid-challengersurveyor," Accessed September 5, 2014.
16. T. Bicer, J. Yin, D. Chiu, G. Agrawal, and K. Schuchardt, "Integrating online compression to accelerate large-scale data analytics applications," in Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing (IPDPS), 2013.
17. FUSE, "http: //fuse.sourceforge.net," Accessed September 5, 2014.
18. D. Zhao, K. Qiao, and I. Raicu, "Hycache+: Towards scalable highperformance caching middleware for parallel file systems," in Proceedings of IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 2014.
19. D. Zhao and I. Raicu, "HyCache: A user-level caching middleware for distributed file systems," in Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing Workshops and PhD Forum, 2013.
20. D. Zhao, D. Zhang, K. Wang, and I. Raicu, "Exploring reliability of exascale systems through simulations," in Proceedings of the 21st ACM/SCS High Performance Computing Symposium (HPC), 2013.
21. T. Li, X. Zhou, K. Brandstatter, D. Zhao, K. Wang, A. Rajendran Z. Zhang, and I. Raicu, "ZHT: A light-weight reliable persistent dynamic scalable zero-hop distributed hash table," in Proceedings of IEEE International Symposium on Parallel and Distributed Processing, 2013.
22. T. Li, R. Verma, X. Duan, H. Jin, and I. Raicu, "Exploring distributed hash tables in highend computing," SIGMETRICS Perform. Eval. Rev. vol. 39, no. 3, Dec. 2011.
23. D. Zhao, C. Shou, T. Malik, and I. Raicu, "Distributed data provenance for large-scale data-intensive computing," in Cluster Computing, IEEE International Conference on, 2013.
24. C. Shou, D. Zhao, T. Malik, and I. Raicu, "Towards a provenance-aware distributed filesystem," in 5th Workshop on the Theory and Practice of Provenance (TaPP), 2013.
25. Gzip, "http: //www.gnu.org/software/gzip/gzip.html," Accessed September 5, 2014.
26. Snappy, "https: //code.google.com/p/snappy/," Accessed September 5 2014.
27. E. Cohen and M. Strauss, "Maintaining time-decaying stream aggregates," in Proceedings of the Twenty-second ACM SIGMOD-SIGACTSIGART Symposium on Principles of Database Systems, 2003.
28. K. B. Ferreira, R. Riesen, D. Arnold, D. Ibtesham, and R. Brightwell "The viability of using compression to decrease message log sizes," in Proceedings of International Conference on Parallel Processing Workshops, 2013.
29. T. Z. Islam, K. Mohror, S. Bagchi, A. Moody, B. R. de Supinski, and R. Eigenmann, "McrEngine: A scalable checkpointing system using dataaware aggregation and compression," in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC), 2012.
30. M. S. Bouguerra, A. Gainaru, L. B. Gomez, F. Cappello, S. Matsuoka and N. Maruyam, "Improving the computing efficiency of hpc systems using a combination of proactive and preventive checkpointing," in Parallel Distributed Processing, IEEE International Symposium on 2013.
31. M. Noeth, J. Marathe, F. Mueller, M. Schulz, and B. de Supinski "Scalable compression and replay of communication traces in massively parallel environments," in Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (SC), 2006.
32. MPEG-1, "http: //en.wikipedia.org/wiki/mpeg-1," Accessed September 5 2014.
33. E. R. Schendel, S. V. Pendse, J. Jenkins, D. A. Boyuka, II, Z. Gong S. Lakshminarasimhan, Q. Liu, H. Kolla, J. Chen, S. Klasky, R. Ross and N. F. Samatova, "Isobar hybrid compression-i/o interleaving for large-scale parallel i/o optimization," in Proceedings of International Symposium on High-Performance Parallel and Distributed Computing 2012.
34. J. Jenkins, E. R. Schendel, S. Lakshminarasimhan, D. A. Boyuka, II T. Rogers, S. Ethier, R. Ross, S. Klasky, and N. F. Samatova, "Byteprecision level of detail processing for variable precision analytics," in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC), 2012.
35. M. Burrows, C. Jerian, B. Lampson, and T. Mann, "On-line data compression in a log-structured file system," in Proceedings of the Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 1992.
36. J. P. MacDonald, "File system support for delta compression," University of California, Berkley, Tech. Rep., 2000.
37. M. A. Olson, K. Bostic, and M. Seltzer, "Berkeley db," in Proceedings of the Annual Conference on USENIX Annual Technical Conference, 1999.
38. N. K. Edel, D. Tuteja, E. L. Miller, and S. A. Brandt, "Mramfs: A compressing file system for non-volatile ram," in Proceedings of the The IEEE Computer Society's 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems (MASCOTS), 2004.
39. A. Muthitacharoen, B. Chen, and D. Mazieres, "A low-bandwidth network file system," in Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles (SOSP), 2001.
40. K. Park, S. Ihm, M. Bowman, and V. S. Pai, "Supporting practical content-addressable caching with czip compression," in 2007 USENIX Annual Technical Conference, 2007.
41. D. Meister, A. Brinkmann, and T. Süß, "File recipe compression in data deduplication systems," in Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST), 2013.
42. S. Lakshminarasimhan, D. A. Boyuka, S. V. Pendse, X. Zou, J. Jenkins V. Vishwanath, M. E. Papka, and N. F. Samatova, "Scalable in situ scientific data encoding for analytical query processing," in Proceedings of the 22nd International Symposium on High-performance Parallel and Distributed Computing (HPDC), 2013.
43. Z. Gong, S. Lakshminarasimhan, J. Jenkins, H. Kolla, S. Ethier, J. Chen R. Ross, S. Klasky, and N. F. Samatova, "Multi-level layout optimization for efficient spatio-temporal queries on isabela-compressed data," in Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium (IPDPS), 2012.
44. I. Raicu, I. Foster, A. Szalay, and G. Turcu, "AstroPortal: A science gateway for large-scale astronomy data analysis," in TeraGrid Conference June 2006.
45. M. Wilde, I. Raicu, A. Espinosa, Z. Zhang, B. Clifford, M. Hategan S. Kenny, K. Iskra, P. Beckman, and I. Foster, "Extreme-scale scripting: Opportunities for large task-parallel applications on petascale computers," Journal of Physics: Conference Series, vol. 180, no. 1, 2009.
46. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova I. Raicu, T. Stef-Praun, and M.Wilde, "Swift: Fast, reliable, loosely coupled parallel computation," in Services, IEEE Congress on, 2007.
47. D. Zhao and L. Yang, "Incremental isometric embedding of highdimensional data using connected neighborhood graphs," IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), vol. 31, no. 1, Jan. 2009.
48. R. Lohfert, J. Lu, and D. Zhao, "Solving sql constraints by incremental translation to sat," in International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, 2008.
49. D. Zhao and L. Yang, "Incremental construction of neighborhood graphs for nonlinear dimensionality reduction," in Proceedings of International Conference on Pattern Recognition, 2006.
50. D. Zhao, K. Burlingame, C. Debains, P. Alvarez-Tabio, and I. Raicu "Towards high-performance and cost-effective distributed storage systems with information dispersal algorithms," in Cluster Computing, IEEE International Conference on, 2013