FusionFS: Toward supporting data-intensive scientific applications on extreme-scale high-performance computing systems

Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014

Volumn , Issue , 2014, Pages 61-70

  Zhao, Dongfang ^a   Zhang, Zhao ^b   Zhou, Xiaobing ^c   Li, Tonglin ^a   Wang, Ke ^a   Kimpe, Dries ^d   Carns, Philip ^d   Ross, Robert ^d   Raicu, Ioan ^a  

^a ILLINOIS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c Hortonworks Inc   (United States)

^d ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLUSTER COMPUTING; COMPUTER ARCHITECTURE; DIGITAL STORAGE; FILE ORGANIZATION; NETWORK ARCHITECTURE;

COMPUTE RESOURCES; DATA INTENSIVE; DISTRIBUTED STORAGE; EXTREME SCALE; HIGH PERFORMANCE COMPUTING SYSTEMS; PERFORMANCE; SCIENTIFIC APPLICATIONS; STATE OF THE ART; STORAGE LAYERS; STORAGE RESOURCES;

SUPERCOMPUTERS;

EID: 84921786320     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BigData.2014.7004214     Document Type: Conference Paper

References (56)

1. I. Raicu, I. Foster, A. Szalay, and G. Turcu, "AstroPortal: A science gateway for large-scale astronomy data analysis," in TeraGrid Conference June 2006.
2. P. Freeman, D. Crawford, S. Kim, and J. Munoz, "Cyberinfrastructure for science and engineering: Promises and challenges," Proceedings of the IEEE, vol. 93, no. 3, 2005.
3. K. Wang, A. Kulkarni, M. Lang, D. Arnold, and I. Raicu, "Using simulation to explore distributed key-value stores for extreme-scale system services," in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, 2013.
4. D. Zhao, D. Zhang, K. Wang, and I. Raicu, "Exploring reliability of exascale systems through simulations," in Proceedings of the 21st ACM/SCS High Performance Computing Symposium (HPC), 2013.
5. N. Liu, J. Cope, P. H. Carns, C. D. Carothers, R. B. Ross, G. Grider A. Crume, and C. Maltzahn, "On the role of burst buffers in leadershipclass storage systems," in Proceedings of IEEE Symposium on Mass Storage Systems and Technologies, 2012.
6. W. Tantisiriroj, S. W. Son, S. Patil, S. J. Lang, G. Gibson, and R. B. Ross, "On the duality of data-intensive file system design: Reconciling HDFS and PVFS," in Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis, 2011.
7. D. Zhao and I. Raicu, "Distributed file systems for exascale computing," in International Conference for High Performance Computing, Networking Storage and Analysis (SC '12), doctoral showcase, 2012.
8. DEEP-ER, "http: //www.hpc.cineca.it/projects/deep-er," Accessed September 5, 2014.
9. M. Dorier, G. Antoniu, F. Cappello, M. Snir, and L. Orf, "Damaris: How to efficiently leverage multicore parallelism to achieve scalable, jitterfree I/O," in Proceedings of IEEE International Conference on Cluster Computing, 2012.
10. Mira, "https: //www.alcf.anl.gov/user-guides/mira-cetus-vesta," Accessed September 5, 2014.
11. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The hadoop distributed file system," in Proceedings of IEEE Symposium on Mass Storage Systems and Technologies, 2010.
12. B. Welch and G. Noer, "Optimizing a hybrid SSD/HDD HPC storage system based on file size distributions," in Mass Storage Systems and Technologies, 2013 IEEE 29th Symposium on, 2013.
13. D. Nagle, D. Serenyi, and A. Matthews, "The Panasas activescale storage cluster: Delivering scalable high bandwidth storage," in Proceedings of ACM/IEEE Conference on Supercomputing, 2004.
14. F. Schmuck and R. Haskin, "GPFS: A shared-disk file system for large computing clusters," in Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST), 2002.
15. Intrepid, "https: //www.alcf.anl.gov/user-guides/intrepid-challengersurveyor," Accessed September 5, 2014.
16. P. H. Carns, W. B. Ligon, R. B. Ross, and R. Thakur, "PVFS: A parallel file system for linux clusters," in Proceedings of the 4th Annual Linux Showcase and Conference, 2000.
17. D. Zhao, J. Yin, K. Qiao, and I. Raicu, "Virtual chunks: On supporting random accesses to scientific data in compressible storage systems," in Proceedings of IEEE International Conference on Big Data, 2014.
18. D. Zhao, J. Yin, and I. Raicu, "Improving the i/o throughput for dataintensive scientific applications with efficient compression mechanisms," in International Conference for High Performance Computing, Networking Storage and Analysis (SC '13), poster session, 2013.
19. FUSE, "http: //fuse.sourceforge.net," Accessed September 5, 2014.
20. P. Schwan, "Lustre: Building a file system for 1,000-node clusters," in Proceedings of the linux symposium, 2003.
21. A. Rajgarhia and A. Gehani, "Performance and extension of user space file systems," in Proceedings of ACM Symposium on Applied Computing 2010.
22. D. Zhao and I. Raicu, "HyCache: A user-level caching middleware for distributed file systems," in Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing Workshops and PhD Forum, 2013.
23. T. Li, X. Zhou, K. Brandstatter, D. Zhao, K. Wang, A. Rajendran Z. Zhang, and I. Raicu, "ZHT: A light-weight reliable persistent dynamic scalable zero-hop distributed hash table," in Proceedings of IEEE International Symposium on Parallel and Distributed Processing, 2013.
24. T. Li, R. Verma, X. Duan, H. Jin, and I. Raicu, "Exploring distributed hash tables in highend computing," SIGMETRICS Perform. Eval. Rev. vol. 39, no. 3, Dec. 2011.
25. D. Zhao, C. Shou, T. Malik, and I. Raicu, "Distributed data provenance for large-scale data-intensive computing," in Cluster Computing, IEEE International Conference on, 2013.
26. C. Shou, D. Zhao, T. Malik, and I. Raicu, "Towards a provenance-aware distributed filesystem," in 5th Workshop on the Theory and Practice of Provenance (TaPP), 2013.
27. D. Zhao, K. Burlingame, C. Debains, P. Alvarez-Tabio, and I. Raicu "Towards high-performance and cost-effective distributed storage systems with information dispersal algorithms," in Cluster Computing IEEE International Conference on, 2013.
28. Protocol Buffers, "http: //code.google.com/p/protobuf/," Accessed September 5, 2014.
29. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google file system," in Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, 2003.
30. epoll, "http: //man7.org/linux/man-pages/man7/epoll.7.html," Accessed September 5, 2014.
31. Y. Gu and R. L. Grossman, "Supporting configurable congestion control in data transport services," in Proceedings of the 2005 ACM/IEEE Conference on Supercomputing, 2005.
32. M. Wilde, I. Raicu, A. Espinosa, Z. Zhang, B. Clifford, M. Hategan S. Kenny, K. Iskra, P. Beckman, and I. Foster, "Extreme-scale scripting: Opportunities for large task-parallel applications on petascale computers," Journal of Physics: Conference Series, vol. 180, no. 1, 2009.
33. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova I. Raicu, T. Stef-Praun, and M. Wilde, "Swift: Fast, reliable loosely coupled parallel computation," in Proceedings of the 2007 IEEE Congress on Services, 2007.
34. I. Raicu, I. T. Foster, Y. Zhao, P. Little, C. M. Moretti, A. Chaudhary, and D. Thain, "The quest for scalable support of data-intensive workloads in distributed systems," in Proceedings of ACM International Symposium on High Performance Distributed Computing, 2009.
35. Top500, "http: //www.top500.org/list/2014/06/," Accessed September 5 2014.
36. ZeptoOS, "http: //www.mcs.anl.gov/zeptoos," Accessed September 5 2014.
37. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, "Falkon: a fast and light-weight task execution framework," in Proceedings of the 2007 ACM/IEEE conference on Supercomputing, 2007.
38. D. Zhao and L. Yang, "Incremental isometric embedding of highdimensional data using connected neighborhood graphs," IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), vol. 31, no. 1, Jan. 2009.
39. R. Lohfert, J. Lu, and D. Zhao, "Solving sql constraints by incremental translation to sat," in International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, 2008.
40. D. Zhao and L. Yang, "Incremental construction of neighborhood graphs for nonlinear dimensionality reduction," in Proceedings of International Conference on Pattern Recognition, 2006.
41. Kodiak, "https: //www.nmc-probe.org/wiki/machines: kodiak," Accessed September 5, 2014.
42. D. R. Mathog, "Parallel BLAST on split databases," Bioinformatics, vol. 19(4), pp. 1865-1866, 2003.
43. Z. Zhang, D. S. Katz, J. M. Wozniak, A. Espinosa, and I. T. Foster "Design and analysis of data management in scalable parallel scripting," in Proceedings of ACM/IEEE conference on Supercomputing, 2012.
44. M. Eisler, R. Labiaga, and H. Stern, "Managing NFS and NIS, 2nd ed." O'Reilly &Associates, Inc., 2001.
45. S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn "Ceph: A scalable, high-performance distributed file system," in Proceedings of the 7th Symposium on Operating Systems Design and Implementation, 2006.
46. Y. Gu, R. L. Grossman, A. Szalay, and A. Thakar, "Distributing the Sloan Digital Sky Survey using UDT and Sector," in Proceedings of IEEE International Conference on e-Science and Grid Computing, 2006.
47. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proceedings of USENIX Symposium on Opearting Systems Design &Implementation, 2004.
48. T. E. Anderson, M. D. Dahlin, J. M. Neefe, D. A. Patterson, D. S. Roselli, and R. Y. Wang, "Serverless network file systems," in Proceedings of ACM symposium on Operating systems principles, 1995.
49. E. B. Nightingale, J. Elson, J. Fan, O. Hofmann, J. Howell, and Y. Suzue "Flat datacenter storage," in Proceedings of USENIX Symposium on Operating Systems Design and Implementation, 2012.
50. Y. Wang, M. Kapritsos, Z. Ren, P. Mahajan, J. Kirubanandam, L. Alvisi and M. Dahlin, "Robustness in the salus scalable block store," in Proceedings of USENIX conference on Networked Systems Design and Implementation, 2013.
51. C. Gkantsidis, D. Vytiniotis, O. Hodson, D. Narayanan, F. Dinu, and A. Rowstron, "Rhea: automatic filtering for unstructured cloud storage," in Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation, 2013.
52. P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y. Yu, and L. Zhuang "Nectar: automatic management of data and computation in datacenters," in Proceedings of the 9th USENIX conference on Operating systems design and implementation (OSDI), 2010.
53. Tachyon, "http: //tachyon-project.org/," 2014.
54. D. Zhao, K. Qiao, and I. Raicu, "Hycache+: Towards scalable highperformance caching middleware for parallel file systems," in Proceedings of IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 2014.
55. I. Raicu, I. Foster, Y. Zhao, A. Szalay, P. Little, C. M. Moretti, A. Chaudhary and D. Thain, Towards Data Intensive Many-Task Computing T. Kosar, Ed. IGI Global, 2012.
56. G. Gibson, G. Grider, A. Jacobson, and W. Lloyd, "Probe: A thousandnode experimental cluster for computer systems research," USENIX;login:, vol. 38, no. 3, June 2013