On the duality of data-intensive file system design: Reconciling HDFS and PVFS

Proceedings of 2011 SC - International Conference for High Performance Computing, Networking, Storage and Analysis

Volumn , Issue , 2011, Pages

  Tantisiriroj, Wittawat ^a   Patil, Swapnil ^a   Gibson, Garth ^a   Son, Seung Woo ^b   Lang, Samuel J ^b   Ross, Robert B ^b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

Cloud computing; File systems; Hadoop; HDFS; PVFS

Indexed keywords

APPLICATION PERFORMANCE; DATA-INTENSIVE APPLICATION; DATA-INTENSIVE COMPUTING; DISK BANDWIDTH; FILE SYSTEMS; HADOOP; HDFS; HIGH-PERFORMANCE COMPUTING; INTERNET SERVICES; KEY COMPONENT; PARALLEL FILE SYSTEM; PVFS; STORAGE SYSTEMS;

BENCHMARKING; COMPUTER SOFTWARE SELECTION AND EVALUATION; FILE ORGANIZATION; INTERNET; SYSTEMS ANALYSIS;

CLOUD COMPUTING;

EID: 83155182853     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2063384.2063474     Document Type: Conference Paper

References (35)

1. Amazon Web Services - High Performance Computing. http://aws.amazon.com/ hpc-applications/.
2. Lustre File System. http://www.lustre.org.
3. Using Lustre with Apache Hadoop. http://wiki.lustre.org/images/1/1b/ Hadoop-wp-v0.4.2.pdf, Jan. 2010
4. Amazon-EBS. Amazon Elastic Block Storage (Amazon EBS). http://www.amazon.com/s3.
5. Amazon-EC2. Amazon Elastic Compute Cloud (Amazon EC2). http://www.amazon.com/ec2.
6. Amazon-S3. Amazon Simple Storage Service (Amazon S3). http://www.amazon.com/s3.
7. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the Outliers in Map-Reduce Clusters using Mantri. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI'2010), Vancouver, Canada, October 2010.
8. R. Ananthanarayanan, K. Gupta, P. Pandey, H. Pucha, P. Sarkar, M. Shah, and R. Tewari. Cloud analytics: Do we really need to reinvent the storage stack? In Proceedings of the 1st USENIX Workshop on Hot Topics in Cloud Computing (HOTCLOUD'2009), San Diego, CA, USA, June 2009.
9. D. Borthakur. The Hadoop Distributed File System: Architecture and Designhttp://hadoop.apache.org/core/docs/r0.16.4/hdfsdesign.html.
10. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber. Bigtable: A Distributed Storage System for Structured Data. In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI'2006), Seattle, WA, USA, November 2006.
11. Cloudstore. Cloudstore distributed file system (formerly, Kosmos file system). http://kosmosfs.sourceforge.net/.
12. J. Dean. Experiences with MapReduce, an Abstraction for Large-Scale Computation. Slides from Keynote talk at the 15th International Conference on Parallel Architecture and Compilation Techniques (PACT'2006) on September 18, 2006 in Seattle WA.
13. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI'2004), San Francisco, CA, USA, December 2004.
14. G. DeCandia, D. Hastorun, M. Jampani, G. Kakula- pati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon's Highly Available Key-Value Store. In Proceedings of the 21st ACM Symposium on Operating Systems Principles (SOSP'2007), Stevenson, WA, USA, October 2007.
15. B. Fu, K. Ren, J. Lopez, E. Fink, and G. Gibson. Dis- cFinder: A data-intensive scalable cluster finder for astrophysics. In Proceedings of the 19th ACM Inter- national Symposium on High Performance Distributed Computing (HPDC'2010), Chicago, IL, USA, June 2010.
16. S. Ghemawat, H. Gobioff, and S.-T. Lueng. Google File System. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP'2003), Bolton Landing, NY, USA, October 2003.
17. G. A. Gibson, B. B. Welch, D. B. Nagle, and B. C. Moxon. ObjectStorage: Scalable Bandwidth for HPC Clusters. In Proceedings of 1st Cluster World Conference and Expo, San Jose, CA, USA, June 2003.
18. Hadoop. Apache Hadoop Project. http://hadoop.apache.org/.
19. Hadoop-Docs. The Hadoop Cluster Setup. http://hadoop.apache.org/core/ docs/current/cluster-setup.html.
20. Hadoop-S3. Using the Amazon S3 backend on Hadoop. http://wiki.apache.org/ hadoop/AmazonS3.
21. N. Horman. Understanding Virtual Memory In Red HatEnterprise Linux 4. Red Hat white paper, Raleigh, NC, 2005, http://people.redhat.com/nhorman/papers/ rhel4-vm.pdf.
22. U. Kang, B. Meeder, and C. Faloutsos. Spectral Analysis for Billion-Scale Graphs: Discoveries and Implementation. In Proceedings of the 14th Paci-c-Asia Conference on Knowledge Discovery and Data Mining (PAKDD'2010), Hyderabad, India, June 2010.
23. S. Lang, P. Carns, R. Latham, R. Ross, K. Harms, and W. Allcock. I/O performance challenges at leadership scale. In Proceedings of the 22nd ACM/IEEE Con- ference on High Performance Computing (SC'2009), Portland, OR, USA, November 2009.
24. S. Lang, R. Latham, R. B. Ross, and D. Kimpe. Interfaces for coordinated access in the file system. In Proceedings of the 11th IEEE International Conference on Cluster Computing (CLUSTER'2009), New Orleans, LA, USA, August 2009.
25. Magellan. Argonne's DOE Cloud Computing: Magellan, A Cloud for Science. http://magellan.alcf.anl.gov/.
26. C. Maltzahn, E. Molina-Estolano, A. Khurana, A. J. Nelson, S. A. Brandt, and S. Weil. Ceph as a scalable alternative to the Hadoop Distributed File System.;login: The USENIX MAGAZINE, 35(4), August 2010.
27. E. Pinheiro, W.-D. Weber, and L. A. Barroso. Failure Trends in a Large Disk Drive Population. In Proceed- ings of the 5th USENIX Conference on File and Storage Technologies (FAST'2007), San Jose, CA, USA, Febru- ary 2007.
28. G. Porter. Towards Decoupling Storage and Computation in Hadoop with SuperDataNodes. In Proceedings of the 3rd ACM SIGOPS International Workshop on Large Scale Distributed Systems and Middleware (LADIS'2009), Big Sky, MT, USA, October 2009.
29. PVFS2. Parallel Virtual File System, Version 2. http://www.pvfs.org.
30. F. Schmuck and R. Haskin. GPFS: A Shared-Disk File System for Large Computing Clusters. In Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST'2002), Monterey, CA, USA, January 2002.
31. B. Schroeder and G. A. Gibson. Disk Failures in the Real World: What Does an MTTF of 1, 000, 000 Hours Mean to You? In Proceedings of the 5th USENIX Con- ference on File and Storage Technologies (FAST'2007), San Jose, CA, USA, February 2007.
32. K. Shvachko, H. Huang, S. Radia, and R. Chansler. The Hadoop Distributed File System. In Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies (MSST'2010), Incline Village, NV, USA, May 2010.
33. W. Tantisiroj, S. V. Patil, and G. Gibson. Data- intensive file systems for Internet services: A rose by any other name ... . Technical Report CMU-PDL-08- 114, Carnegie Mellon University, Oct. 2008.
34. B. Welch, M. Unangst, Z. Abbasi, G. Gibson, B. Mueller, J. Small, J. Zelenka, and B. Zhou. Scalable Performance of the Panasas Parallel File System. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST'2008), San Jose, CA, USA, February 2008.
35. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving Map Reduce Performance in Heterogeneous Environments. In Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI'2008), San Diego, CA, USA, December 2008.