Small-file access in parallel file systems

IPDPS 2009 - Proceedings of the 2009 IEEE International Parallel and Distributed Processing Symposium

Volumn , Issue , 2009, Pages

  Carns, Philip ^a   Lang, Sam ^a   Ross, Robert ^a   Vilayannur, Murali ^b   Kunkel, Julian ^c   Ludwig, Thomas ^c  

^a ARGONNE NATIONAL LABORATORY   (United States)

^b VMWARE INC   (United States)

^c UNIVERSITY OF HEIDELBERG   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

BEST PRACTICE; COMPUTATIONAL SCIENCE; COMPUTING PLATFORM; FILE ACCESS; I/O OPERATIONS; MICRO-BENCHMARK; PARALLEL COMPUTER; PARALLEL FILE SYSTEM; REMOVAL RATE; SMALL FILES; STORAGE SYSTEMS; TEST PLATFORMS;

LARGE SCALE SYSTEMS; SYSTEMS ENGINEERING;

DISTRIBUTED PARAMETER NETWORKS;

EID: 70449975613     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2009.5161029     Document Type: Conference Paper

References (33)

1. Petascale Data Storage Institute, "NERSC file system statistics," World Wide Web electronic publication, 2007. [Online]. Available: http://pdsi.nersc.gov/filesystem.htm
2. E. Felix, "Environmental Molecular Sciences Laboratory file system statistics," 2007. [Online]. Available: http://pdsi-scidac.org/fsstats/ index.html
3. A. Chervenak, J. M. Schopf, L. Pearlman, M.-H. Su, S. Bharathi, L. Cinquini, M. D'Arcy, N. Miller, and D. Bernholdt, "Monitoring the Earth System Grid with MDS4," in E-SCIENCE '06: Proceedings of the Second IEEE International Conference on e-Science and Grid Computing. Washington: IEEE Computer Society, 2006, p. 69.
4. E. H. Neilsen Jr., "The Sloan Digital Sky Survey data archive server," Computing in Science and Engineering, vol. 10, no. 1, pp. 13-17, 2008.
5. J. K. Bonfield and R. Staden, "ZTR: A new format for DNA sequence trace data," Bioinformatics, vol. 18, no. 1, pp. 3-10, 2002.
6. "The Parallel Virtual File System," http://www.pvfs.org.
7. R. Latham, N. Miller, R. B. Ross, and P. H. Carns, "A next-generation parallel file system for Linux clusters," LinuxWorld Magazine, January 2004. [Online]. Available: http://www.pvfs.org/ documentation/papers/linuxworld-JAN2004-PVFS2.pdf
8. P. H. Carns, W B. L. III, R. Ross, and P. Wyckoff, "BMI: A network abstraction layer for parallel I/O," in Proceedings of IPDPS '05, April 2005.
9. A. Devulapalli, D. Dalessandro, P. Wyckoff, N. Ali, and P. Sadayappan, "Integrating parallel file systems with object-based storage devices," in Proceedings of Supercomputing, November 2007. [Online]. Available: http://www.osc.edu/ pw/papers/devulapalli-pvfs-osd-sc07.pdf
10. A. Ching, R. Ross, W keng Liao, L. Ward, and A. Choudhary, "Noncontiguous locking techniques for parallel file systems," in Proceedings of Supercomputing, November 2007.
11. Y. Zhu and H. Jiang, "Ceft: A cost-effective, fault-tolerant parallel virtual file system," Journal of Parallel and Distributed Computing, vol. 66, pp. 291-306, February 2006. [Online]. Available: http://portal.acm. org/citation.cfm?id=1 140805
12. W Yu, S. Liang, and D. K. Panda, "High performance support of Parallel Virtual File System (PVFS2) over Quadrics," in International Conference on Supercomputing (ICS '05), 2005, pp. 323-331. [Online]. Available: http://www.cse.ohio-state.edu/ liangs/paper/yuw-ics05.pdf
13. D. Hildebrand and P. Honeyman, "Exporting storage systems in a scalable manner with pNFS," in 13th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2005.
14. B. Pawlowski, C. Juszczak, P. Staubach, C. Smith, D. Lebel, and D. Hitz, "NFS version 3: Design and implementation," in USENIX Summer, 1994, pp. 137-152. [Online]. Available: citeseer.ist.psu.edu/pawlowski94nfs.html
15. A. Robertson, "Linux-HA heartbeat design," in Proceedings of the 4th Annual Linux Showcase and Conference, October 2000.
16. S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn, "Ceph: A scalable, high-performance distributed file system," in OSDI '06: Proceedings of the 7th Symposium on Operating Systems Design and Implementation. Berkeley, CA, USA: USENIX Association, 2006, pp. 307-320.
17. "Lustre file system," http://www.sun.com/software/products/ lustre/.
18. S. R. Soltis, T. M. Ruwart, G. M. Erickson, K. W Preslan, and M. T. O'Keefe, "The global file system," in High Performance Mass Storage and Parallel I/O: Technologies and Applications, H. Jin, T. Cortes, and R. Buyya, Eds. New York: IEEE Computer Society Press and Wiley, 2001, ch. 23, pp. 344-363.
19. B. Welch, M. Unangst, Z. Abbasi, G. Gibson, B. Mueller, J. Small, J. Zelenka, and B. Zhou, "Scalable performance of the Panasas parallel file system," in FAST'08: Proceedings of the 6th USENIX Conference on File and Storage Technologies. Berkeley, CA: USENIX Association, 2008, pp. 1-17.
20. P. Helland, H. Sammer, J. Lyon, R. Carr, P. Garrett, and A. Reuter, "Group commit timers and high volume transaction systems," in Proceedings of the 2nd International Workshop on High Performance Transaction Systems, 1987, pp. 301-329.
21. E. Lusk, N. Doss, and A. Skjellum, "A high-performance, portable implementation of the MPI message passing interface standard," Parallel Computing, vol. 22, pp. 789-828, 1996.
22. "POSIX HEC Extensions Project," http://www.pdl.cmu.edu/posix/.
23. G. Grider, L. Ward, R. Ross, and G. Gibson, "A business case for extensions to the POSIX I/O API for high end, clustered, and highly concurrent computing," http://www.opengroup.org/platform/hecewg/uploads/40/10891/ POSIX-IO-API-Business-case-HEC-ggrider.pdf, 2006.
24. D. Thain and C. Moretti, "Efficient access to many small files in a filesystem for grid computing," in 2007 8th IEEE/ACM International Conference on Grid Computing, 2007, pp. 243-250.
25. J. J. Ritsko, I. Ames, S. I. Raider, and J. H. Robinson, "Blue Gene," IBM Journal of Research and Development, vol. 49, March/May 2005.
26. K. Iskra, J. W Romein, K. Yoshii, and P. Beckman, "ZOID: I/O-forwarding infrastructure for petascale architectures," in Proceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Salt Lake City, UT, Feb. 2008, pp. 153-162.
27. A. Devulapalli and P. Wyckoff, "File creation strategies in a distributed metadata file system," in Proceedings of IPDPS'07, March 2007.
28. M. Kuhn, J. M. Kunkel, and T. Ludwig, "Directory-based metadata optimizations for small files in PVFS," in Euro-Par, 2008, pp. 90-99.
29. P. H. Carns, "Achieving scalability in parallel file systems," Ph.D. dissertation, Clemson University, May 2005.
30. P. H. Carns, B. W. Settlemyer, and I. Walter B. Ligon, "Using server-to-server communication in parallel file systems to simplify consistency and improve performance," in SC '08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing. Piscataway, NJ: IEEE Press, 2008, pp. 1-8.
31. J. Cope, M. Oberg, H. Tufo, and M. Woitaszek, "Shared parallel file systems in heterogeneous Linux multi-cluster environments," in Proceedings of the 6th LCI International Conference on Linux Clusters: The HPC Revolution.
32. W Yu, J. S. Vetter, and S. Oral, "Performance characterization and optimization of parallel I/O on the Cray XT," in IPDPS, 2008, pp. 1-11.
33. S. V. Patil, G. A. Gibson, S. Lang, and M. Polte, "GIGA+: Scalable directories for shared file systems," in Petascale Data Storage Workshop (in Conjunction with SC07), Nov. 2007.