Dynamic metadata management for petabyte-scale file systems

IEEE/ACM SC2004 Conference, Proceedings

Volumn , Issue , 2004, Pages 523-534

  Weil, Sage A ^a   Pollack, Kristal T ^a   Brandt, Scott A ^a   Miller, Ethan L ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

FILE SYSTEMS; METADATA MANAGEMENT; METADATA WORKLOADS; SYSTEM PERFORMANCE;

COMPUTER SIMULATION; FILE ORGANIZATION; HIERARCHICAL SYSTEMS; SERVERS; SYSTEMS ANALYSIS; TREES (MATHEMATICS);

METADATA;

EID: 23944471836     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. P. Braam, M. Callahan, and P. Schwan. The intermezzo file system. In Proceedings of the 3rd of the Perl Conference, O'Reilly Open Source Convention, Monterey, CA, USA, Aug. 1999.
2. P. J. Braam. The Lustre storage architecture, 2002.
3. S. A. Brandt, L. Xue, E. L. Miller, and D. D. E. Long. Efficient metadata management in large distributed file systems. In Proceedings of the 20th IEEE/ 11th NASA Goddard Conference on Mass Storage Systems and Technologies, pages 290-298, Apr. 2003.
4. L.-F. Cabrera and D. D. E. Long. Swift: Using distributed disk striping to provide high I/O data rates. Computing Systems, 4(4):405-436, 1991.
5. P. F. Corbett and D. G. Feitelson. The Vesta parallel file system. ACM Transactions on Computer Systems, 14(3):225-264, 1996.
6. R. A. Floyd and C. S. Ellis. Directory reference patterns in hierarchical file systems. IEEE Transactions on Knowledge and Data Engineering, 1(2):238-247, 1989.
7. G. R. Ganger and M. F. Kaashoek. Embedded inodes and explicit groupings: Exploiting disk bandwidth for small files. In Proceedings of the 1997 USENIX Annual Technical Conference, pages 1-17. USENIX Association, Jan. 1997.
8. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP'03), Bolton Landing, NY, Oct. 2003. ACM.
9. G. A. Gibson, D. F. Nagle, K. Amiri, J. Butler, F. W. Chang, H. Gobioff, C. Hardin, E. Riedel, D. Rochberg, and J. Zelenka. A cost-effective, high-bandwidth storage architecture. In Proceedings of the 8th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 92-103, San Jose, CA, Oct. 1998.
10. D. Hitz, J. Lau, and M. Malcom. File system design for an NFS file server appliance. In Proceedings of the Winter 1994 USENIX Technical Conference, pages 235-246, San Francisco, CA, Jan. 1994.
11. R. J. Honicky and E. L. Miller. Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution. In Proceedings of the 18th International Parallel & Distributed Processing Symposium (IPDPS 2004), Santa Fe, NM, Apr. 2004. IEEE.
12. D. Long, S. Brandt, E. Miller, F. Wang, Y. Lin, L. Xue, and Q. Xin. Design and implementation of large scale object-based storage system. Technical Report ucsc-crl-02-35, University of California, Santa Cruz, Nov. 2002.
13. M. K. McKusick, W. N. Joy, S. J. Leffler, and R. S. Fabry. A fast file system for UNIX. ACM Transactions on Computer Systems, 2(3):181-197, Aug. 1984.
14. E. L. Miller and R. H. Katz. RAMA: An easy-to-use, high-performance parallel file system. Parallel Computing, 23(4):419-446, 1997.
15. J. H. Morris, M. Satyanarayanan, M. H. Conner, J. H. Howard, D. S. H. Rosenthal, and F. D. Smith. Andrew: A distributed personal computing environment Communications of the ACM, 29(3): 184-201, Mar. 1986.
16. J. K. Ousterhout, A. R. Cherenson, F. Douglis, M. N. Nelson, and B. B. Welch. The Sprite network operating system. IEEE Computer, 21(2):23-36, Feb. 1988.
17. B. Pawlowski, C. Juszczak, P. Staubach, C. Smith, D. Lebel, and D. Hitz. NFS version 3: Design and implementation. In Proceedings of the Summer 1994 USENIX Technical Conference, pages 137-151, 1994.
18. O. Rodeh and A. Teperman. zFS - a scalable distributed file system using object disks. In Proceedings of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, pages 207-218, Apr. 2003.
19. D. Roselli, J. Lorch, and T. Anderson. A comparison of file system workloads. In Proceedings of the 2000 USENIX Annual Technical Conference, pages 41-54, June 2000.
20. M. Satyanarayanan, J. J. Kistler, P. Kumar, M. E. Okasaki, E. H. Siegel, and D. C. Steere. Coda: A highly available file system for a distributed workstation environment. IEEE Transactions on Computers, 39(4):447-459, 1990.
21. F. Schmuck and R. Haskin. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 2002 Conference on File and Storage Technologies (FAST), pages 231-244. USENIX, Jan. 2002.
22. P. Schwan. Lustre: Building a file system for 1000-node clusters. In Proceedings of the 2003 Linux Symposium, July 2003.
23. A. Sweeney, D. Doucette, W. Hu, C. Anderson, M. Nishimoto, and G. Peck. Scalability in the XFS file system. In Proceedings of the 1996 USENIX Annual Technical Conference, pages 1-14, Jan. 1996.
24. C. A. Thekkath, J. Wilkes, and E. D. Lazowska. Techniques for file system simulation. Software - Practice and Experience (SPE), 24(11):981-999, Nov. 1994.
25. F. Wang, S. A. Brandt, E. L. Miller, and D. D. E. Long. OBFS: A file system for object-based storage devices. In Proceedings of the 21st IEEE/ 12th NASA Goddard Conference on Mass Storage Systems and Technologies, College Park, MD, Apr. 2004. IEEE.
26. F. Wang, Q. Xin, B. Hong, S. A. Brandt, E. L. Miller, D. D. E. Long, and T. T. McLarty. File system workload analysis for large scale scientific computing applications. In Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, College Park, MD, Apr. 2004.
27. J. Wilkes. The Pantheon storage-system simulator. Technical Report HPL-SSP-95-14, Storage Systems Program, Computer Systems Laboratory, Hewlett-Packard Laboratories, Palo Alto, CA, May 1996.
28. Q. Xin, E. L. Miller, T. J. Schwarz, D. D. E. Long, S. A. Brandt, and W. Litwin. Reliability mechanisms for very large storage systems. In Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies, pages 146-156, Apr. 2003.