Adaptive and scalable metadata management to support A trillion files

Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC '09

Volumn , Issue , 2009, Pages

  Xing, Jing ^a,b   Xiong, Jin ^a   Sun, Ninghui ^a   Ma, Jie ^a  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

^b UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

[No Author keywords available]

Indexed keywords

CACHE MANAGEMENT; CLUSTER FILE SYSTEMS; DIRECTORY STRUCTURE; FILE SYSTEMS; LOAD-BALANCE; MEMORY UTILIZATION; METADATA MANAGEMENT; METADATA MANAGEMENT SYSTEM; METADATA SERVERS; PARALLEL PROCESSING; PROTOTYPE IMPLEMENTATIONS; SCALE-UP; SYSTEM USE;

SOFTWARE PROTOTYPING;

METADATA;

EID: 74049134327     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1654059.1654086     Document Type: Conference Paper

References (24)

1. Large synoptic survey telescope. http://www.lsst.org/lsst, 2008.
2. P. Braam, M. Callahan, and P. Schwan. The intermezzo file system. In In Proceedings of the 3rd of the Perl Conference, Oa̧ŕReilly Open Source Convention, Monterey, 1999.
3. Peter J. Braam. The lustre storage architecture. 2004.
4. Peter F. Corbett and Dror G Feitelson. The vesta parallel file system. ACM Trans. Comput. Syst., 14(3):225-264, 1996.
5. DARPA/IPTO. Exascale computing study: Technology challenges in achieving exascale systems. 2008.
6. John R. Douceur and Jon Howell. Distributed directory service in the farsite file system. In OSDI '06: Proceedings of the 7th symposium on Operating systems design and implementation, pages 321-334, Berkeley, CA, USA, 2006. USENIX Association.
7. Ronald Fagin, Jurg Nievergelt, Nicholas Pippenger, and H. Raymond Strong. Extendible hashing - a fast access method for dynamic files. ACM Trans. Database Syst., 4(3):315-344, 1979.
8. Gregory R. Ganger and M. Frans Kaashoek. Embedded inodes and explicit grouping: exploiting disk bandwidth for small files. In ATEC '97: Proceedings of the annual conference on USENIX Annual Technical Conference, pages 1-17, Berkeley, CA, USA, 1997. USENIX Association.
9. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The google file system. In SOSP '03: Proceedings of the nineteenth ACM symposium on Operating systems principles, pages 29-43, New York, NY, USA, 2003. ACM.
10. Todd Hoff. Flickr architecture. http://highscalability.com/flickr- architecture, 2007.
11. Yu Hua, Yifeng Zhu, Hong Jiang, Dan Feng, and Lei Tian. Scalable and adaptive metadata management in ultra large-scale file systems. In ICDCS '08: Proceedings of the 2008 The 28th International Conference on Distributed Computing Systems, pages 403-410, Washington, DC, USA, 2008. IEEE Computer Society.
12. Lawrence Livermore National Laboratory. mdtest-1.7.4. http://sourceforge.net/projects/mdtest/, 2007.
13. E. L. Miller and R. H. Katz. Rama: An easy-to-use, high-performance parallel file system. In Parallel Computing, pages 419-446, 1997.
14. John K. Ousterhout, Andrew R. Cherenson, Frederick Douglis, Michael N. Nelson, and Brent B. Welch. The sprite network operating system. Computer, 21(2):23-36, 1988.
15. Swapnil V. Patil and Garth Gibson. Giga+: scalable directories for shared file systems. http://highscalability.com/flickr-architecture, 2008.
16. Swapnil V. Patil, Garth Gibson, Sam Lang, and Milo Polte. Giga+: scalable directories for shared file systems. In PDSW '07: Proceedings of the 2nd international workshop on Petascale data storage, pages 26-29, 2007.
17. Brian Pawlowski, Chet Juszczak, Peter Staubach, Carl Smith, Diane Lebel, and David Hitz. Nfs version 3 - design and implementation. In In Proceedings of the Summer USENIX Conference, pages 137-152, 1994.
18. Daniel Phillips. A directory index for ext2. In ALS '01: Proceedings of the 5th annual Linux Showcase & Conference, pages 20-20, Berkeley, CA, USA, 2001. USENIX Association.
19. O. Rodeh and A. Teperman. zfs - a scalable distributed file system using object disks. In In Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies, pages 207-218, 2003.
20. Frank Schmuck and Roger Haskin. Gpfs: A shared-disk file system for large computing clusters. In FAST '02: Proceedings of the 1st USENIX Conference on File and Storage Technologies, pages 231-244, Berkeley, CA, USA, 2002. USENIX Association.
21. Adam Sweeney, Doug Doucette, Wei Hu, Curtis Anderson, Mike Nishimoto, and Geoff Peck. Scalability in the xfs file system. Proceedings. of the USENIX 1996 Annual Technical Conference, 1996.
22. Peter Vajgel. Needle in a haystack:efficient storage of billions of photos. http://www.facebook.com/note.php?note-id=76191543919, 2009.
23. Sage A. Weil, Kristal T. Pollack, Scott A. Brandt, and Ethan L. Miller. Dynamic metadata management for petabyte-scale file systems. In SC '04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing, page 4, Washington, DC, USA, 2004. IEEE Computer Society.
24. Yifeng Zhu, Hong Jiang, Jun Wang, and Feng Xian. Hba: Distributed metadata management for large cluster-based storage systems. IEEE Trans. Parallel Distrib. Syst., 19(6):750-763, 2008.