A self-organizing storage cluster for parallel data-intensive applications

IEEE/ACM SC2004 Conference, Proceedings

Volumn , Issue , 2004, Pages 635-647

  Tang, Hong ^b   Gulbeden, Aziz ^a   Zhou, Jingyu ^a   Strathearn, William ^a   Yang, Tao ^a,b   Chu, Lingkun ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b AskJeeves ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA-INTENSIVE APPLICATIONS; HASHING; SORRENTO;

BENCHMARKING; DATA REDUCTION; PERFORMANCE;

PARALLEL PROCESSING SYSTEMS;

EID: 23944437100     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (53)

1. Ask Jeeves, Inc. http://www.ask.com/.
2. CERN (Conseil Européen pour la Recherche Nucléaire). http://www.cern.ch/.
3. Lustre, http://www.lustre.org/.
4. Project: AVFS: A Virtual Filesystem. http://sourceforge.net/projects/avf/.
5. SPEC SFS97_R1 Results, http://www.spec.org/osg/sfs97r1/results/sfs97r1.html.
6. The Wellcome Trust Sanger Institute. http://www.sanger.ac.uk/.
7. NFS: Network File System version 3 protocol specification. Technical Report SUN Microsystems, 1994.
8. S. F. Altschul, W. Gish, W. Miller, E. W. Myers, and D. J. Lipman. Basic Local Alignment Search Tool. Journal of Molecular Biology, 215, 1990.
9. D. Anderson. Object based storage devices: A command set proposal. Technical report, National Storage Industry Consortium, October 1999.
10. D. Anderson, J. Chase, and A. Vahdat. Interposed request routing for scalable network storage. In OSDI, 2000.
11. T. Anderson, Y. Breitbart, H. Korth, and A. Wool. Replication, consistency, and practicality: Are these mutually exclusive? In SIGMOD, 1998.
12. T. Anderson, M. Dahlin, J. Neefe, D. Patterson, D. Roselli, and R. Wang. Serverless network file systems. In SOSP, 1995.
13. M. Baker, J. Hartman, M. Kupfer, K. Shirriff, and J. Ousterhout. Measurements of a distributed file system. In SOSP, 1991.
14. A. Brinkmann, K. Salzwedel, and C. Scheideler. Compact, adaptive placement schemes for non-uniform requirements. In SPAA, 2002.
15. L.-F. Cabrera and D. Long. Swift: Using distributed disk striping to provide high I/O data rates. Technical Report UCSC-CRL-91-46, 1991.
16. P. Carns, W. Ligon III, R. Ross, and R. Thakur. PVFS: A parallel file system for linux clusters. In Proc. of the 4th Annual Linux Showcase and Conf., 2000.
17. C. Chang, B. Moon, A. Acharya, C. Shock, A. Sussman, and J. Saltz. Titan: a high-performance remote-sensing database. In ICDE, 1997.
18. F. Dabek, F. Kaashoek, D. Karger, R. Morris, and I. Stoica. Wide-area cooperative storage with CFS. In SOSP, 2001.
19. A. Fox and E. A. Brewer. Harvest, Yield, and Scalable Tolerant Systems. In HotOS-VII, 1999.
20. G. R. Ganger and M. F. Kaashoek. Embedded Inodes and Explicit Grouping: Exploiting Disk Bandwidth for Small Files. In USENIX Annual Technical Conference, 1997.
21. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. In SOSP, 2003.
22. G. A. Gibson and R. Van Meter. Network attached storage architecture. Communications of ACM, 43(11), 2000.
23. J. Gray, P. Heiland, P. O'Neil, and D. Shasha. The dangers of replication and a solution. In SIGMOD, 1996.
24. J. Hartman, I. Murdock, and T. Spalink. The Swarm scalable storage system. In Proc. of Intl. Conf. on Distributed Computing Systems, 1999.
25. J. Hartman and J. Ousterhout. The Zebra striped network file system. TOCS, 13(3), 1995.
26. K. Hildrum, J. Kubiatowicz, S. Rao, and B. Zhao. Distributed object location in a dynamic network. In SPAA, 2002.
27. R. J. Honicky and E. L. Miller. A fast algorithm for online placement and reorganization of replicated data. In IEEE International Parallel and Distributed Processing Symposium, 2003.
28. R. J. Honicky and E. L. Miller. Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution. In IEEE International Parallel and Distributed Processing Symposium, 2004.
29. J. H. Howard, M. L. Kazar, S. G. Menees, D. A. Nichols, M. Satyanarayanan, R. N. Sidebotham, and M. J. West. Scale and performance in a distributed file system. TOCS, 6(1), 1988.
30. D. Karger, E. Lehman, T. Leighton, M. Levine, D. Levin, and R. Panigraphy. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web. In STOC, 1997.
31. J. Kistler and M. Satyanarayanan. Disconnected operation in the Coda file system. 10(1), 1992.
32. J. Kubiatowicz, D. Bindel, Y. Chen, P. Eaton, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoon, W. Weimer, C. Wells, and B. Zhao. OceanStore: An architecture for global-scale persistent storage. In ASPLOS, 2000.
33. E. Lee and C. Thekkath. Petal: Distributed virtual disks. In ASPLOS, 1996.
34. S. Mullender and A. Tanenbaum. A distributed file service based on optimistic concurrency control. In SOSP, 1985.
35. M. Olson, K. Bostic, and M. Seltzer. Berkeley DB. In Proc. of USENIX Tech. Conf., FREENIX Track, 1999.
36. D. Oppenheimer, A. Ganapathi, and D. A. Patterson. Why do internet services fail, and what can be done about it? In USITS, 2003.
37. S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Schenker. A scalable content-addressable network. In SIGCOMM, 2001.
38. D. P. Reed and L. Svobodova. SWALLOW: a distributed data storage system for a local network. In Local Networks for Computer Communications, pages 355-373, North-Holland, Amsterdam, 1981.
39. Y. Saito, B. N. Bershad, and H. M. Levy. Manageability, Availability and Performance in Porcupine: A Highly Scalable, Cluster-based Mail Service. In Proc. of the 17th SOSP, pages 1-15, 1999.
40. D. S. Santry, M. J. Feeley, N. C. Hutchinson, A. C. Veitch, R. W. Carton, and J. Ofir. Deciding when to forget in the Elephant file system. In SOSP, 1999.
41. F. Schmuck and R. Haskin. GPFS: A shared-disk file system for large computing clusters. In FAST, 2002.
42. K. Shen, T. Yang, L. Chu, J. L. Holliday, D. A. Kuschner, and H. Zhu. Neptune: Scalable replication management and programming support for cluster-based network services. In USITS, 2001.
43. C. A. Soules, G. R. Goodson, I. D. Strunk, and G. R. Ganger. Metadata efficiency in versioning file systems. In FAST, 2003.
44. I. Stoica, R. Morris, D. Karger, F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. In SIGCOMM, 2001.
45. H. Tang, A. Gulbeden, J. Zhou, L. Chu, and T. Yang. Sorrento: a self-organizing storage cluster for parallel data-intensive applications. Technical Report 2003-30, UCSB, September 2003.
46. H. Tang and T. Yang. An efficient data location protocol for self-organizing storage clusters. In Proc. of ACM/IEEE SC'03, 2003.
47. D. Terry, M. Theimer, K. Petersen, A. Demers, M. Spreitzer, and C. Hauser. Managing update conflicts in Bayou, a weakly connected replicated storage system. In SOSP, 1995.
48. W. Vogels. File system usage in Windows NT 4.0. In SOSP, 1999.
49. G. Weikum and G. Vossen. Transactional information systems - theory, algorithms, and the practice of concurrency control and recovery. Morgan Kaufmann, 2002.
50. R. F. V. D. Wijngaart and P. Wong. NAS Parallel Benchmarks I/O Version 2.4. Technical report, NASA Advanced Supercomputing (NAS), 2003.
51. J. Wilkes. Data services - from data to containers. FAST Keynote Speech, 2003.
52. Q. Xin, E. L. Miller, T. Schwarz, S. A. Brandt, and D. D. E. Long. Reliability mechanisms for very large storage systems. In MSST, 2003.
53. H. Yu and A. Vahdat. Design and Evaluation of a Continuous Consistency Model for Replicated Services. In Proc. of the 4th OSDI, Oct. 2000.