Constructing collaborative desktop storage caches for large scientific datasets

ACM Transactions on Storage

Volumn 2, Issue 3, 2006, Pages 221-254

  Vazhkudai, Sudharshan S ^a   Ma, Xiaosong ^b,c   Freeh, Vincent W ^b,c   Strickland, Jonathan W ^b,c   Tammineedi, Nandan ^b,c   Simon, Tyler ^a   Scott, Stephen L ^a  

^a OAK RIDGE NATIONAL LABORATORY   (United States)

^b NORTH CAROLINA STATE UNIVERSITY   (United States)

^c North Carolina State University   (United States)

Author keywords

Distributed storage; Parallel I O; Scientific data management; Server less storage system; Storage cache; Storage networking; Storage resoucce management; Storage scavenging; Striped storage

Indexed keywords

DISTRIBUTED STORAGE; PARALLEL I/O; SCIENTIFIC DATA MANAGEMENT; SERVER LESS STORAGE SYSTEMS; STORAGE CACHE; STORAGE NETWORKING; STORAGE RESOURCE MANAGEMENT; STORAGE SCAVENGING; STRIPED STORAGE;

BANDWIDTH; COMPUTER SIMULATION; COMPUTER SUPPORTED COOPERATIVE WORK; COMPUTER WORKSTATIONS; DATA PROCESSING; DATA STORAGE EQUIPMENT; PERSONAL COMPUTERS;

DATABASE SYSTEMS;

EID: 33750485344     PISSN: 15533077     EISSN: 15533093     Source Type: Journal    
DOI: 10.1145/1168910.1168911     Document Type: Article

References (60)

1. ADYA, A., BOLOSKY, W., CASTRO, M., CHAIKEN, R., CEHMAK, G., DOUCEUR, J., HOWELL, J., LORCH, J., THEIMER, M., AND WATTENHOFER, R. 2002. FARSITE: Federated, available, and reliable storage for an incompletely trusted environment. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation. Boston. MA. 36, 1-14.
2. ANDERSON, D., COBB, J., KORPELA, E., LEBOFSKY, M., AND WEHTHIMER, D. 2002. SETI@home: An experiment in public-resource computing. Comm. ACM 45, 11, 56-61.
3. ANNAPUREDDY, S., FREEDMAN, M. J., AND MAZIERES, D. 2005. Shark: Scaling file servers via cooperative caching. In Proceedings of the 2nd USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI'). Boston, MA. 129-142.
4. AVERY, P. AND FOSTER, I. 2001. The griphyn project: Towards petascale virtual data grids. Tech. rep. GriPhyn-2001-15, http://www.griphyn.org.
5. BAIR, R., DIACHIN, L., KENT, S., MICHAELS, G., MEZZACAPPA, T., MOUNT, R., PORDES, R., RAHN, L., SHOSHANI, A., STEVENS, R., AND WILLIAMS, D. 2004. Planning ascr/office of science data-management strategy. In Proceedings of the Department of Energy Office of Science Data Management Workshop. Menlo Park, CA.
6. BECK, M., MOORE, T., AND PLANK, J. 2002. An end-to-end approach to globally scalable network storage. In Proceedings of the ACM SIGCOMM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication. Pittsburgh, PA. 32, 339-346.
7. BENT, J., THAIN, D., ARPACI-UUSSEAU, A., ARPACI-UUSSEAU, R., AND LIVNY, M. 2004. Explicit control in a batch aware distributed file system. In Proceedings of the 1st USENIX/ACM Conference on Networked Systems Design and Implementation. San Francisco CA. 365-378.
8. BENT, J., VENKATARAMANI, V., LEROY, N., ROY, A., STANLEY, J., ARPACI-DUSSEAU, A., ARPACI-DUSSEAU, R., AND LIVNY, M. 2002. Flexibility, manageability, and performance in a grid storage appliance. In Proceedings of the 11th High Performance Distributed Computing Symposium, Edinburgh, UK. 3-12.
9. BESTER, J., FOSTER, I., KESSELMAN, C., TEDESCO, J., AND TUECKE, S. 1999. GASS: A data movement and access service for wide area computing systems. In Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems. Atlanta, GA. 78-88.
10. BUTT, A., JOHNSON, T., ZHENG, Y., AND Hu, Y. 2004. Kosha: A peer-to-peer enhancement for the network file system. In Proceedings of the ACM/IEEE Conference on Supercomputing. Pittsburgh, PA. 51.
11. CABRERA, L.-F. AND LONG, D. D. E. 1991. SWIFT: Using distributed disk striping to provide high I/O data rates. Comput. Syst. 4, 4, 405-436.
12. GARNS, P., III, W. L., ROSS, R., AND THAKUR, R. 2000. PVFS: A parallel file system for linux clusters. In Proceedings of the 4th Annual Linux Showcase and Conference. Atlanta, GA. 317-327.
13. CHIEN, A., CALDER, B., ELBERT, S., AND BHATIA, K. 2003. Entropia: Architecture and performance of an enterprise desktop grid system. J. Parallel Distrib. Comput. 63, 5, 597-610.
14. CLARKE, I., SANDBERG, O., WILEY, B., AND HONG, T. W. 2000. Freenet: A distributed anonymous information storage and retrieval system. In Proceedings of the ICSI Workshop on Design Issues in Anonymity and Unobservability. Berkeley, CA. 2009, 46-66.
15. CLUSTER FILE SYSTEMS, INC. 2002. Lustre: A scalable, high-performance file system. http://www.lustre.org/docs/whitepaper.pdf.
16. COHEN, B. 2003. Incentives build robustness in bittorrent. In Proceedings of the 1st Workshop on Economics of Peer-to-Peer Systems. Berkeley, CA. 251-260.
17. Cox, L., MURRAY, C., AND NOBLE, B. 2002. Pastiche: Making backup cheap and easy. In Proceedings of the OSDI Conference. Boston, MA. 285-298.
18. COYNE, R. AND WATSON, R. 1995. The parallel I/O architecture of the high-performance storage system (hpss). In Proceedings of the IEEE MSS Symposium. Monterey, CA. 27-44.
19. CROWCROFT, J. AND PRATT, I. 2002. Peer to peer: Peering into the future. In NETWORKING Tutorials. 1-19.
20. DABEL, F., KAASHOEK, M., KARGER, D., MORRIS, R., AND STOICA, I. 2001. Wide-Area cooperative storage with cfs. In Proceedings of the SOSP Conference. Banff, Alberta, Canada. 202-215.
21. DAHLIN, M., WANG, R., ANDERSON, T. E., AND PATTERSON, D. A. 1994. Cooperative caching: Using remote client memory to improve file system performance. In Proceedings of the Conference Operating Systems Design and Implementation. Monterey, CA. 267-280.
22. DOUCEUR, J. AND BOLOSKY, W. 1999. A large-scale study of file-system contents. In Proceedings of the Conference SIGMETRICS. Atlanta, GA. 59-70.
23. DRUSCHEL, P. AND ROWSTRON, A. 2001. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In Proceedings of the 18th ACM Symposium on Operating System Principles. Banff, Alberta, Canada. 188-201.
24. FEELEY, M. J., MORGAN, W. E., PIGHIN, F. H., KARLIN, A. R., LEVY, H. M., AND THEKKATH, C. A. 1995. Implementing global memory management in a workstation cluster. In Proceedings of the 15th ACM Symposium on Operating Systems Principles. Copper Mtn., CO. 129-140.
25. FROLUND, S., MERCHANT, A., SAITO, Y., SPENCE, S., AND VEITCH, A. 2003. Fab: Enterprise storage systems on a shoestring. In Proceedings of the Hot Topics in Operating System Conference. Lihue, HI. 133-138.
26. GADDE, S., CHASE, J., AND RABINOVICH, M. 1998. A taste of crispy squid. In Proceedings of the Workshop on Internet Server Performance. Madison, WI. 129-136.
27. GAREY, M. AND JOHNSON, D. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness, 4th ed. W. H. Freeman.
28. GHEMAWAT, S., GOBIOFF, H., AND LEUNG, S. 2003. The Google file system. In Proceedings of the 19th Symposium on Operating Systems Principles. Lake George, NY. 29-43.
29. GRAY, J., LIU, D., NIETO-SANTISTEBAN, M., SZALAY, A., HEBER, G., AND DEWITT, D. 2005. Scientific data management in the coming decade. Tech. rep. MSR-TR-2005-10, Microsoft.
30. GRAY, J. AND SZALAY, A. S. 2003. Scientific data federation. In The Grid 2: Blueprint for a New Computing Infrastructure, I. Foster and C. Kesselman, eds. Elsevier, New York. 95-108.
31. GUMMADI, P., DUNN, R., SAROIU, S., GRIBBLE, S., LEVY, H., AND ZAHORJAN, J. 2003. Measurement, modeling, and analysis of a peer-to-peer file-sharing workload. In Proceedings of the 19th Symposium on Operating Systems Principles. Lake George, NY. 314-329.
32. GUPTA, A., LIN, B., AND DINDA, P. 2004. Measuring and understanding user comfort with resource borrowing. In Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing. Honolulu, HI. 00, 214-224.
33. HARTMAN, J. AND OUSTERHOUT, J. 1995. The Zebra striped network file system. ACM Trans. Comput. Syst. 13, 3, 274-310.
34. HOWARD, J. H. 1988. An overview of the andrew file system. In Proceedings of the USENIXWinter technical Conference. Dallas, TX. 23-26.
35. IAMNITCHI, A., RIPEANU, M., AND FOSTER, I. 2004. Small-World file-sharing communities. In Proceedings of the INFOCOM Conference. Hong Kong. 2, 952-963.
36. IYER, S., ROWSTRON, A., AND DRUSCHEL, P. 2002. Squirrel: A decentralized peer-to-peer web cache. In Proceedings of the 21st ACM Symposium on Principles of Distributed Computing. Monterey, CA. 213-222.
37. KUBIATOWICZ, J., BINDEL, D., CHEN, Y., CZERWINSKI, S., EATON, P., GEELS, D., GUMMADI, R., RHEA, S., WEATHERSPOON, H., WEIMER, W., WELLS, C., AND ZHAO, B. 2000. Oceanstore: An architecture for global-scale persistent storage. In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems. Cambridge, MA. 28, 190-201.
38. LEE, E. AND THEKKATH, C. 1996. Petal: Distributed virtual disks. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Cambridge, MA. 84-92.
39. LEE, J., MA, X., Ross, R., THAKUR, R., AND WINSLETT, M. 2004. RFS: Efficient and flexible remote file access for MPI-IO. In Proceedings of the IEEE International Conference on Cluster Computing. San Diego, CA. 71-81.
40. LEE, J., MA, X., WINSLETT, M., AND Yu, S. 2002. Active buffering plus compressed migration: An integrated solution to parallel simulations' data transport needs. In Proceedings of the 16th ACM International Conference on Supercomputing. New York. 156-166.
41. LITZKOW, M., LIVNY, M., AND MUTKA, M. 1988. Condor - A hunter of idle workstations. In Proceedings of the 8th International Conference on Distributed Computing Systems. San Jose, CA. 104-111.
42. MARKATOS, E. 2002. Tracing a large-scale peer to peer system: An hour in the life of gnutella. In Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Orid. Berlin, Germany. 65-74.
43. MORRIS, J. H., SATYANARAYANAN, M., CONNER, M., HOWARD, J., ROSENTHAL, D., AND SMITH, F. 1986. Andrew: A distributed personal computing environment. Comm. ACM 29, 3, 184-201.
44. MUTHITACHAROEN, A., MORRIS, R., GIL, T., AND CHEN, B. 2002. Ivy: A read/write peer-to-peer file system. In Proceedings of the OSDI Conference. Boston, MA. 31-44.
45. ncbi. 2005. National center for biotechnology information, http://www.ncbi.nlm.nih.gov/.
46. NOVAES, R., ROISENBERG, P., SCHEEH, R., NOHTHFLEET, C., JORNADA, J., AND CIHNE, W. 2005. Non-dedicated distributed environment: A solution for safe and continuous exploitation of idle cycles. Scalable Comput. Practice Experi. 6, 3, 107-115.
47. NOWICKI, B. 1989. NFS: Network File System Protocol Specification. Network Working Group RFC1094.
48. OTOO, E. J., ROTEM, D., AND ROMOSAN, A. 2004. Optimal file-bundle caching algorithms for datagrids. In Proceedings of the ACM/IEEE Conference on Supercomputing. Pittsburgh, PA. 6.
49. POPEK, G. AND WALKER, B. J. 1985. The LOCUS Distributed System Architecture. MIT Press, Combridge, MA.
50. RATNASAMY, S., FRANCIS, P., HANDLEY, M., KARP, R., AND SCHENKER, S. 2001. A scalable content-addressable network. In Proceedings of the ACM SIGCOMM Conference. San Diego, CA. 161-172.
51. RIVEST, R. 1992. The MD5 Message-Digest Algorithm. Network Working Group RFC1321.
52. SARKAR, P. AND HARTMAN, J. 1996. Efficient cooperative caching using hints. In Proceedings of the 2nd ACM Symposium on Operating Systems Design and Implementation (OSDI). Seattle, WA. 35-46.
53. SATYANARAYANAN, M., KISTLER, J. J., KUMAR, P., OKASAKI, M. E., SIEGEL, E. H., AND STEERE, D. C. 1990. Coda: A highly available file system for a distributed workstation environment. ACM Trans. Comput. Syst. 39, 4, 447-459.
54. SCHMUCK, F. AND HASKIN, R. 2002. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 1st Conference on File and Storage Technologies. Monterey, CA. 19, 231-244.
55. SHARMAN NETWORKS, INC. 2005. The kazaa media desktop, http://www.kazaa.com.
56. STRICKLAND, J., FREER, V., MA, X., AND VAZHKUDAI, S. 2005. Governor: Autonomic throttling for aggressive idle resource scavenging. In Proceedings of the 2nd IEEE International Conference on Autonomic Computing. Seattle, WA. 64-75.
57. SZALAY, A. AND GRAY, J. 2001. The world-wide telescope. Science 293, 14, 2037-2040.
58. THE ASTROPHYSICAL RESEARCH CONSORTIUM. 2005. Sloan digital sky survey project book. http://www.astro.princeton.edu/PBOOK/welcome.htm.
59. THEKKATH, C., MANN, T., AND LEE, E. 1997. Frangipani: A scalable distributed file system. In Proceedings of the 16th Symposium on Operating Systems Principles. Saint-Malo. 31, 224-237.
60. VAZHKUDAI, S. AND SCHOPF, J. 2003. Using regression techniques to predict large data transfers. High Performance Comput. Appli. Special Issue on Grid Computing: Infrastructure Appli. 17, 3, 249-268.