Availability in globally distributed storage systems

Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010

Volumn , Issue , 2019, Pages 61-74

  Ford, Daniel ^a   Labelle, François ^a   Popovici, Florentina I ^a   Stokely, Murray ^a   Truong, Van Anh ^a,b   Barroso, Luiz ^a   Grimes, Carrie ^a   Quinlan, Sean ^a  

^a GOOGLE INC   (United States)

^b Columbia University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AVAILABILITY; MULTIPROCESSING SYSTEMS; OUTAGES; SYSTEMS ANALYSIS;

CLOUD STORAGE SYSTEMS; DISTRIBUTED STORAGE SYSTEM; DISTRIBUTED SYSTEMS; INDIVIDUAL COMPONENTS; NETWORK CONNECTIVITY; RECOVERY TECHNIQUES; REPLICATION STRATEGIES; STORAGE INFRASTRUCTURE;

DIGITAL STORAGE;

EID: 85076902294     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. HDFS (Hadoop Distributed File System) architecture. http://hadoop.apache.org/common/docs/current/hdfs_design.html, 2009.
2. ANDREAS, E. S., HAEBERLEN, A., DABEK, F., GON CHUN, B., WEATHERSPOON, H., MORRIS, R., KAASHOEK, M. F., AND KUBIATOWICZ, J. Proactive replication for data durability. In Proceedings of the 5th Intl Workshop on Peer-to-Peer Systems (IPTPS) (2006).
3. BAIRAVASUNDARAM, L. N., GOODSON, G. R., PASUPATHY, S., AND SCHINDLER, J. An analysis of latent sector errors in disk drives. In SIGMETRICS'07: Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (2007), pp. 289-300.
4. BAIRAVASUNDARAM, L. N., GOODSON, G. R., SCHROEDER, B., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEA, R. H. An analysis of data corruption in the storage stack. In FAST'08: Proceedings of the 6th USENIX Conference on File and Storage Technologies (2008), pp. 1-16.
5. BAKER, M., SHAH, M., ROSENTHAL, D. S. H., ROUSSOPOULOS, M., MANIATIS, P., GIULI, T., AND BUNGALE, P. A fresh look at the reliability of long-term digital storage. In EuroSys'06: Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems (2006), pp. 221-234.
6. BAKKALOGLU, M., WYLIE, J. J., WANG, C., AND GANGER, G. R. Modeling correlated failures in survivable storage systems. In Fast Abstract at International Conference on Dependable Systems & Networks (June 2002).
7. BOLOSKY, W. J., DOUCEUR, J. R., ELY, D., AND THEIMER, M. Feasibility of a serverless distributed file system deployed on an existing set of desktop PCs. In SIGMETRICS'00: Proceedings of the 2000 ACM SIGMETRICS international conference on measurement and modeling of computer systems (2000), pp. 34-43.
8. BROWN, D. M. The first passage time distribution for a parallel exponential system with repair. In Reliability and fault tree analysis (1974), Defense Technical Information Center.
9. CHANG, F., DEAN, J., GHEMAWAT, S., HSIEH, W. C., WALLACH, D. A., BURROWS, M., CHANDRA, T., FIKES, A., AND GRUBER, R. E. Bigtable: a distributed storage system for structured data. In OSDI'06: Proceedings of the 7th Symposium on Operating Systems Design and Implementation (Nov. 2006), pp. 205-218.
10. CHARACTERISTICS, M. F., YALAG, P., NATH, S., YU, H., GIBBONS, P. B., AND SESHAN, S. Beyond availability: Towards a deeper understanding of machine failure characteristics in large distributed systems. In WORLDS'04: First Workshop on Real, Large Distributed Systems (2004).
11. CHUN, B.-G., DABEK, F., HAEBERLEN, A., SIT, E., WEATHERSPOON, H., KAASHOEK, M. F., KUBIATOWICZ, J., AND MORRIS, R. Efficient replica maintenance for distributed storage systems. In NSDI'06: Proceedings of the 3rd conference on Networked Systems Design & Implementation (2006), pp. 45-58.
12. CORBETT, P., ENGLISH, B., GOEL, A., GRCANAC, T., KLEIMAN, S., LEONG, J., AND SANKAR, S. Row-diagonal parity for double disk failure correction. In FAST'04: Proceedings of the 3rd USENIX Conference on File and Storage Technologies (2004), pp. 1-14.
13. EFRON, B., AND TIBSHIRANI, R. An Introduction to the Bootstrap. Chapman and Hall, 1993.
14. EPSTEIN, R. TheTheoryofGamblingandStatisticalLogic. Academic Press, 1977.
15. FELLER, W. An Introduction to Probability Theory and Its Application. John Wiley and Sons, 1968.
16. GHEMAWAT, S., GOBIOFF, H., AND LEUNG, S.-T. The Google file system. In SOSP'03: Proceedings of the 19th ACM Symposium on Operating Systems Principles (Oct. 2003), pp. 29-43.
17. ISARD, M. Autopilot: automatic data center management. ACM SIGOPS Opererating Systems Review 41, 2 (2007), 60-67.
18. JAVADI, B., KONDO, D., VINCENT, J.-M., AND ANDERSON, D. Mining for statistical models of availability in large-scale distributed systems: An empirical study of SETI@home (2009). pp. 1-10.
19. JIANG, W., HU, C., ZHOU, Y., AND KANEVSKY, A. Are disks the dominant contributor for storage failures?: a comprehensive study of storage subsystem failure characteristics. In FAST'08: Proceedings of the 6th USENIX Conference on File and Storage Technologies (2008), pp. 1-15.
20. KEETON, K., SANTOS, C., BEYER, D., CHASE, J., AND WILKES, J. Designing for disasters. In FAST'04: Proceedings of the 3rd USENIX Conference on File and Storage Technologies (2004), pp. 59-62.
21. LIAN, Q., CHEN, W., AND ZHANG, Z. On the impact of replica placement to the reliability of distributed brick storage systems. In ICDCS'05: Proceedings of the 25th IEEE International Conference on Distributed Computing Systems (2005), pp. 187-196.
22. MCKUSICK, M. K., AND QUINLAN, S. GFS: Evolution on fast-forward. Communications of the ACM 53, 3 (2010), 42-49.
23. MICKENS, J. W., AND NOBLE, B. D. Exploiting availability prediction in distributed systems. In NSDI'06: Proceedings of the 3rd conference on Networked Systems Design & Implementation (2006), pp. 73-86.
24. NATH, S., YU, H., GIBBONS, P. B., AND SESHAN, S. Subtleties in tolerating correlated failures in wide-area storage systems. In NSDI'06: Proceedings of the 3rd conference on Networked Systems Design & Implementation (2006), pp. 225-238.
25. PINHEIRO, E., WEBER, W.-D., AND BARROSO, L. A. Failure trends in a large disk drive population. In FAST'07: Proceedings of the 5th USENIX conference on File and Storage Technologies (2007), pp. 17-23.
26. RAMABHADRAN, S., AND PASQUALE, J. Analysis of long-running replicated systems. In INFOCOM 2006. 25th IEEE International Conference on Computer Communications (2006), pp. 1-9.
27. RESNICK, S. I. Adventures in stochastic processes. Birkhauser Verlag, 1992.
28. RODRIGUES, R., AND LISKOV, B. High availability in DHTs: Erasure coding vs. replication. In Proceedings of the 4th International Workshop on Peer-to-Peer Systems (2005).
29. SCHROEDER, B., AND GIBSON, G. A. Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you? In FAST'07: Proceedings of the 5th USENIX conference on File and Storage Technologies (2007), pp. 1-16.
30. SCHROEDER, B., PINHEIRO, E., AND WEBER, W.-D. DRAM errors in the wild: a large-scale field study. In SIGMETRICS '09: Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems (2009), pp. 193-204.
31. SCHWARZ, T. J. E., XIN, Q., MILLER, E. L., LONG, D. D. E., HOSPODOR, A., AND NG, S. Disk scrubbing in large archival storage systems. International Symposium on Modeling, Analysis, and Simulation of Computer Systems (2004), 409-418.
32. T., S. Generalized Reed Solomon codes for erasure correction in SDDS. In WDAS-4: Workshop on Distributed Data and Structures (2002).
33. WEATHERSPOON, H., AND KUBIATOWICZ, J. Erasure coding vs. replication: A quantitative comparison. In IPTPS'01: Revised Papers from the First International Workshop on Peer-to-Peer Systems (2002), Springer-Verlag, pp. 328-338.
34. WEATHERSPOON, H., MOSCOVITZ, T., AND KUBIATOWICZ, J. Introspective failure analysis: Avoiding correlated failures in peer-to-peer systems. In SRDS'02: Proceedings of the 21st IEEE Symposium on Reliable Distributed Systems (2002), pp. 362-367.
35. XIN, Q., MILLER, E. L., SCHWARZ, T., LONG, D. D. E., BRANDT, S. A., AND LITWIN, W. Reliability mechanisms for very large storage systems. In MSS'03: Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies (2003), pp. 146-156.