Network coding for distributed storage systems

IEEE Transactions on Information Theory

Volumn 56, Issue 9, 2010, Pages 4539-4551

  Dimakis, Alexandros G ^a   Godfrey, P Brighten ^b   Wu, Yunnan ^c   Wainwright, Martin J ^d   Ramchandran, Kannan ^d  

^a University of Southern California ^*  (United States)

^b University of Illinois at Urbana Champaign   (United States)

^c MICROSOFT RESEARCH   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Distributed storage; network coding; peer to peer storage; regenerating codes

Indexed keywords

APPLICATION SCENARIO; CODED SYSTEM; DATA CENTERS; DATA OBJECTS; DISTRIBUTED STORAGE; DISTRIBUTED STORAGE SYSTEM; ERASURE CODES; NETWORK CODING; OPTIMAL TRADEOFFS; PEER-TO-PEER STORAGE; PEER-TO-PEER STORAGE SYSTEMS; REGENERATING CODES; SINGLE NODE FAILURE;

DATA STORAGE EQUIPMENT; DISTRIBUTED COMPUTER SYSTEMS; INFORMATION THEORY; OPTIMIZATION; QUALITY ASSURANCE; REDUNDANCY; REPAIR;

PEER TO PEER NETWORKS;

EID: 77955726417     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2010.2054295     Document Type: Article

References (40)

1. A. G. Dimakis, P. G. Godfrey, M. J. Wainwright, and K. Ramchandran, "Network coding for distributed storage systems," presented at the IEEE INFOCOM, Anchorage, AK, May 2007.
2. Y. Wu, A. G. Dimakis, and K. Ramchandran, "Deterministic regenerating codes for distributed storage," presented at the Allerton Con. Control, Computing, and Communication, Urbana-Champaign, IL, Sep. 2007.
3. S. Rhea, C. Wells, P. Eaton, D. Geels, B. Zhao, H. Weatherspoon, and J. Kubiatowicz, "Maintenance-free global data storage," IEEE Internet Comput., pp. 40-49, Sep. 2001.
4. R. Bhagwan, K. Tati, Y.-C. Cheng, S. Savage, and G. M. Voelker, "Total recall: System support for automated availability management," presented at the Symp. Networked Systems Design and Implementation (NSDI), 2004.
5. F. Dabek, J. Li, E. Sit, J. Robertson, M. Kaashoek, and R. Morris, "Designing a DHT for low latency and high throughput," presented at the Symp. Networked Systems Design and Implementation (NSDI), 2004.
6. A. G. Dimakis, V. Prabhakaran, and K. Ramchandran, "Decentralized erasure codes for distributed networked storage," Trans. Inf. Theory IEEE/ACM Trans. Netw., Jun. 2006, joint issue.
7. A. Kamra, J. Feldman, V. Misra, and D. Rubenstein, "Growth codes: Maximizing sensor network data persistence," presented at the ACM SIGCOMM, 2006.
8. S. Rhea, P. Eaton, D. Geels, H. Weatherspoon, B. Zhao, and J. Kubiatowicz, "Pond: The OceanStore prototype," presented at the USENIX File and Storage Technologies (FAST), 2003.
9. H. Weatherspoon and J. D. Kubiatowicz, "Erasure coding vs. replication: A quantitiative comparison," in Proc. IPTPS, 2002.
10. R. Rodrigues and B. Liskov, "High availability in DHTs: Erasure coding vs. replication," presented at the IPTPS, 2005.
11. M. Luby,M.Mitzenmacher,M. A. Shokrollahi, and D. Spielman, "Improved low-density parity check codes using irregular graphs," IEEE Trans. Inf. Theory, vol.47, pp. 585-598, Feb. 2001.
12. M. Luby, "LT codes," presented at the Proc. IEEE Foundations of Computer Science (FOCS), 2002.
13. A. Shokrollahi, "Raptor codes," IEEE Trans. Inf. Theory, vol.52, no.6, pp. 2551-2567, Jun. 2006.
14. M. Blaum, J. Brady, J. Bruck, and J. Menon, "EVENODD: An efficient scheme for tolerating double disk failures," IEEE Trans. Comput., vol.44, pp. 192-202, Feb. 1995.
15. L. Xu and J. Bruck, "X-code:MDSarray codes with optimal encoding," IEEE Trans. Inf. Theory, vol.45, no.1, pp. 272-276, Jan. 1999.
16. C. Huang and L. Xu, "STAR: An efficient coding scheme for correcting triple storage node failures," presented at the FAST-2005: 4th Usenix Conf. File and Storage Technologies, San Francisco, CA, Dec. 2005.
17. J. L. Hafner, "WEAVER codes: Highly fault tolerant erasure codes for storage systems," presented at the FAST-2005: 4th Usenix Conf. File and Storage Technologies, San Francisco, CA, Dec. 2005.
18. J. S. Plank, "Erasure codes for storage applications," presented at the Tutorial, FAST-2005: 4th Usenix Conf. File and Storage Technologies, San Francisco, CA, Dec. 2005.
19. R. Ahlswede, N. Cai, S.-Y. R. Li, and R. W. Yeung, "Network information flow," IEEE Trans. Inf. Theory, vol.46, pp. 1204-1216, Jul. 2000.
20. S.-Y. R. Li, R. W. Yeung, and N. Cai, "Linear network coding," IEEE Trans. Inf. Theory, vol.49, no.2, pp. 371-381, Feb. 2003.
21. R.Koetter and M. Médard, "An algebraic approach to network coding," IEEE/ACM Trans. Netw., Oct. 2003.
22. T. Ho, M. Médard, R. Koetter, D. R. Karger, M. Effros, J. Shi, and B. Leong, "A random linear network coding approach to multicast," IEEE Trans. Inf. Theory, vol.52, pp. 4413-4430, Oct. 2006. (Pubitemid 46445282)
23. P. Sanders, S. Egner, and L. Tolhuizen, "Polynomial time algorithms for network information flow," in Proc. Symp. Parallel Algorithms and Architectures (SPAA), San Diego, CA, Jun. 2003, pp. 286-294.
24. S. Jaggi, P. Sanders, P. A. Chou, M. Effros, S. Egner, K. Jain, and L. Tolhuizen, "Polynomial time algorithms for network code construction," IEEE Trans. Inf. Theory, vol.51, no.1, pp. 1973-1982, Jun. 2005.
25. A. G. Dimakis, V. Prabhakaran, and K. Ramchandran, "Ubiquitous acess to distributed data in large-scale sensor networks through decentralized erasure codes," presented at the IEEE/ACM Int. Symposium on Information Processing in Sensor Networks (IPSN), Apr. 2005.
26. C. Huang, M. Chen, and J. Li, "Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems," presented at the IEEE Int. Symp. Network Computing and Applications, Jul. 2007.
27. D. Wang, Q. Zhang, and J. Liu, "Partial network coding: Theory and application for continuous sensor data collection," presented at the 14th IEEE International Workshop on Quality of Service (IWQoS), 2006.
28. A. Jiang, "Network coding for joint storage and transmission with minimum cost," presented at the Int. Symp. Information Theory (ISIT), Jul. 2006.
29. F. Dabek, F. Kaashoek, D. Karger, R.Morris, and I. Stoica, "Wide-area cooperative storage with CFS," presented at the ACM SOSP, 2001.
30. A. Rowstron and P. Druschel, "Storage management and caching in past, a large-scale, persistent peer-to-peer storage utility," presented at the ACM SOSP, 2001.
31. K. Tati and G. M. Voelker, "On object maintenance in peer-to-peer systems," presented at the IPTPS, 2006.
32. P. B. Godfrey, S. Shenker, and I. Stoica, "Minimizing churn in distributed systems," presented at the ACM SIGCOMM, 2006.
33. B.-G. Chun, F. Dabek, A. Haeberlen, E. Sit, H. Weatherspoon, M. F. Kaashoek, J. Kubiatowicz, and R. Morris, "Efficient replica maintenance for distributed storage systems," presented at the Symp. Networked Systems Design and Implementation (NSDI), 2006.
34. J. Stribling, Planetlab all Pairs Ping [Online]. Available: http://infospect. planet-lab.org/pings
35. W. J. Bolosky, J. R. Douceur, D. Ely, and M. Theimer, "Feasibility of a serverless distributed file system deployed on an existing set of desktop PCs," presented at the SIGMETRICS 2000.
36. S. Guha, N. Daswani, and R. Jain, "An experimental study of the Skype peer-to-peer VoIP system," presented at the IPTPS 2006.
37. S. Saroiu, P. K. Gummadi, and S. D. Gribble, "A measurement study of peer-to-peer file sharing systems," presented at the MMCN San Jose, CA, Jan. 2002.
38. C. Gkantsidis, J. Miller, and P. Rodriguez, "Anatomy of a P2P content distribution system with network coding," presented at the IPTPS 2006.
39. Online Wiki Bibliography for Distributed Storage Papers [Online]. Available: http://csi.usc.edu/~dimakis/StorageWiki
40. J. Bang-Jensen and G. Gutin, Digraphs: Theory, Algorithms and Applications. New York: Springer, 2001.