A Low-Cost Multi-failure Resilient Replication Scheme for High Data Availability in Cloud Storage

Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016

Volumn , Issue , 2017, Pages 242-251

  Liu, Jinwei ^a   Shen, Haiying ^b  

^a AMRL   (United States)

^b University of Virginia   (United States)

Author keywords

Cloud storage; Data availability; Replication; SLA

Indexed keywords

AVAILABILITY; COSTS; FAILURE (MECHANICAL); INTEGER PROGRAMMING; MAINTENANCE;

CLOUD STORAGE SYSTEMS; CLOUD STORAGES; CONSISTENCY MAINTENANCE; DATA AVAILABILITY; MINIMIZING COSTS; NON-LINEAR INTEGER PROGRAMMING; NUMERICAL RESULTS; REPLICATION;

DIGITAL STORAGE;

EID: 85015161757     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/HiPC.2016.036     Document Type: Conference Paper

References (48)

1. K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The hadoop distributed file system. In MSST, pages 1-10, 2010.
2. D. Ongaro, S. Rumble, R. Stutsman, J. Ousterhout, and M. Rosenblum. Fast crash recovery in ramcloud. In Proc. of SOSP, pages 29-41, 2011.
3. S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system. In Proc. of SOSP, pages 29-43, 2003.
4. B. Calder, J. Wang, A. Ogus, N. Nilakantan, A. Skjolsvold, S. McKelvie, Y. Xu, S. Srivastav, J. Wu, H. Simitci, J. Haridas, C. Uddaraju, H. Khatri, A. Edwards, V. Bedekar, S. Mainali, R. Abbasi, A. Agarwal, M. Haq, M. Haq, D. Bhardwaj, S. Dayanand, A. Adusumilli, M. McNett, S. Sankaran, K. Manivannan, and L. Rigas. Windows azure storage: a highly available cloud storage service with strong consistency. In Proc. of SOSP, 2011.
5. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon's highly available key-value store. In Proc. of SOSP, 2007.
6. V. Rawat. Reducing failure probability of cloud storage services using multi-clouds. In Proc. of CoRR, 2013.
7. S. Nath, H. Yu, P.B. Gibbons, and S. Seshan. Subtleties in tolerating correlated failures in wide-area storage systems. In Proc. of NSDI, 2006.
8. M. Zhong, K. Shen, and J. Seiferas. Replication degree customization for high availability. In Proc. of EuroSys, Glasgow, 2008.
9. A. Haeberlen, A. Mislove, and P. Druschel. Glacier: Highly durable, decentralized storage despite massive correlated failures. In NSDI, 2005.
10. R. Chansler. Data availability and durability with the hadoop distributed file system. ;login: The USENIX Magazine, 37(1), 2012.
11. J. Dean. Evolution and future directions of large-scale storage and computation systems at google. In Proc. of SoCC, page 1, 2010.
12. A. Cidon, S. Rumble, R. Stutsman, S. Katti, J. Ousterhout, and M. Rosenblum. Copysets: Reducing the frequency of data loss in cloud storage. In Proc. of USENIX ATC, 2013.
13. D. Borthakur, J. Gray, J. Sarma, K. Muthukkaruppan, N. Spiegelberg, H. Kuang, K. Ranganathan, D. Molkov, A. Menon, S. Rash, R. Schmidt, and A. Aiyer. Apache hadoop goes realtime at facebook. In Proc. of SIGMOD, 2011.
14. H. Shen, J. Liu, K. Chen, J. Liu, and S. Moyer. SCPS: A socialaware distributed cyber-physical human-centric search engine. IEEE Transactions on Computers (TC), 64:518-532, 2015.
15. J. Liu, L. Yu, H. Shen, Y. He, and J. Hallstrom. Characterizing data deliverability of greedy routing in wireless sensor networks. In Proc. of SECON, Seattle, June 2015.
16. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris. Scarlett: Coping with skewed content popularity inmapreduce clusters. In Proc. of EuroSys, Salzburg, 2011.
17. E. Keller, J. Szefer, J. Rexford, and R. B. Lee. Nohype: Virtualized cloud infrastructure without the virtualization. In Proc. of ISCA, 2010.
18. J. Liu, H. Shen, and L. Chen. CORP: Cooperative opportunistic resource provisioning for short-lived jobs in cloud systems. In Proc. of IEEE CLUSTER, 2016.
19. N. Bonvin, T. Papaioannou, and K. Aberer. A self-organized, faulttolerant and scalable replication scheme for cloud storage. In Proc. of SoCC, 2010.
20. H. Yu and A. Vahdat. The costs and limits of availability for replicated services. In Proc. of SOSP, pages 29-42, 2001.
21. J. Liu, H. Shen, and H. Hu. Load-aware and congestion-free state management in network function virtualization. In Proc. of ICNC, 2017.
22. C. Tang and S. Dwarkadas. Hybrid global-local indexing for efficient peer-to-peer information retrieval. In Proc. of NSDI, 2004.
23. J. Liu, H. Shen, and X. Zhang. A survey of mobile crowdsensing techniques: A critical component for the internet of things. In Proc. of ICCCN workshop on Context QoS, 2016.
24. Y. Ma, T. Nandagopal, K. P. N. Puttaswamy, and S. Banerjee. An ensemble of replication and erasure codes for cloud file systems. In Proc. of INFOCOM, pages 1276-1284, 2013.
25. A. Thomson and D. Abadi. Calvinfs: Consistent wan replication and scalable metadata management for distributed file systems. In Proc. of FAST, Santa Clara.
26. H. Abu-Libdeh, R. Renesse, and Y. Vigfusson. Leveraging sharding in the design of scalable replication protocols. In Proc. of SoCC, 2013.
27. S. Y. Ko, I. Hoque, B. Cho, and I. Gupta. Making cloud intermediate data fault-tolerant. In Proc. of SoCC, Indianapolis, 2010.
28. Amazon S3. http://aws.amazon.com/s3 [accessed in Jun. 2016].
29. A. Cidon, R. Escriva, S. Katti, M. Rosenblum, and E. G. Sirer. Tiered replication: A cost-effective alternative to full cluster geo-replication. In Proc. of ATC, pages 31-43, 2015.
30. Y. Zhang, J. Yang, A. Memaripour, and S. Swanson. Mojim: A reliable and highly-available non-volatile memory system. In ASPLOS, 2015.
31. H. Yu, P. Gibbons, and S. Nath. Availability of multi-object operations. In Proc. of NSDI, pages 211-224, 2006.
32. S. Houghten, L. Thiel, J. Janssen, and C. Lam. There is no (46, 6, 1) block design∗. Journal of Combinatorial Designs, 9(1):60-71, 2001.
33. P. Kaski and P. Östergård. There exists no (15, 5, 4) rbibd. Journal of Combinatorial Designs, 9:227-232, 2001.
34. H. Shen, Z. Li, Y. Lin, and J. Li. Social Tube: P2P-assisted Video Sharing in Online Social Networks. TPDS, 2014.
35. F. André, A. Kermarrec, E. Merrer, N. Scouarnec, G. Straub, and A. Kempen. Archiving cold data in warehouses with clustered network coding. In Proc. of Euro Sys, 2014.
36. J. Liu, H. Shen, and L. Yu. Question quality analysis and prediction in community question answering services with coupled mutual reinforcement. TSC, PP(99):1-14, 2015.
37. H. Shen, Z. Li, J. Liu, and J. E. Grant. Knowledge sharing in the online social network of yahoo! answers and its implications. IEEE Transactions on Computers (TC), 64(6):1715-1728, June 2015.
38. J. Kubiatowicz, D. Bindel, Y. Chen, P. Eaton, D. Geels, R. Gummadi, S. Rhea, H. Weatherspoon, W. Weimer, C. Wells, and B. Zhao. Oceanstore: An architecture for global-scale persistent storage. In Proc. of ASPLOS, 2000.
39. M. Wittie, V. Pejovic, L. Deek, K. Almeroth, and B. Zhao. Exploiting locality of interest in online social networks. In Proc. of CoNEXT, 2010.
40. M. Bazaraa, H. Sherali, and C. Shetty. Nonlinear programming: Theory and algorithms. Wiley Interscience, 2006.
41. IPOPT. https://projects.coin-or.org/Ipopt [accessed in Jun. 2016].
42. A. Wächter and L.T. Biegler. On the implementation of an interior-point filter linesearch algorithm for large-scale nonlinear programming. Math. Program, 106(1), 2006.
43. Intelligent block placement policy to decrease probability of block loss. https://issues.apache.org/jira/browse/HDFS-1094.
44. N. Nakka, A. Choudhary, R. Klundt, M. Weston, and L. Ward. Detailed analysis of i/o traces for large scale applications. In HiPC, 2009.
45. J. Cook, A. Wolf, and B. Zorn. Partition selection policies in object database garbage collection. In Proc. of SIGMOD, New York, 1994.
46. D. Ford, F. Labelle, F. I. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan. Availability in globally distributed storage systems. In Proc. of OSDI, Berkeley, CA, USA, 2010.
47. J. Liu and H. Shen. Dependency-aware and resource-efficient scheduling for heterogeneous jobs in clouds. In Proc. of Cloud Com, 2016.
48. S3 FAQs. https://aws.amazon.com/s3/faqs [accessed in Jun. 2016].