Cloud storage reliability for Big Data applications: A state of the art survey

Journal of Network and Computer Applications

Volumn 97, Issue , 2017, Pages 35-47

  Nachiappan, Rekha ^a   Javadi, Bahman ^a   Calheiros, Rodrigo N ^a   Matawie, Kenan M ^a  

^a UNIVERSITY OF WESTERN SYDNEY   (Australia)

Author keywords

Big Data applications; Cloud storage; Data reliability; Erasure coding; Fault tolerance; Replication

Indexed keywords

BANDWIDTH; DIGITAL STORAGE; ECONOMIC AND SOCIAL EFFECTS; ENERGY UTILIZATION; FAULT TOLERANCE; RELIABILITY; SURVEYS;

BIG DATA APPLICATIONS; CLOUD STORAGES; DATA RELIABILITY; ERASURE CODING; REPLICATION;

BIG DATA;

EID: 85029597099     PISSN: 10848045     EISSN: 10958592     Source Type: Journal    
DOI: 10.1016/j.jnca.2017.08.011     Document Type: Review

References (85)

1. Abu-Libdeh, H., Princehouse, L., Weatherspoon, H., 2010. Racs: a case for cloud storage diversity. In: Proceedings of the 1st ACM symposium on Cloud computing. ACM, pp. 229–240.
2. Agrawal, B., Wiktorski, T., Rong, C., 2015. Analyzing and predicting failure in hadoop clusters using distributed hidden markov model. In: Proceedings of International Conference on Cloud Computing and Big Data in Asia. Springer, pp. 232–246.
3. Araujo, J., Giroire, F., Monteiro, J., Hybrid approaches for distributed storage systems. Data Manag. Grid Peer-to-Peer Syst., 2011, 1–12.
4. AWS, 2016. Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US East Region. URL 〈 https://aws.amazon.com/message/5467D2〉.
5. Baesens, B., Analytics in a Big Data World: The Essential Guide to Data Science and Its Applications. 2014, John Wiley&Sons.
6. Beaver, D., Kumar, S., Li, H.C., Sobel, J., Vajgel, P. et al., 2010. Finding a needle in haystack: Facebook's photo storage. In: Proceedings of OSDI., vol. 10. pp. 1–8.
7. Blaum, M., Brady, J., Bruck, J., Menon, J., Evenodd: an efficient scheme for tolerating double disk failures in raid architectures. IEEE Trans. Comput. 44:2 (1995), 192–202.
8. Bonvin, N., Papaioannou, T.G., Aberer, K., 2009. Dynamic cost-efficient replication in data clouds. In: Proceedings of the 1st Workshop on Automated Control for Datacenters and Clouds. ACM, pp. 49–56.
9. Boru, D., Kliazovich, D., Granelli, F., Bouvry, P., Zomaya, A.Y., Energy-efficient data replication in cloud computing datacenters. Clust. Comput. 18:1 (2015), 385–402.
10. Brewer, E., Ying, L., Greenfield, L., Cypher, R., Tso, T., 2016. Disks Fordata Centers. White Paper for FAST 1(1), p. 4.
11. Butt, A.R., Bhattacharjee, P., Wang, G., Gniady, C., Exploring trade-offs between energy savings and reliability in storage systems. Green Comput. Book: Tackling Energy Effic. Large Scale, 2014, 149.
12. Cadambe, V.R., Huang, C., Jafar, S.A., Li, J., 2011. Optimal repair of mds codes in distributed storage via subspace interference alignment. arXiv preprint arXiv:1106.1250.
13. Cadambe, V.R., Jafar, S.A., Maleki, H., 2010. Distributed data storage with minimum storage regenerating codes-exact and functional repair are asymptotically equally efficient. arXiv preprint arXiv:1004.4299.
14. Chen, C.P., Zhang, C.-Y., Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275 (2014), 314–347.
15. Chen, Y., Wang, Y., On the non-existence of minimum storage regenerating codes with repair-by-transfer property. IEEE Commun. Lett. 19:12 (2015), 2070–2073.
16. Cidon, A., Rumble, S.M., Stutsman, R., Katti, S., Ousterhout, J.K., Rosenblum, M., 2013. Copysets: Reducing the frequency of data loss in cloud storage. In: Proceedings of Usenix Annual Technical Conference. pp. 37–48.
17. Cook, J.D., Primme, R., de Kwant, A., Compare cost and performance of replication and erasure coding. Hitachi Rev., 63, 2014, 304.
18. Dai, D., Chen, Y., Kimpe, D., Ross, R., 2014. Provenance-based object storage prediction scheme for scientific big data applications. In: 2014 IEEE International Conference on Big Data (Big Data). IEEE, pp. 271–280.
19. Dimakis, A.G., Godfrey, P.B., Wu, Y., Wainwright, M.J., Ramchandran, K., Network coding for distributed storage systems. IEEE Trans. Inf. Theory 56:9 (2010), 4539–4551.
20. Ford, D., Labelle, F., Popovici, F.I., Stokely, M., Truong, V.-A., Barroso, L., Grimes, C., Quinlan, S., 2010. Availability in globally distributed storage systems. In: Proceedings of Osdi., vol. 10. pp. 1–7.
21. Gantz, J., Reinsel, D., The digital universe in 2020: big data, bigger digital shadows, and biggest growth in the far east. IDC iView: IDC Anal. Future 2007:2012 (2012), 1–16.
22. Greenan, K.M., Long, D.D., Miller, E.L., Schwarz, T.J., Wylie, J.J., 2008. A spin-up saved is energy earned: achieving power-efficient, erasure-coded storage. In: Proceedings of HotDep.
23. Groenfeldt, T., 2012. Big Databig Money Says it is a Paradigm Buster.
24. Gunawi, H. S., Do, T., Joshi, P., Hellerstein, J.M., Arpaci-Dusseau, A.C., Arpaci-Dusseau, R.H., Sen, K., 2010. Towards automatically checking thousands of failures with micro-specifications. In: Proceedings of HotDep.
25. Harnik, D., Naor, D., Segall, I., 2009. Low power mode in cloud storage systems. In: Proceedings of IEEE International Symposium on Parallel&Distributed Processing, 2009. IPDPS 2009. IEEE, pp. 1–8.
26. Hu, Y., Xu, Y., Wang, X., Zhan, C., Li, P., Cooperative recovery of distributed storage systems from multiple losses with network coding. IEEE J. Sel. Areas Commun., 28, 2010, 2.
27. Hu, Y., Lee, P.P., Shum, K.W., 2013. Analysis and construction of functional regenerating codes with uncoded repair for distributed storage systems. In: INFOCOM, 2013 Proceedings IEEE. IEEE, pp. 2355–2363.
28. Huang, C., Xu, L., Star: an efficient coding scheme for correcting triple storage node failures. IEEE Trans. Comput. 57:7 (2008), 889–901.
29. Huang, C., Chen, M., Li, J., Pyramid codes: flexible schemes to trade space for access efficiency in reliable data storage systems. ACM Trans. Storage (TOS), 9(1), 2013, 3.
30. Huang, C., Simitci, H., Xu, Y., Ogus, A., Calder, B., Gopalan, P., Li, J., Yekhanin, S., et al., 2012. Erasure coding in windows azure storage. In: Proceedings of Usenix Annual Technical Conference. Boston, MA, pp. 15–26.
31. Hughes, G.F., Murray, J.F., Kreutz-Delgado, K., Elkan, C., Improved disk-drive failure warnings. IEEE Trans. Reliab. 51:3 (2002), 350–357.
32. Hussein, M.-K., Mousa, M.-H., A light-weight data replication for cloud data centers environment. Int. J. Eng. Innov. Technol. 1:6 (2012), 169–175.
33. Jin, C., Jiang, H., Feng, D., Tian, L., 2009. P-code: a new raid-6 code with optimal properties. In: Proceedings of the 23rd International Conference on Supercomputing. ACM, pp. 360–369.
34. Kermarrec, A.-M., Le Scouarnec, N., Straub, G., 2011. Repairing multiple failures with coordinated and adaptive regenerating codes. In: Proceedings of the 2011 International Symposium on Network Coding (NetCod). IEEE, pp. 1–6.
35. Khan, O., Burns, R.C., Plank, J.S., Pierce, W., Huang, C., 2012. Rethinking erasure codes for cloud file systems: minimizing i/o for recovery and degraded reads. In: Proceedings of FAST. p. 20.
36. Kumar, A., Tandon, R., Clancy, T. C., 2014. On the latency and energy efficiency of erasure-coded cloud storage systems. arXiv preprint arXiv:1405.2833.
37. Kune, R., Konugurthi, P.K., Agarwal, A., Chillarige, R.R., Buyya, R., The anatomy of big data computing. Softw.: Pract. Exp. 46:1 (2016), 79–105.
38. Le Scouarnec, N., 2012. Exact scalar minimum storage coordinated regenerating codes. In: 2012 IEEE International Symposium on Information Theory Proceedings (ISIT). IEEE, pp. 1197–1201.
39. Li, J., Li, B., Erasure coding for cloud storage systems: a survey. Tsinghua Sci. Technol. 18:3 (2013), 259–272.
40. Li, P., Li, J., Stones, R.J., Wang, G., Li, Z., Liu, X., 2016. Procode: a proactive erasure coding scheme for cloud storage systems. In: 2016 IEEE Proceedings of the 35th Symposium on Reliable Distributed Systems (SRDS). IEEE, pp. 219–228.
41. Li, W., Yang, Y., Yuan, D., 2011. A novel cost-effective dynamic data replication strategy for reliability in cloud data centres. In: 2011 IEEE Proceedings of the Ninth International Conference on Dependable, Autonomic and Secure Computing (DASC). IEEE, pp. 496–502.
42. Lin, S.-J., Chung, W.-H., Novel repair-by-transfer codes and systematic exact-mbr codes with lower complexities and smaller field sizes. IEEE Trans. Parallel Distrib. Syst. 25:12 (2014), 3232–3241.
43. Lin, W., Chiu, D.M., Lee, Y., 2004. Erasure code replication revisited. In: Proceedings of the Fourth International Conference on Peer-to-Peer Computing, 2004. IEEE, pp. 90–97.
44. Liu, J., Shen, H., 2016. A low-cost multi-failure resilient replication scheme for high data availability in cloud storage. In: 2016 IEEE Proceedings of the 23rd International Conference on High Performance Computing (HiPC). IEEE, pp. 242–251.
45. Long, S.-Q., Zhao, Y.-L., Chen, W., Morm: a multi-objective optimized replication management strategy for cloud storage cluster. J. Syst. Archit. 60:2 (2014), 234–244.
46. Ma, A., Traylor, R., Douglis, F., Chamness, M., Lu, G., Sawyer, D., Chandra, S., Hsu, W., RAIDshield: characterizing, monitoring, and proactively protecting against disk failures. ACM Trans. Storage (TOS), 11(4), 2015, 17.
47. Ma, Y., Nandagopal, T., Puttaswamy, K.P., Banerjee, S., 2013. An ensemble of replication and erasure codes for cloud file systems. In: INFOCOM, 2013 Proceedings IEEE. IEEE, pp. 1276–1284.
48. Mesnier, M., Ganger, G.R., Riedel, E., Object-based storage. IEEE Commun. Mag. 41:8 (2003), 84–90.
49. Muralidhar, S., Lloyd, W., Roy, S., Hill, C., Lin, E., Liu, W., Pan, S., Shankar, S., Sivakumar, V., Tang, L., et al., 2014. f4: Facebooks warm blob storage system. In: Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation. USENIX Association, pp. 383–398.
50. O'Reilly, J., Network Storage: Tools and Technologies for Storing Your Companyas Data. 2016, Morgan Kaufmann, Volanto, USA.
51. Papailiopoulos, D.S., Dimakis, A.G., 2011. Distributed storage codes through hadamard designs. In: 2011 IEEE International Symposium on Information Theory Proceedings (ISIT). IEEE, pp. 1230–1234.
52. Pei, X., Wang, Y., Ma, X., Fu, Y., Xu, F., 2015. Cooperative repair based on tree structure for multiple failures in distributed storage systems with regenerating codes. In: Proceedings of the 12th ACM International Conference on Computing Frontiers. ACM, p. 14.
53. Pinheiro, E., Bianchini, R., Dubnicki, C., 2006. Exploiting redundancy to conserve energy in storage systems. In: ACM SIGMETRICS Performance Evaluation Review. vol. 34. ACM, pp. 15–26.
54. Pinheiro, E., Weber, W.-D., Barroso, L.A., 2007. Failure trends in a large disk drive population. In: Proceedings of FAST., vol. 7. pp. 17–23.
55. Plank, J.S., The raid-6 liber8tion code. Int. J. High. Perform. Comput. Appl. 23:3 (2009), 242–251.
56. Plank, J.S., Erasure codes for storage systems: a brief primer. Usenix Mag. 38:6 (2013), 44–50.
57. Plank, J.S., Blaum, M., Hafner, J.L., 2013. Sd codes: erasure codes designed for how storage systems really fail. In: Proceedings of FAST. pp. 95–104.
58. Qu, Y., Xiong, N., 2012. Rfh: A resilient, fault-tolerant and high-efficient replication algorithm for distributed cloud storage. In: 2012 Proceedings of the 41st International Conference on Parallel Processing (ICPP). IEEE, pp. 520–529.
59. Rajasekharan, 2014. Data Reliability in Highly Fault-tolerant Cloud Systems. URL 〈 https://pdfs.semanticscholar.org/abe7/7e70864a0d914365ed755cac5ce1abc3b8b0.pdf/〉.
60. Rashmi, K.V., Shah, N.B., Kumar, P.V., Optimal exact-regenerating codes for distributed storage at the msr and mbr points via a product-matrix construction. IEEE Trans. Inf. Theory 57:8 (2011), 5227–5239.
61. Rashmi, K., Shah, N.B., Gu, D., Kuang, H., Borthakur, D., Ramchandran, K., 2013. A solution to the network challenges of data recovery in erasure-coded distributed storage systems: a study on the facebook warehouse cluster. In: 2012 Proceedings of HotStorage.
62. Rashmi, K., Shah, N.B., Gu, D., Kuang, H., Borthakur, D., Ramchandran, K., 2014. A hitchhiker's guide to fast and efficient data reconstruction in erasure-coded data centers. In: Proceedings of ACM SIGCOMM Computer Communication Review. vol. 44. ACM, pp. 331–342.
63. Rashmi, K., Shah, N.B., Kumar, P.V., Ramchandran, K., 2009. Explicit construction of optimal exact regenerating codes for distributed storage. In: Proceedings of the 47th Annual Allerton Conference on Communication, Control, and Computing, 2009. Allerton 2009. IEEE, pp. 1243–1249.
64. Rodrigues, R., Liskov, B., High availability in dhts: erasure coding vs. replication: erasure coding vs. replication. Peer Peer Syst. IV, 2005, 226–239.
65. Sathiamoorthy, M., Asteris, M., Papailiopoulos, D., Dimakis, A.G., Vadali, R., Chen, S., Borthakur, D., 2013. Xoring elephants: novel erasure codes for big data. In: Proceedings of the VLDB Endowment. vol. 6. VLDB Endowment, pp. 325–336.
66. Shah, N.B., Rashmi, K.V., Kumar, P.V., Ramchandran, K., Distributed storage codes with repair-by-transfer and nonachievability of interior points on the storage-bandwidth tradeoff. IEEE Trans. Inf. Theory 58:3 (2012), 1837–1852.
67. Sharma, Y., Javadi, B., Si, W., Sun, D., Reliability and energy efficiency in cloud computing systems: survey and taxonomy. J. Netw. Comput. Appl. 74 (2016), 66–85.
68. Shum, K.W., 2011. Cooperative regenerating codes for distributed storage systems. In: Proceedings of the 2011 IEEE International Conference on Communications (ICC). IEEE, pp. 1–5.
69. Shum, K.W., Hu, Y., 2011. Exact minimum-repair-bandwidth cooperative regenerating codes for distributed storage systems. In: Proceedings of the 2011 IEEE International Symposium on Information Theory Proceedings (ISIT). IEEE, pp. 1442–1446.
70. Shum, K.W., Hu, Y., 2012. Functional-repair-by-transfer regenerating codes. In: Proceedings of the 2012 IEEE International Symposium on Information Theory Proceedings (ISIT). IEEE, pp. 1192–1196.
71. Suh, C., Ramchandran, K., Exact-repair mds code construction using interference alignment. IEEE Trans. Inf. Theory 57:3 (2011), 1425–1442.
72. Suh, C., Ramchandran, K., 2010. On the existence of optimal exact-repair mds codes for distributed storage. arXiv preprint arXiv:1004.4663.
73. Sun, D.-W., Chang, G.-R., Gao, S., Jin, L.-Z., Wang, X.-W., Modeling a dynamic data replication strategy to increase system availability in cloud computing environments. J. Comput. Sci. Technol. 27:2 (2012), 256–272.
74. Tamo, I., Wang, Z., Bruck, J., 2011. Mds array codes with optimal rebuilding. In: Proceedings of the 2011 IEEE International Symposium on Information Theory Proceedings (ISIT). IEEE, pp. 1240–1244.
75. Vijayakumari, R., Kirankumar, R., Rao, K.G., 2014. Comparative Analysis of Google File System and Hadoop Distributed File System.
76. Vins, V.A., Umamageswari, S., Saranya, P., 2014. A Survey on Regenerating Codes.
77. Vishwanath, K.V., Nagappan, N., 2010. Characterizing cloud computing hardware reliability. In: Proceedings of the 1st ACM Symposium on Cloud computing. ACM, pp. 193–204.
78. Wang, A., Zhang, Z., 2013. Exact cooperative regenerating codes with minimum-repair-bandwidth for distributed storage. In: INFOCOM, 2013 Proceedings IEEE. IEEE, pp. 400–404.
79. Wang, P., Dean, D.J., Gu, X., 2015. Understanding real world data corruptions in cloud systems. In: Proceedings of the 2015 IEEE International Conference on Cloud Engineering (IC2E). IEEE, pp. 116–125.
80. Weatherspoon, H., Kubiatowicz, J., et al., 2002. Erasure Coding vs. Replication: A Quantitative Comparison. In: IPTPS., vol. 1. Springer, pp. 328–338.
81. Wu, Y., A construction of systematic mds codes with minimum repair bandwidth. IEEE Trans. Inf. Theory 57:6 (2011), 3738–3741.
82. Xiang, L., Xu, Y., Lui, J., Chang, Q., Optimal recovery of single disk failure in rdp code storage systems. ACM SIGMETRICS Perform. Eval. Rev. 38:1 (2010), 119–130.
83. Yixian, Y., Peiod distribution for blaum-roth code. J. Beijing Univ. Posts Telecommun., 1994, 3.
84. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I., Spark: cluster computing with working sets. HotCloud, 10(10–10), 2010, 95.
85. Zeng, W., Zhao, Y., Ou, K., Song, W., 2009. Research on cloud storage architecture and key technologies. In: Proceedings of the 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human. ACM, pp. 1044–1048.