A survey of large scale data management approaches in cloud environments

IEEE Communications Surveys and Tutorials

Volumn 13, Issue 3, 2011, Pages 311-336

  Sakr, Sherif ^a   Liu, Anna ^a   Batista, Daniel M ^b   Alomari, Mohammad ^c  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b UNIVERSITY OF CAMPINAS   (Brazil)

^c UNIVERSITY OF SYDNEY   (Australia)

Author keywords

Cloud Computing; Cloud Data Storage; MapReduce; NoSQL

Indexed keywords

CLOUD DATA STORAGE; COMPUTATIONAL POWER; COMPUTING ARCHITECTURE; COMPUTING INFRASTRUCTURES; DATA-INTENSIVE APPLICATION; DESIGN DECISIONS; END-USERS; FLOW OF DATA; FUTURE CHALLENGES; HARDWARE AND SOFTWARE; INDUSTRIAL COMMUNITIES; LARGE SCALE DATA; LARGE-SCALE DATA MANAGEMENT; MAP-REDUCE; NOSQL; PARADIGM SHIFTS; WEB TECHNOLOGIES;

DATA HANDLING; INDUSTRIAL RESEARCH; INFORMATION MANAGEMENT; SURVEYS; USER INTERFACES;

CLOUD COMPUTING;

EID: 80053256327     PISSN: None     EISSN: 1553877X     Source Type: Journal    
DOI: 10.1109/SURV.2011.032211.00087     Document Type: Review

References (80)

1. T. Hey, S. Tansly, and K. Tolle, editors. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, October 2009.
2. G. Bell, J. Gray, and A. Szalay. Petascale computational systems. IEEE Computer, 39(1):110-112, 2006. (Pubitemid 43191709)
3. Massimo Lamanna. High-Energy Physics Applications on the Grid. In Lizhe Wang and Wei Jie and Jinjun Chen, editor, Grid Computing: Infrastructure, Service, and Applications, pages 433-458. CRC Press, 2009.
4. Yehia El khatib and Christopher Edwards. A Survey-Based Study of Grid Traffic. In GridNets '07, pages 4:1-4:8, 2007.
5. Daniel M. Batista, Luciano J. Chaves, Nelson L. S. da Fonseca, and A. Ziviani. Performance Analysis of Available Bandwidth Estimation Tools for Grid Networks. Journal of Supercomputing, 53:103-121, July 2010.
6. Gartner. Gartner top ten disruptive technologies for 2008 to 2012. Emerging trends and technologies roadshow, 2008.
7. M. Armbrust, A. Fox, G. Rean, A. Joseph, R. Katz, A. Konwinski, L. Gunho, P. David, A. Rabkin, I. Stoica, and M. Zaharia. Above the clouds: A Berkeley View of Cloud Computing, Feb 2009.
8. L. Gonzalez, L. Merino, J. Caceres, and M. Lindner. A Break in the Clouds: Towards a Cloud Definition. Computer Communication Review, 39(1), 2009.
9. D. Plummer, T. Bittman, T. Austin, D. Cearley, and D. Smith. Cloud computing: Defining and describing an emerging phenomenon. Technical report, Gartner, June 2008.
10. J. Staten, S. Yates, F. Gillett, W. Saleh, and R. Dines. Is cloud computing ready for the enterprise? Technical report, Forrester Research, March 2008.
11. P. Mell and T. Grance. Definition of cloud computing. Technical report, National Institute of Standard and Technology (NIST), July 2009.
12. D. Parkhill. The challenge of the computer utility. Addison-Wesley, 1966.
13. Q. Zhang, L. Cheng, and R. Boutaba. Cloud computing: State-of-theart and research challenges. J. Internet Services and Applications, 1(1), 2010.
14. Kernal based virtual machine. www.linux-kvm.org.
15. Vmware esx server. www.vmware.com.
16. Xensource inc. www.xensource.com.
17. Ibrahim W. Habib, Qiang Song, Zhaoming Li, and Nageswara S. V. Rao. Deployment of the GMPLS control plane for grid applications in experimental high-performance networks. IEEE Commun. Mag., 44(3):65-73, 2006.
18. Thomas Lehman, Jerry Sobieski, and Bijan Jabbari. DRAGON: A framework for service provisioning in heterogeneous grid networks. IEEE Commun. Mag., 44(3):84-90, 2006.
19. Wei Guo, Weiqiang Sun, Yaohui Jin, Weisheng Hu, and Chunming Qiao. Demonstration of Joint Resource Scheduling in an Optical Network Integrated Computing Environment. IEE Communications Magazine, 48(5):76-83, 2010.
20. I. Foster, Y. Zhao, I. Raicu, and S. Lu. Cloud Computing and Grid Computing 360-degree Compared. CoRR, abs/0901.0131, 2009.
21. C. Cardenas, M. Gagnaire, V. Lopez, and J. Aracil. Performance Evaluation of the Flow-Aware Networking (FAN) Architecture under Grid Environment. In IEEE/IFIP NOMS 2008, pages 481-487, Apr 2008.
22. Daniel M. Batista, Nelson L. S. da Fonseca, Flavio K. Miyazawa, and Fabrizio Granelli. Self-Adjustment of Resource Allocation for Grid Applications. Computer Networks, 52:1762-1781, June 2008.
23. Daniel M. Batista and Nelson L. S. da Fonseca. A Survey of Self-Adaptive Grids. IEEE Commun. Mag., 48:94-100, Jul 2010.
24. D. Agrawal, A. El Abbadi, F. Emekci, and A. Metwally. Database Management as a Service: Challenges and Opportunities. In ICDE, 1709-1716, 2009.
25. A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi, A. Rasin, and A. Silberschatz. Hadoopdb: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads. PVLDB, 2(1):922-933, 2009.
26. B. Cooper, E. Baldeschwieler, R. Fonseca, J. Kistler, P. Narayan, C. Neerdaels, T. Negrin, R. Ramakrishnan, A. Silberstein, U. Srivastava, and R. Stata. Building a Cloud for Yahoo! IEEE Data Eng. Bull., 32(1):36-43, 2009.
27. L. Youseff, M. Butrico, and D. Da Silva. Towards a unified Ontology of Cloud Computing. In GCE, 2008.
28. J. Hofstader. Communications as a service. http://msdn.microsoft.com/en- us/library/bb896003.aspx.
29. Connected services framework. http://www.microsoft.com/serviceprovid ers/solutions/connectedservicesframework.mspx
30. D. Abadi. Data management in the cloud: Limitations and opportunities. IEEE Data Eng. Bull., 32(1):3-12, 2009.
31. M. Stonebraker. The case for shared nothing. IEEE Database Eng. Bull., 9(1):4-9, 1986.
32. F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber. Bigtable: A distributed storage system for structured data. ACM Trans. Comput. Syst., 26(2), 2008.
33. S. Ghemawat, H. Gobioff, and S. Leung. The google file system. In SOSP, pages 29-43, 2003. (Pubitemid 40929686)
34. M. Burrows. The chubby lock service for loosely-coupled distributed systems. In OSD, pages 335-350, 2006.
35. W. Vogels. Eventually Consistent. Commun. ACM, 52(1):40-44, 2009.
36. B. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon, H. Jacobsen, N. Puz, D. Weaver, and R. Yerneni. Pnuts: Yahoo!'s hosted data serving platform. PVLDB, 1(2):1277-1288, 2008.
37. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon's highly available key-value store. In SOSP, pages 205-220, 2007. (Pubitemid 351429377)
38. D. Karger, E. Lehman, F. Leighton, R. Panigrahy, M. Levine, and D. Lewin. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. In STOC, pages 654-663, 1997.
39. M. Brantner, D. Florescu, D. Graf, D. Kossmann, and T. Kraska. Building a Database on S3. In SIGMOD, pages 251-264, 2008.
40. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys, pages 59-72, 2007. (Pubitemid 47281575)
41. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U . Erlingsson, P. Gunda, and J. Currey. Dryadlinq: A system for general-purpose distributed dataparallel computing using a high-level language. In OSDI, pages 1-14, 2008.
42. A. Lakshman and P. Malik. Cassandra: Structured storage system on a p2p network. In PODC, page 5, 2009.
43. A. Li, X. Yang, S. Kandula, and M. Zhang. CloudCmp: Shopping for a Cloud Made Easy. In HotCloud, USENIX Workshop, 2010.
44. Asfandyar Qureshi, Rick Weber, Hari Balakrishnan, John Guttag, and Bruce Maggs. Cutting the Electric Bill for Internet-Scale Systems. SIGCOMM Comput. Commun. Rev., 39:123-134, August 2009.
45. E. Brewer. Towards robust distributed systems (abstract). In PODC, page 7, 2000.
46. S. Gilbert and N. Lynch. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News, 33(2):51-59, 2002.
47. A. Tanenbaum and M. Steen, editors. Distributed Systems: Principles and Paradigms. Prentice Hall, 2002.
48. E. Tam R. Ramakrishnan B. Cooper, A. Silberstein and R. Sears. Benchmarking Cloud Serving Systems with YCSB. In ACM SoCC, 2010.
49. T. Kraska, M. Hentschel, G. Alonso, and D. Kossmann. Consistency rationing in the cloud: Pay only when it matters. PVLDB, 2(1):253-264, 2009.
50. K. Keeton, C. Morrey, N. Soules, and A. Veitch. LazyBase: freshness vs. performance in information management. SIGOPS Oper. Syst. Rev., 44(1):15-19, 2010.
51. D. Florescu and D. Kossmann. Rethinking cost and performance of database systems. SIGMOD Record, 38(1):43-48, 2009.
52. J. Gray. Distributed computing economics. Microsoft Research Technical Report MSRTR-2003-24, Microsoft Research, 2003.
53. D. Kossmann, T. Kraska, and S. Loesing. An evaluation of alternative architectures for transaction processing in the cloud. In SIGMOD, 2010.
54. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSDI, pages 137-150, 2004.
55. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Commun. ACM, 51(1):107-113, 2008.
56. H. Yang, A. Dasdan, R. Hsiao, and D. Parker. Map-Reduce-Merge: Simplified relational data processing on large clusters. In SIGMOD, pages 1029-1040, 2007. (Pubitemid 47630879)
57. J. Dean and S. Ghemawat. Mapreduce: A flexible data processing tool. Commun. ACM, 53(1):72-77, 2010.
58. S. Blanas, J. Patel, V. Ercegovac, J. Rao, E. Shekita, and Y. Tian. A comparison of join algorithms for log processing in mapreduce. In SIGMOD, pages 975-986, 2010.
59. H. Yang and D. Parker. Traverse: Simplified indexing on large MapReduce-Merge clusters. In DASFAA, pages 308-322, 2009.
60. P. Alvaro J. Hellerstein K. Elmeleegy T. Condie, N. Conway and R. Sears. Mapreduce Online. In NSDI, 2010.
61. Y. Gu and R. Grossman. Lessons learned from a year's worth of benchmarks of large data clouds. In SC-MTAGS, 2009.
62. C. Wang, J. Wang, X. Lin, W. Wang, H. Wang, H. Li, W. Tian, J. Xu, and R. Li. MapDupReducer: Detecting near duplicates over massive datasets. In SIGMOD, pages 1119-1122, 2010.
63. R. Chen, X. Weng, B. He, and M. Yang. Large graph processing in the cloud. In SIGMOD, pages 1123-1126, 2010.
64. G. Malewicz, M. Austern, A. Bik, J. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In SIGMOD, pages 135-146, 2010.
65. S. Das, Y. Sismanis, K. Beyer, R. Gemulla, P. Haas, and J. McPherson. Ricardo: Integrating R and Hadoop. In SIGMOD, pages 987-998, 2010.
66. A. Gates, O. Natkovich, S. Chopra, P. Kamath, S. Narayanam, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava. Building a highlevel dataflow system on top of mapreduce: The pig experience. PVLDB, 2(2):1414-1425, 2009.
67. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In SIGMOD, pages 1099-1110, 2008.
68. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming, 13(4):277-298, 2005. (Pubitemid 43063583)
69. R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. Scope: Easy and efficient parallel processing of massive data sets. PVLDB, 1(2):1265-1276, 2008.
70. J. Zhou, P. Larson, and R. Chaiken. Incorporating partitioning and parallel plans into the scope optimizer. In ICDE, pages 1060-1071, 2010.
71. D. Murray and S. Hand. Scripting the cloud with Skywriting. In HotCloud, USENIX Workshop, 2010.
72. E. Friedman, P. Pawlowski, and J. Cieslewicz. SQL/MapReduce: A practical approach to self-describing, polymorphic, and parallelizable user-defined functions. PVLDB, 2(2):1402-1413, 2009.
73. A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive-a warehousing solution over a mapreduce framework. PVLDB, 2(2):1626-1629, 2009.
74. A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy. Hive-a petabyte scale data warehouse using hadoop. In ICDE, pages 996-1005, 2010.
75. A. Pavlo, E. Paulson, A. Rasin, D. Abadi, D. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In SIGMOD, pages 165-178, 2009.
76. M. Stonebraker, D. Abadi, D. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin. MapReduce and Parallel DBMSs: friends or foes? Commun. ACM, 53(1):64-71, 2010.
77. A. Abouzeid, K. Bajda-Pawlikowski, J. Huang, D. Abadi, and A. Silberschatz. HadoopDB in action: Building real world applications. In SIGMOD, 2010.
78. Y. Xu, P. Kostamaa, and L. Gao. Integrating Hadoop and Parallel DBMS. In SIGMOD, pages 969-974, 2010.
79. A. Avetisyan, R. Campbell, I. Gupta, M. Heath, S. Ko, G. Ganger, M. Kozuch, D. O'Hallaron, M. Kunze, T. Kwan, K. Lai, M. Lyons, D. Milojicic, H. Yan Lee, Y. Soh, N. Ming, J. Luke, and H. Namgoong. Open Cirrus: A Global Cloud Computing Testbed. IEEE Computer, 43(4):35-43, 2010.
80. R. Grossman, Y. Gu, M. Sabala, C. Bennet, J. Seidman, and J. Mambretti. The Open Cloud Testbed: A Wide Area Testbed for Cloud Computing Utilizing High Performance Network Services. CoRR, abs/0907.4810, 2009.