Big data processing: Big challenges

Journal of Interconnection Networks

Volumn 13, Issue 3-4, 2012, Pages

  Ji, Changqing ^a,b   Li, Yu ^c   Qiu, Wenming ^c   Jin, Yingwei ^c   Xu, Yujie ^a   Awada, Uchechukwu ^c   Li, Keqiu ^c   Qu, Wenyu ^a  

^a DALIAN MARITIME UNIVERSITY   (China)

^b DALIAN UNIVERSITY   (China)

^c DALIAN UNIVERSITY OF TECHNOLOGY   (China)

Author keywords

Big data; cloud computing; data management; distributed processing

Indexed keywords

CLOUD COMPUTING; COMPUTER SYSTEMS; DISTRIBUTED DATABASE SYSTEMS; INFORMATION MANAGEMENT; LOCATION BASED SERVICES; SENSOR NETWORKS;

BIG DATUM; CLOUD COMPUTING ENVIRONMENTS; CLOUD DATA MANAGEMENTS; DISTRIBUTED APPLICATIONS; DISTRIBUTED FILE SYSTEMS; DISTRIBUTED PROCESSING; ISSUES AND CHALLENGES; OPTIMIZATION STRATEGY;

DATA PROCESSING;

EID: 84876589704     PISSN: 02192659     EISSN: 17936713     Source Type: Journal    
DOI: 10.1142/S0219265912500090     Document Type: Article

References (64)

1. (2008) Big data: Science in the petabyte era. Nature 455 (7209): 1.
2. Douglas and Laney (2008) The importance of 'big data': A definition.
3. Ji, C., Li, Y., Qiu, W., Awada, U., and Li, K. (2012) Big data processing in cloud computing environments. Pervasive Systems, Algorithms and Networks (ISPAN), 2012 12th International Symposium on, pp. 17-23, IEEE.
4. Kossmann, D., Kraska, T., and Loesing, S. (2010) An evaluation of alternative archi-tectures for transaction processing in the cloud. Proceedings of the 2010 international conference on Management of data, pp. 579-590, ACM.
5. Horey, J., Begoli, E., Gunasekaran, R., Lim, S., and Nutaro, J. (2012) Big data platforms as a service: Challenges and approach. Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing, pp. 16-16, USENIX Association.
6. Ghemawat, S., Gobioff, H., and Leung, S. (2003) The google file system. ACM SIGOPS Operating Systems Review, vol. 37, pp. 29-43, ACM.
7. Dean, J. and Ghemawat, S. (2008) Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51, 107-113.
8. Borthakur, D. (2007) The hadoop distributed file system: Architecture and design. Hadoop Project Website, 11.
9. Rabkin, A. and Katz, R. (2010) Chukwa: A system for reliable large-scale log collection. USENIX Conference on Large Installation System Administration, pp. 1-15.
10. Sakr, S., Liu, A., Batista, D., and Alomari, M. (2011) A survey of large scale data management approaches in cloud environments. Communications Surveys & Tutorials, IEEE, 13, 311-336.
11. Cao, Y., Chen, C., Guo, F., Jiang, D., Lin, Y., Ooi, B., Vo, H.,Wu, S., and Xu, Q. (2011) Es2: A cloud data storage system for supporting both oltp and olap. Data Engineering (ICDE), 2011 IEEE 27th International Conference on, pp. 291-302, IEEE.
12. Chang, F., Dean, J., Ghemawat, S., Hsieh, W., Wallach, D., Burrows, M., Chandra, T., Fikes, A., and Gruber, R. (2006) Bigtable: A distributed structured data storage system. 7th OSDI, pp. 305-314.
13. Cooper, B., Ramakrishnan, R., Srivastava, U., Silberstein, A., Bohannon, P., Jacobsen, H., Puz, N., Weaver, D., and Yerneni, R. (2008) Pnuts: Yahoo!'s hosted data serving platform. Proceedings of the VLDB Endowment, 1, 1277-1288.
14. DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., and Vogels, W. (2007) Dynamo: Amazon's highly available key-value store. ACM SIGOPS Operating Systems Review, vol. 41, pp. 205-220, ACM. (Pubitemid 351429377)
15. Lin, Y., Agrawal, D., Chen, C., Ooi, B., and Wu, S. (2011) Llama: Leveraging columnar storage for scalable join processing in the mapreduce framework. Proceedings of the 2011 international conference on Management of data, pp. 961-972, ACM.
16. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., and Zagorodnov, D. (2009) The eucalyptus open-source cloud-computing system. Cluster Computing and the Grid, 2009. CCGRID'09. 9th IEEE/ACM International Symposium on, pp. 124-131, IEEE.
17. Sempolinski, P. and Thain, D. (2010) A comparison and critique of eucalyptus, open-nebula and nimbus. Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on, pp. 417-426, Ieee.
18. DeWitt, D. and Stonebraker, M. (2008) Mapreduce: A major step backwards. The Database Column, 1.
19. Stonebraker, M., Abadi, D., DeWitt, D., Madden, S., Paulson, E., Pavlo, A., and Rasin, A. (2010) Mapreduce and parallel dbmss: friends or foes. Communications of the ACM, 53, 64-71.
20. Abouzeid, A., Bajda-Pawlikowski, K., Abadi, D., Silberschatz, A., and Rasin, A. (2009) Hadoopdb: An architectural hybrid of mapreduce and dbms technologies for analytical workloads. Proceedings of the VLDB Endowment, 2, 922-933.
21. Xu, Y., Kostamaa, P., and Gao, L. (2010) Integrating hadoop and parallel dbms. Pro-ceedings of the 2010 international conference on Management of data, pp. 969-974, ACM.
22. Dittrich, J., Quian'e-Ruiz, J., Jindal, A., Kargin, Y., Setty, V., and Schad, J. (2010) Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). Proceedings of the VLDB Endowment, 3, 515-529.
23. Das, S., Sismanis, Y., Beyer, K., Gemulla, R., Haas, P., and McPherson, J. (2010) Ricardo: Integrating r and hadoop. Proceedings of the 2010 international conference on Management of data, pp. 987-998, ACM.
24. Stupar, A., Michel, S., and Schenkel, R. (2010) Rankreduce- processing k-nearest neigh-bor queries on top of mapreduce. Proceedings of the 8th Workshop on Large-Scale Dis-tributed Systems for Information Retrieval, pp. 13-18.
25. Ferreira Cordeiro, R., Traina Junior, C., Machado Traina, A., L'opez, J., Kang, U., and Faloutsos, C. (2011) Clustering very large multi-dimensional datasets with mapreduce. Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 690-698, ACM.
26. Wang, C., Wang, J., Lin, X., Wang, W., Wang, H., Li, H., Tian, W., Xu, J., and Li, R. (2010) Mapdupreducer: Detecting near duplicates over massive datasets. Proceedings of the 2010 international conference on Management of data, pp. 1119-1122, ACM.
27. Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., and Kozyrakis, C. (2007) Evaluating mapreduce for multi-core and multiprocessor systems. High Performance Computer Architecture, 2007. HPCA 2007. IEEE 13th International Symposium on, pp. 13-24, IEEE. (Pubitemid 47208148)
28. Wang, G., Salles, M., Sowell, B., Wang, X., Cao, T., Demers, A., Gehrke, J., and White, W. (2010) Behavioral simulations in mapreduce. Proceedings of the VLDB Endowment, 3, 952-963.
29. He, B., Fang, W., Luo, Q., Govindaraju, N., and Wang, T. (2008) Mars: A mapreduce framework on graphics processors. Proceedings of the 17th international conference on Parallel architectures and compilation techniques, pp. 260-269, ACM.
30. Olston, C., Reed, B., Srivastava, U., Kumar, R., and Tomkins, A. (2008) Pig latin: A not-so-foreign language for data processing. Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1099-1110, ACM.
31. Thusoo, A., Sarma, J., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., and Murthy, R. (2009) Hive: A warehousing solution over a map-reduce framework. Proceedings of the VLDB Endowment, 2, 1626-1629.
32. Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, ́ U., Gunda, P., and Currey, J. (2008) Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language. Proceedings of the 8th USENIX conference on Operating systems design and implementation, pp. 1-14.
33. Peng, D. and Dabek, F. (2010) Large-scale incremental processing using distributed transactions and notifications. Proceedings of the 9th USENIX conference on Operating systems design and implementation, pp. 1-15, USENIX Association.
34. Melnik, S., Gubarev, A., Long, J., Romer, G., Shivakumar, S., Tolton, M., and Vassi-lakis, T. (2010) Dremel: Interactive analysis of web-scale datasets. Proceedings of the VLDB Endowment, 3, 330-339.
35. Corbett, J., et al. (2012) Spanner: Googles globally-distributed database. To appear in Proceedings of OSDI, p. 1.
36. Zaharia, M., Chowdhury, M., Franklin, M., Shenker, S., and Stoica, I. (2010) Spark: Cluster computing with working sets. Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pp. 10-10, USENIX Association.
37. Neumeyer, L., Robbins, B., Nair, A., and Kesari, A. (2010) S4: Distributed stream com-puting platform. Data Mining Workshops (ICDMW), 2010 IEEE International Con-ference on, pp. 170-177, IEEE
38. Qi, H., Li, K., Shen, Y., and Qu, W. (2012) Object-based image retrieval with kernel on adjacency matrix and local combined features. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), 8, 54.
39. Lee, K., Lee, Y., Choi, H., Chung, Y., and Moon, B. (2012) Parallel data processing with mapreduce: A survey. ACM SIGMOD Record, 40, 11-20.
40. Yang, H., Dasdan, A., Hsiao, R., and Parker, D. (2007) Map-reduce-merge: Simplified relational data processing on large clusters. Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pp. 1029-1040, ACM. (Pubitemid 47630879)
41. Jiang, D., Tung, A., and Chen, G. (2011) Map-Join-Reduce: Toward scalable and effi-cient data analysis on large clusters. Knowledge and Data Engineering, IEEE Transac-tions on, 23, 1299-1311.
42. Nykiel, T., Potamias, M., Mishra, C., Kollios, G., and Koudas, N. (2010) Mrshare: Sharing across multiple queries in mapreduce. Proceedings of the VLDB Endowment, 3, 494-505.
43. Liu, T., Rosenberg, C., and Rowley, H. (2007) Clustering billions of images with large scale nearest neighbor search. Applications of Computer Vision, 2007. WACV'07. IEEE Workshop on, pp. 28-28, IEEE.
44. Akdogan, A., Demiryurek, U., Banaei-Kashani, F., and Shahabi, C. (2010) Voronoi-based geospatial query processing with mapreduce. Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on, pp. 9-16, IEEE.
45. Menon, R., Bhat, G., and Schatz, M. (2011) Rapid parallel genome indexing with mapreduce. Proceedings of the second international workshop on MapReduce and its applications, pp. 51-58, ACM.
46. Logothetis, D. and Yocum, K. (2008) Ad-hoc data processing in the cloud. Proceedings of the VLDB Endowment, 1, 1472-1475.
47. Ji, C., Dong, T., Li, Y., Shen, Y., Li, K., Qiu, W., Qu, W., and Guo, M. (2012) Inverted grid-based knn query processing with mapreduce. ChinaGrid, 2012 Seventh ChinaGrid Annual Conference on, pp. 25-33, IEEE.
48. Wang, J., Wu, S., Gao, H., Li, J., and Ooi, B. (2010) Indexing multi-dimensional data in a cloud system. Proceedings of the 2010 international conference on Management of data, pp. 591-602, ACM.
49. Ekanayake, J., Li, H., Zhang, B., Gunarathne, T., Bae, S., Qiu, J., and Fox, G. (2010) Twister: A runtime for iterative mapreduce. Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp. 810-818, ACM.
50. Bu, Y., Howe, B., Balazinska, M., and Ernst, M. (2010) Haloop: Efficient iterative data processing on large clusters. Proceedings of the VLDB Endowment, 3, 285-296.
51. Malewicz, G., Austern, M., Bik, A., Dehnert, J., Horn, I., Leiser, N., and Czajkowski, G. (2010) Pregel: A system for large-scale graph processing. Proceedings of the 2010 international conference on Management of data, pp. 135-146, ACM.
52. Condie, T., Conway, N., Alvaro, P., Hellerstein, J., Elmeleegy, K., and Sears, R. (2010) Mapreduce online. Proceedings of the 7th USENIX conference on Networked systems design and implementation, pp. 21-21.
53. Condie, T., Conway, N., Alvaro, P., Hellerstein, J., Gerth, J., Talbot, J., Elmeleegy, K., and Sears, R. (2010) Online aggregation and continuous query support in mapreduce. ACM SIGMOD, pp. 1115-1118.
54. Jiang, D., Ooi, B., Shi, L., and Wu, S. (2010) The performance of mapreduce: An in-depth study. Proceedings of the VLDB Endowment, 3, 472-483.
55. Vernica, R., Carey, M., and Li, C. (2010) Efficient parallel set-similarity joins using mapreduce. SIGMOD conference, pp. 495-506, Citeseer.
56. Zhang, C., Li, F., and Jestes, J. (2012) Efficient parallel knn joins for large data in mapreduce. Proceedings of the 15th International Conference on Extending Database Technology, pp. 38-49, ACM.
57. Ibrahim, S., Jin, H., Lu, L., Wu, S., He, B., and Qi, L. (2010) Leen: Locality/fairness-aware key partitioning for mapreduce in the cloud. Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on, pp. 17-24, IEEE.
58. Gufler, B., Augsten, N., Reiser, A., and Kemper, A. (2012) Load balancing in mapreduce based on scalable cardinality estimates. Data Engineering (ICDE), 2012 IEEE 28th International Conference on, pp. 522-533, IEEE.
59. Xu, Y., Zou, P., Qu, W., Li, Z., Li, K., and Cui, X. (2012) Sampling-based partitioning in mapreduce for skewed data. ChinaGrid, 2012 Seventh ChinaGrid Annual Conference on, pp. 1-8, IEEE.
60. Tan, J., Meng, X., and Zhang, L. (2012) Delay tails in mapreduce scheduling. Proceed-ings of the 12th ACM SIGMETRICS/PERFORMANCE joint international conference on Measurement and Modeling of Computer Systems, pp. 5-16, ACM.
61. Wolf, J., Rajan, D., Hildrum, K., Khandekar, R., Kumar, V., Parekh, S., Wu, K., and Balmin, A. (2010) Flex: A slot allocation scheduling optimizer for mapreduce workloads. Middleware 2010, pp. 1-20.
62. Sandholm, T. and Lai, K. (2009) Mapreduce optimization using regulated dynamic pri-oritization. Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems, pp. 299-310, ACM.
63. Kllapi, H., Sitaridi, E., Tsangaris, M., and Ioannidis, Y. (2011) Schedule optimization for data processing flows on the cloud. Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 289-300.
64. Zhou, X., Lu, J., Li, C., and Du, X. (2012) Big data challenge in the management perspective. Communications of the CCF, 8, 16-20.