A survey of big data management: Taxonomy and state-of-the-art

Journal of Network and Computer Applications

Volumn 71, Issue , 2016, Pages 151-166

  Siddiqa, Aisha ^a   Hashem, Ibrahim Abaker Targio ^a   Yaqoob, Ibrar ^a   Marjani, Mohsen ^a   Shamshirband, Shahabuddin ^a   Gani, Abdullah ^a   Nasaruddin, Fariza ^a  

^a UNIVERSITY OF MALAYA   (Malaysia)

Author keywords

Big data; Big data management; Processing; Security; Storage

Indexed keywords

DATA HANDLING; DIGITAL STORAGE; ENERGY STORAGE; INFORMATION MANAGEMENT; PROCESSING; STORAGE MANAGEMENT; SURVEYS; TAXONOMIES;

CLOUD COMPUTING TECHNOLOGIES; DATA MANAGEMENT TECHNIQUES; EMERGING APPLICATIONS; FUTURE RESEARCH DIRECTIONS; PRE-PROCESSING; RAPID GROWTH; SECURITY; STATE OF THE ART;

BIG DATA;

EID: 84978946777     PISSN: 10848045     EISSN: 10958592     Source Type: Journal    
DOI: 10.1016/j.jnca.2016.04.008     Document Type: Article

References (138)

1. Agrawal, D., El Abbadi, A., Antony, S., Das, S., 2010. Data management challenges in cloud computing infrastructures, Databases in Networked Information Systems, vol. 5999, 2010, Springer, pp.1–10.
2. Azevedo, D.R., de Oliveira, J.P., Application of data mining techniques to storage management and online distribution of satellite images. Nedjah, N., de Macedo Mourelle, L., Kacprzyk, J., (eds.) Innovative Applications in Data Mining, 2009, Springer, Berlin Heidelberg, 1–15.
3. Azeem, R., Khan, M.I.A., Techniques about data replication for mobile ad-hoc network databases. Int J. Multidiscip. Sci. Eng., 3, 2012, 5.
4. Ahamed, B.B., Ramkumar, T., Hariharan, S., 2014. Data integration progression in large data source using mapping affinity. In: Proceedings of the 7th International Conference on Advanced Software Engineering and Its Applications (ASEA).
5. Abu-Libdeh, H., et al. Symbiotic routing in future data centers. ACM SIGCOMM Comput. Commun. Rev. 41:4 (2011), 51–62.
6. Armstrong, J., OFDM for optical communications. J. Light Technol. 27:3 (2009), 189–204.
7. Achtert, E., et al., 2007. On exploring complex relationships of correlation clusters. In: Proceedings of the IEEE SSBDM’07 19th International Conference on Scientific and Statistical Database Management.
8. Agrawal, D., Aggarwal, C.C., 2001. On the design and quantification of privacy preserving data mining algorithms. In: Proceedings of the Twentieth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems. ACM.
9. Agrawal, D., Aggarwal, C.C., 2001. On the design and quantification of privacy preserving data mining algorithms. In: Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of database systems. ACM, Santa Barbara, California, USA, pp. 247–255.
10. Assunção, M.D., et al. Big Data computing and clouds: trends and future directions. J. Parallel Distrib. Comput., 2014.
11. Assunçaoa, M.D., et al., 2013. Big Data Computing and Clouds: Challenges, Solutions, and Future Directions. arXiv preprint arXiv:1312.4722.
12. Borkar, V., Carey, M.J., Li, C., 2012. Inside Big Data management: ogres, onions, or parfaits? In: Proceedings of the 15th International Conference on Extending Database Technology. ACM.
13. Baker, T, 2014. Designing and managing Big Data – How are you researching your outcomes. In SimTecT 2014.
14. Buza, K., Nagy, G.I., Nanopoulos, A., Storage-optimizing clustering algorithms for high-dimensional tick data. Expert Syst. Appl. 41:9 (2014), 4148–4157.
15. Buza, K., Buza, A., Kis, P.B., 2011. A distributed genetic algorithm for graph-based clustering Man-Machine Interactions 2, Springer, pp. 323–331.
16. Bautista Villalpando, L.E., April, A., Abran, A., DIPAR: a framework for implementing big data science in organizations. Mahmood, Z., (eds.) Continued Rise of the Cloud, 2014, Springer, London, 177–194.
17. Baskar, S., Arockiam, L., Charles, S., A systematic approach on data pre-processing in data mining. Int J. Adv. Comput. Technol. (IJACT), 2, 2013 2320-0790.
18. Bohannon, P., et al., 2007. Conditional functional dependencies for data cleaning. In: Proceedings of the IEEE 23rd International Conference on ICDE Data Engineering.
19. Begoli, E., Horey, J., 2012. Design principles for effective knowledge discovery from big data. In: Proceedings of the IEEE 2012 Joint Working IEEE/IFIP Conference on Software Architecture (WICSA) and European Conference on Software Architecture (ECSA).
20. Bertsekas, D.P., 1999. Nonlinear programming.
21. Bakshi, K., 2012. Considerations for big data: architecture and approach. In: Proceedings of IEEE Aerospace Conference.
22. Beldiceanu, N., et al. Toward sustainable development in constraint programming. Constraints 19:2 (2014), 139–149.
23. Candela, L., Castelli, D., Pagano, P., Managing big data through hybrid data infrastructures. ERCIM News 89 (2012), 37–38.
24. Chaudhuri, S., 2012. What next?: a half-dozen data management research goals for big data and the cloud. In: Proceedings of the 31st Symposium on Principles of Database Systems. ACM.
25. Chaudhuri, S., Dayal, U., Narasayya, V., An overview of business intelligence technology. Commun. ACM 54:8 (2011), 88–98.
26. Chen, H., et al., 2010. Leveraging spatio-temporal redundancy for RFID data cleansing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. ACM.
27. Chen, J., et al. Big data challenge: a data management perspective. Front. Comput. Sci. 7:2 (2013), 157–164.
28. Craig, T., Ludloff, M.E., Privacy and Big Data. 2011, O'Reilly Media, Inc.
29. Chen, M., Mao, S., Liu, Y., Big data: a survey. Mob. Netw. Appl. 19:2 (2014), 171–209.
30. Chen, T., et al. A sentence vector based over-sampling method for imbalanced emotion classification. Gelbukh, A., (eds.) Computational Linguistics and Intelligent Text Processing, 2014, Springer, Berlin Heidelberg, 62–72.
31. Chapman, A.D., Principles of Data Quality. 2005, GBIF.
32. Chapman, A.D., Principles and Methods of Data Cleaning: Primary Species and Species-Occurrence Data. 2005, Global Biodiversity Information Facility.
33. Dahan, H., Cohen, S., Rokach, L., & Maimon, O., 2014. Proactive Data Mining Using Decision Trees Proactive Data Mining with Decision Trees, New York, NY: Springer New York, pp. 21-33.
34. Dittrich, J., Blunschi, L., Salles, M. Vaz, MOVIES: indexing moving objects by shooting index images. GeoInformatica 15:4 (2011), 727–767.
35. Dalcin, E.C., Data Quality Concepts and Techniques Applied to Taxonomic Databases. 2005, University of Southampton.
36. Davies, D.L., Bouldin, D.W., A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence 2 (1979), 224–227.
37. Dunn, J.C., A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters. Journal of cybernetics 3:3 (1973), 32–57.
38. Estivill-Castro, V., Why so many clustering algorithms: a position paper. ACM SIGKDD Explor. Newsl 4:1 (2002), 65–75.
39. Erway, C., et al., 2009. Dynamic provable data possession. In: Proceedings of the 16th ACM conference on Computer and Communications Security. ACM.
40. Ferguson, P., Huston, G., Quality of Service: Delivering QoS on the Internet and in Corporate Networks, 1998, John Wiley & Sons, Inc.
41. Farrington, N., et al. Helios: a hybrid electrical/optical switch architecture for modular data centers. SIGCOMM Comput. Commun. Rev. 40:4 (2010), 339–350.
42. Fluhrer, S., Mantin, I., Shamir, A., 2001. Weaknesses in the key scheduling algorithm of RC4. In Selected areas in cryptography, Springer, Berlin Heidelberg, pp. 1–24.
43. Gani, A., et al. A survey on indexing techniques for big data: taxonomy and performance evaluation. Knowl. Inf. Syst., 2015, 1–44.
44. Greenwald, G., 2014a NSA Collecting Phone Records of Millions of Verizon Customers Daily, Guardian (cited 03.03.14); Available from: 〈 http://www.guardian.co.uk/world/2013/jun/06/nsa-phone-records-verizon-court-order〉.
45. Greenwald, G., 2014b. Glenn Greenwald & Ewen MacAskill, NSA Prism Program Taps in to User Data of Apple, Google and Others (cited 07.03.14); Available from: 〈 http://www.guardian.co.uk/world/2013/jun/06/us-tech-giants-nsa-data〉.
46. Giese, M., et al. Scalable end-user access to big data. Big Data Comput., 2013, 205–245.
47. Gandomi, A., Haider, M., Beyond the hype: big data concepts, methods, and analytics. Int. J. Inf. Manag. 35:2 (2015), 137–144.
48. Han, H., et al. Toward scalable systems for big data analytics: a technology tutorial. IEEE Access 2 (2014), 652–687.
49. Hu, H., et al., 2014. Towards Scalable Systems for Big Data Analytics: A Technology Tutorial.
50. Han, J., 2006. Data Mining: Concepts and Techniques.
51. Herbert, K.G., Wang, J.T., Biological data cleaning: a case study. Int. J. Inf. Qual. 1:1 (2007), 60–82.
52. Huang, Y., Sistla, P., Wolfson, O., 1994. Data replication for mobile computers. SIGMOD Rec., 23 (2), 13–24, http://dx.doi.org/10.1145/191843.191845.
53. Hashem, I.A.T., et al. The rise of “big data” on cloud computing: Review and open research issues. Inf. Syst. 47 (2015), 98–115.
54. Halevy, A., Rajaraman, A., Ordille, J., 2006. Data integration: the teenage years. In: Proceedings of the 32nd international conference on Very large data bases. VLDB Endowment.
55. Hyoung Woo, P., et al., 2013. Study on big data center traffic management based on the separation of large-scale data stream. In: Proceedings of the Seventh International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS).
56. Ivezić, Ž., et al. Statistics, Data Mining, and Machine Learning in Astronomy: A Practical Python Guide for the Analysis of Survey Data, 2014, Princeton University Press.
57. Jinno, M., Takara, H., Kozicki, B., 2009. Dynamic optical mesh networks: drivers, challenges and solutions for the future. ECOC 2009.
58. Kambatla, K., et al. Trends in big data analytics. J. Parallel Distrib. Comput. 74:7 (2014), 2561–2573.
59. Karaboga, D., Ozturk, C., A novel clustering approach: artificial Bee Colony (ABC) algorithm. Appl. Soft Comput. 11:1 (2011), 652–657.
60. Kumar, V., Chadha, A., Mining association rules in student's assessment data. Int. J. Comput. Sci. Issues 9:5 (2012), 211–216.
61. Kachris, C., Tomkos, I., 2012. The rise of optical interconnects in data centre networks. In: Proceedings of the IEEE 14th International Conference on Transparent Optical Networks (ICTON).
62. Kohavi, R., et al. Lessons and challenges from mining retail e-commerce data. Mach. Learn. 57:1–2 (2004), 83–113.
63. Khoussainova, N., Balazinska, M., Suciu, D., 2008. Probabilistic event extraction from RFID data. In: ICDE.
64. Karim, Ahmad, S.A.A.S., Salleh, Rosli, Arif, Muhammad, Noor, Rafidah Md, Shamshirband, Shahaboddin, Mobile botnet attacks – an emerging threat: classification, review and open issues. KSII Trans. Internet Inf. Syst., 9, 2015, 4.
65. Karim, A., Shah, S.A.A., Salleh, R., Mobile Botnet Attacks: A Thematic Taxonomy. Rocha, Á., Correia, M.A., Tan, B.F., Stroetmann, A.K., (eds.) New Perspectives in Information Systems and Technologies, vol. 2, 2014, Springer International Publishing, Cham, 153–164.
66. Kumar, M.A., Karthikeyan, S., Investigating the efficiency of blowfish and rejindael (AES) algorithms. Int. J. Comput. Netw. Inf. Secur. (IJCNIS), 4(2), 2012, 22.
67. Karim, A., et al. Botnet detection techniques: review, future trends, and issues. J. Zhejiang Univ. Sci. C 15:11 (2014), 943–983.
68. Kwon, O., Lee, N., Shin, B., Data quality management, data usage experience and acquisition intention of big data analytics. Int. J. Inf. Manag. 34:3 (2014), 387–394.
69. Khatri, V., Brown, C.V., Designing data governance. Commun. ACM 53:1 (2010), 148–152.
70. Kshetri, N., Big data's impact on privacy, security and consumer welfare. Telecommun. Policy 38:11 (2014), 1134–1145.
71. Lam, C.F., et al. Fiber optic communication technologies: what's needed for datacenter network operations. IEEE Commun. Mag. 48:7 (2010), 32–39.
72. Lamehamedi, H., et al., 2002. Data replication strategies in grid environments. In: Proceedings of the IEEE Fifth International Conference on Algorithms and Architectures for Parallel Processing.
73. Labrinidis, A., Jagadish, H., Challenges and opportunities with big data. Proc. VLDB Endow. 5:12 (2012), 2032–2033.
74. Liu, C., et al. External integrity verification for outsourced big data in cloud and IoT: a big picture. Future Gener. Comput. Syst., 2014.
75. Li, H. Lu, X., 2014. Challenges and trends of big data analytics. In: Proceedings of the IEEE Ninth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC).
76. Liu, J., Zhang, X., Data integration in fuzzy XML documents. Inf. Sci. 280 (2014), 82–97.
77. Lafuente, G., The big data security challenge. Netw. Secur. 2015:1 (2015), 12–14.
78. Manyika, J., et al., 2011. Big Data: The Next Frontier for Innovation, Competition, and Productivity.
79. McAfee, A., Brynjolfsson, E., Big data: the management revolution. Harv. Bus. Rev. 68:90 (2012), 60–66 128.
80. Mendel, J.M., Korjani, M.M., On establishing nonlinear combinations of variables from small to big data for use in later processing. Inf. Sci. 280:0 (2014), 98–110.
81. Maletic, J.I., Marcus, A. Data Cleansing: Beyond Integrity Analysis. In IQ, 2000, pp. 200–209.
82. Michael, K., Miller, K.W., Big data: new opportunities and new challenges. IEEE Comput. 46:6 (2013), 22–24 Editorial.
83. Meroufel, B., Belalem, G., Managing data replication and placement based on availability. AASRI Procedia 5 (2013), 147–155.
84. Ma'ayan, A., et al. Lean big data integration in systems biology and systems pharmacology. Trends Pharmacol. Sci. 35:9 (2014), 450–460.
85. O'Leary, D.E., Artificial intelligence and big data. IEEE Intell. Syst. 28:2 (2013), 96–99.
86. Philip Chen, C.L., Zhang, C.-Y., Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275:0 (2014), 314–347.
87. Paul, A., et al. Video search and indexing with reinforcement agent for interactive multimedia services. ACM Trans. Embed. Comput. Syst. 12:2 (2013), 1–16.
88. Park, H.W., et al., 2013. Study on big data center traffic management based on the separation of large-scale data stream. In: Proceedings of the IEEE Seventh International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS).
89. Priyadharshini, B., Parvathi, P., 2012. Data integrity in cloud storage. In: Proceedings of IEEE International Conference on Advances in Engineering, Science and Management (ICAESM).
90. Putnik, G., et al. Scalability in manufacturing systems design and operation: state-of-the-art and future developments roadmap. CIRP Ann. – Manuf. Technol. 62:2 (2013), 751–774.
91. Palmieri, F., et al. GRASP-based resource re-optimization for effective big data access in federated clouds. Future Gener. Comput. Syst., 2015.
92. Russom, P., 2011. Big data analytics. TDWI Best Practices Report, Fourth Quarter.
93. Russom, P., 2013. Managing big data. TDWI.
94. Roweis, S.T., Saul, L.K., Nonlinear dimensionality reduction by locally linear embedding. Science 290:5500 (2000), 2323–2326.
95. Rousseeuw, P.J., Driessen, K.V., A fast algorithm for the minimum covariance determinant estimator. Technometrics 41:3 (1999), 212–223.
96. Rubinstein, I.S., 2012. Big Data: The End of Privacy or a New Beginning?.
97. Rubinstein, I., Big data: the end of privacy or a new beginning?. NYU Sch. Law Public Law Res. Pap., 2012, 12–56.
98. Redman, T.C., Data Quality for the Information Age, 1997, Artech House Inc, 303.
99. Spaho, E., et al. P2P data replication and trustworthiness for a JXTA-Overlay P2P system using fuzzy logic. Appl. Soft Comput. 13:1 (2013), 321–328.
100. Sun, D.-W., et al. Modeling a dynamic data replication strategy to increase system availability in cloud computing environments. J. Comput. Sci. Technol. 27:2 (2012), 256–272.
101. Santhanam, T, Padmavathi, M.S., 2014. Comparison of K-Means clustering and statistical outliers in reducing medical datasets. In: Proceedings of International Conference on Science Engineering and Management Research (ICSEMR).
102. Shieh, W., OFDM for flexible high-speed optical networks. J. Light Technol. 29:10 (2011), 1560–1577.
103. Squicciarini, A., Sundareswaran, S., Lin, D, 2010. Preventing information leakage from indexing in the cloud. In: Proceedings of the IEEE 3rd International Conference on Cloud Computing (CLOUD).
104. Sebé, F., et al. Efficient remote data possession checking in critical information infrastructures. IEEE Trans. Knowl. Data Eng. 20:8 (2008), 1034–1038.
105. Standaert, F.-X., Rouvroy, G., Quisquater, J.J., 2006. FPGA implementations of the DES and triple-DES masked against power analysis attacks. In: Proceedings of FPL’06 IEEE International Conference on Field Programmable Logic and Applications.
106. Schneier, B., et al. The Twofish Encryption Algorithm: A 128-bit Block Cipher, 1999, John Wiley & Sons, Inc.
107. Steinwart, I., Christmann, A., Support Vector Machines, 2008, Springer.
108. Steed, C.A., et al. Big data visual analytics for exploratory earth system simulation analysis. Comput. Geosci. 61 (2013), 71–82.
109. Shamshirband, S., et al. Co-FAIS: cooperative fuzzy artificial immune system for detecting intrusion in wireless sensor networks. J. Netw. Comput. Appl. 42 (2014), 102–117.
110. Tan, P.-N., Steinbach, M., Kumar, V., Introduction to Data Mining, First Ed., 2005, Addison-Wesley Longman Publishing Co., Inc.
111. Taheri, J., et al. A bee colony based optimization approach for simultaneous job scheduling and data replication in grid environments. Comput. Oper. Res. 40:6 (2013), 1564–1578.
112. Talia, D., Clouds for scalable big data analytics. Computer 46:5 (2013), 98–101.
113. Tankard, C., Big data security. Netw. Secur. 2012:7 (2012), 5–8.
114. Tene, O., Polonetsky, J., 2013. Big Data for All: Privacy and User Control in the Age of Analytics.
115. Thakur, J., Kumar, N., DES, AES and Blowfish: symmetric key cryptography algorithms simulation based performance analysis. Int. J. Emerg. Technol. Adv. Eng. 1:2 (2011), 6–12.
116. Talia, D., 2013. Toward Cloud-based Big Data Analytics.
117. Tallon, P.P., Corporate governance of big data: Perspectives on value, risk, and cost. Computer 46:6 (2013), 32–38.
118. Van den Broeck, J., et al. Maintaining data integrity in a rural clinical trial. Clin. Trials 4:5 (2007), 572–582.
119. Wang, M., et al. High volumes of event stream indexing and efficient multi-keyword searching for cloud monitoring. Future Gener. Comput. Syst. 29:8 (2013), 1943–1962.
120. Wright, D., De Hert, P., Introduction to Privacy Impact Assessment, 2012, Springer.
121. Wang, Q., Wang, C., Li, J., Ren, K., Lou, W., 2009. Enabling public verifiability and data dynamics for storage security in cloud computing. In Computer Security–ESORICS, Springer, Berlin Heidelberg, pp. 355–370.
122. Wang, C., et al., 2010. Privacy-preserving public auditing for data storage security in cloud computing. In: Proceedings of the IEEE INFOCOM.
123. Weatherspoon, H., Kubiatowicz, J.D., 2002. Erasure coding vs. replication: A quantitative comparison. In Peer-to-Peer Systems, Springer Berlin Heidelberg, pp. 328–337.
124. Wolfson, O., Jajodia, S., Huang, Y., An adaptive data replication algorithm. ACM Trans. Database Syst. (TODS) 22:2 (1997), 255–314.
125. Wu, Y., Network big data: a literature survey on stream data mining. J. Softw. 9:9 (2014), 2427–2434.
126. Witten, I.H., Frank, E., Data Mining: Practical Machine Learning Tools and Techniques, 2005, Morgan Kaufmann.
127. Xuyun, Z., et al., 2012. Privacy-preserving layer over map reduce on cloud. In: Proceedings of Second International Conference on Cloud and Green Computing (CGC).
128. Xuyun, Z., et al. A privacy leakage upper bound constraint-based approach for cost-effective privacy preserving of intermediate data sets in cloud. IEEE Trans. Parallel Distrib. Syst. 24:6 (2013), 1192–1202.
129. Xu, Q., Liu, G., 2009. Configuring clark-wilson integrity model to enforce flexible protection. In: Proceedings of IEEE CIS’09 International Conference on Computational Intelligence and Security.
130. Xu, L.-X., Sun, D., Liu, D., 2010. Study on methods for data confidentiality and data integrity in relational database. In: Proceedings of the IEEE 3rd IEEE International Conference onComputer Science and Information Technology (ICCSIT).
131. Xu, L., et al., 2015. ABSS: an attribute-based sanitizable signature for integrity of outsourced database with public cloud. In: Proceedings of the 5th ACM Conference on Data and Application Security and Privacy. ACM, San Antonio, Texas, USA. pp. 167–169.
132. Yogish, H., Raju, D.G., Manjunath, T., The descriptive study of knowledge discovery from web usage mining. IJCSI Int. J. Comput. Sci. Issues, 8(5), 2011.
133. Yiqun, Z., Yiu-ming, C., 2014. Discretizing numerical attributes in decision tree for big data analysis. In: Proceedings of IEEE International Conference on Data Mining Workshop (ICDMW).
134. Yun-Peng, Z., et al., 2009. Digital image encryption algorithm based on chaos and improved DES. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, SMC.
135. Zhao, W., Ma, H., He, Q., 2009. Parallel k-means clustering based on mapreduce. In Cloud computing, Springer, Berlin Heidelberg, pp. 674–679.
136. Zhao, Z., Ng, W., 2012. A model-based approach for rfid data stream cleansing. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management. ACM.
137. Zhang, M. Strict integrity policy of biba model with dynamic characteristics and its correctness. In: Proceedings of IEEE International Conference on Computational Intelligence and Security, CIS’09.
138. Zografos, K.G., Androutsopoulos, K.N., A heuristic algorithm for solving hazardous materials distribution problems. Eur. J. Oper. Res. 152:2 (2004), 507–519.