Big Data Reduction Methods: A Survey

Data Science and Engineering

Volumn 1, Issue 4, 2016, Pages 265-284

  ur Rehman, Muhammad Habib ^a   Liew, Chee Sun ^a   Abbas, Assad ^b   Jayaraman, Prem Prakash ^c   Wah, Teh Ying ^a   Khan, Samee U ^b  

^a UNIVERSITY OF MALAYA   (Malaysia)

^b North Dakota State University   (United States)

^c SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Big data; Data complexity; Data compression; Data reduction; Dimensionality reduction

Indexed keywords

EID: 85029474107     PISSN: 23641185     EISSN: 23641541     Source Type: Journal    
DOI: 10.1007/s41019-016-0022-0     Document Type: Article

References (85)

1. Wu X et al (2014) Data mining with big data. IEEE Trans Knowl Data Eng 26(1):97–107
2. Che D, Safran M, Peng Z (2013) From big data to big data mining: challenges, issues, and opportunities. In: Database systems for advanced applications
3. Battams K (2014) Stream processing for solar physics: applications and implications for big solar data. arXiv preprint arXiv:1409.8166
4. Zhai Y, Ong Y-S, Tsang IW (2014) The emerging “big dimensionality”. Comput Intell Mag IEEE 9(3):14–26
5. Fan J, Han F, Liu H (2014) Challenges of big data analysis. Nat Sci Rev 1(2):293–314
6. Chandramouli B, Goldstein J, Duan S (2012) Temporal analytics on big data for web advertising. In: 2012 IEEE 28th international conference on data engineering (ICDE)
7. Ward RM et al (2013) Big data challenges and opportunities in high-throughput sequencing. Syst Biomed 1(1):29–34
8. Weinstein M et al (2013) Analyzing big data with dynamic quantum clustering. arXiv preprint arXiv:1310.2700
9. Hsieh C-J et al (2013) BIG & QUIC: sparse inverse covariance estimation for a million variables. In: Advances in neural information processing systems
10. Vervliet N et al (2014) Breaking the curse of dimensionality using decompositions of incomplete tensors: tensor-based scientific computing in big data analysis. IEEE Signal Process Mag 31(5):71–79
11. Feldman D, Schmidt M, Sohler C (2013) Turning big data into tiny data: constant-size coresets for k-means, pca and projective clustering. In: Proceedings of the twenty-fourth annual ACM-SIAM symposium on discrete algorithms
12. Fu Y, Jiang H, Xiao N (2012) A scalable inline cluster deduplication framework for big data protection. In: Middleware 2012. Springer, pp 354–373
13. Zhou R, Liu M, Li T (2013) Characterizing the efficiency of data deduplication for big data storage management. In: 2013 IEEE international symposium on workload characterization (IISWC)
14. Dong W et al (2011) Tradeoffs in scalable data routing for deduplication clusters. In: FAST
15. Xia W et al (2011) SiLo: a similarity-locality based near-exact deduplication scheme with low RAM overhead and high throughput. In: USENIX annual technical conference
16. Trovati M, Asimakopoulou E, Bessis N (2014) An analytical tool to map big data to networks with reduced topologies. In: 2014 international conference on intelligent networking and collaborative systems (INCoS)
17. Fang X, Zhan J, Koceja N (2013) Towards network reduction on big data. In: 2013 international conference on social computing (SocialCom)
18. Wilkerson AC, Chintakunta H, Krim H (2014) Computing persistent features in big data: a distributed dimension reduction approach. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)
19. Di Martino B et al (2014) Big data (lost) in the cloud. Int J Big Data Intell 1(1–2):3–17
20. Brown CT (2012) BIGDATA: small: DA: DCM: low-memory streaming prefilters for biological sequencing data
21. Lin M-S et al (2013) Malicious URL filtering—a big data application. In 2013 IEEE international conference on big data
22. Chen J et al (2013) Big data challenge: a data management perspective. Front Comput Sci 7(2):157–164
23. Chen X-W, Lin X (2014) Big data deep learning: challenges and perspectives. IEEE Access 2:514–525
24. Chen Z et al (2015) A survey of bitmap index compression algorithms for big data. Tsinghua Sci Technol 20(1):100–115
25. Hashem IAT et al (2015) The rise of “big data” on cloud computing: review and open research issues. Inf Syst 47:98–115
26. Gani A et al (2015) A survey on indexing techniques for big data: taxonomy and performance evaluation. In: Knowledge and information systems, pp 1–44
27. Kambatla K et al (2014) Trends in big data analytics. J Parallel Distrib Comput 74(7):2561–2573
28. Jin X et al (2015) Significance and challenges of big data research. Big Data Res 2(2):59–64
29. Li F, Nath S (2014) Scalable data summarization on big data. Distrib Parallel Databases 32(3):313–314
30. Lohr S (2014) For big-data scientists, ‘janitor work’ is key hurdle to insights. http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html
31. Ma C, Zhang HH, Wang X (2014) Machine learning for big data analytics in plants. Trends Plant Sci 19(12):798–808
32. Ordonez C (2013) Can we analyze big data inside a DBMS? In: Proceedings of the sixteenth international workshop on data warehousing and OLAP
33. Oliveira J, Osvaldo N et al (2014) Where chemical sensors may assist in clinical diagnosis exploring “big data”. Chem Lett 43(11):1672–1679
34. Shilton K (2012) Participatory personal data: an emerging research challenge for the information sciences. J Am Soc Inform Sci Technol 63(10):1905–1915
35. Shuja J et al (2012) Energy-efficient data centers. Computing 94(12):973–994
36. Ahmad RW et al (2015) A survey on virtual machine migration and server consolidation frameworks for cloud data centers. J Netw Comput Appl 52:11–25
37. Bonomi F et al (2014) Fog computing: a platform for internet of things and analytics. In: Big data and internet of things: a roadmap for smart environments. Springer, pp 169–186
38. Rehman MH, Liew CS, Wah TY (2014) UniMiner: towards a unified framework for data mining. In: 2014 fourth world congress on information and communication technologies (WICT)
39. Patty JW, Penn EM (2015) Analyzing big data: social choice and measurement. Polit Sci Polit 48(01):95–101
40. Trovati M (2015) Reduced topologically real-world networks: a big-data approach. Int J Distrib Syst Technol (IJDST) 6(2):13–27
41. Trovati M, Bessis N (2015) An influence assessment method based on co-occurrence for topologically reduced big data sets. In: Soft computing, pp 1–10
42. Dey TK, Fan F, Wang Y (2014) Computing topological persistence for simplicial maps. In: Proceedings of the thirtieth annual symposium on computational geometry
43. Zou H et al (2014) Flexanalytics: a flexible data analytics framework for big data applications with I/O performance improvement. Big Data Res 1:4–13
44. Ackermann K, Angus SD (2014) A resource efficient big data analysis method for the social sciences: the case of global IP activity. Procedia Comput Sci 29:2360–2369
45. Yang C et al (2014) A spatiotemporal compression based approach for efficient big data processing on Cloud. J Comput Syst Sci 80(8):1563–1583
46. Monreale A et al (2013) Privacy-preserving distributed movement data aggregation. In: Geographic information science at the heart of Europe. Springer, pp 225–245
47. Jalali B, Asghari MH (2014) The anamorphic stretch transform: putting the squeeze on “big data”. Opt Photonics News 25(2):24–31
48. Wang W et al (2013) Statistical wavelet-based anomaly detection in big data with compressive sensing. EURASIP J Wirel Commun Netw 2013(1):1–6
49. He B, Li Y (2014) Big data reduction and optimization in sensor monitoring network. J Appl Math. doi:10.1155/2014/294591
50. Brinkmann BH et al (2009) Large-scale electrophysiology: acquisition, compression, encryption, and storage of big data. J Neurosci Methods 180(1):185–192
51. Zou H et al (2014) Improving I/O performance with adaptive data compression for big data applications. In: 2014 IEEE international parallel & distributed processing symposium workshops (IPDPSW)
52. Lakshminarasimhan S et al (2011) Compressing the incompressible with ISABELA: in situ reduction of spatio-temporal data. In: Euro-Par 2011 parallel processing. Springer, pp 366–379
53. Ahrens JP et al (2009) Interactive remote large-scale data visualization via prioritized multi-resolution streaming. In: Proceedings of the 2009 workshop on ultrascale visualization
54. Compression utility, gzip. http://www.gzip.org
55. Bi C et al (2013) Proper orthogonal decomposition based parallel compression for visualizing big data on the K computer. In: 2013 IEEE symposium on large-scale data analysis and visualization (LDAV)
56. Bhagwat D, Eshghi K, Mehra P (2007) Content-based document routing and index partitioning for scalable similarity-based searches in a large corpus. In: Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining
57. Rupprecht L (2013) Exploiting in-network processing for big data management. In: Proceedings of the 2013 SIGMOD/PODS Ph.D. symposium
58. Zhao D et al (2015) COUPON: a cooperative framework for building sensing maps in mobile opportunistic networks. IEEE Trans Parallel Distrib Syst 26(2):392–402
59. Zerbino DR, Birney E (2008) Velvet: algorithms for de novo short read assembly using de Bruijn graphs. Genome Res 18(5):821–829
60. Cheng Y, Jiang P, Peng Y (2014) Increasing big data front end processing efficiency via locality sensitive Bloom filter for elderly healthcare. In: 2014 IEEE symposium on computational intelligence in big data (CIBD)
61. Dredze M, Crammer K, Pereira F (2008) Confidence-weighted linear classification. In: Proceedings of the 25th international conference on machine learning
62. Crammer K et al (2006) Online passive-aggressive algorithms. J Mach Learn Res 7:551–585
63. Hillman C et al (2014) Near real-time processing of proteomics data using Hadoop. Big Data 2(1):44–49
64. Sugumaran R, Burnett J, Blinkmann A (2012) Big 3d spatial data processing using cloud computing environment. In: Proceedings of the 1st ACM SIGSPATIAL international workshop on analytics for big geospatial data
65. Friedman J, Hastie T, Tibshirani R (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3):432–441
66. Scheinberg K, Ma S, Goldfarb D (2010) Sparse inverse covariance selection via alternating linearization methods. In: Advances in neural information processing systems
67. Qiu J, Zhang B (2013) Mammoth data in the cloud: clustering social images. Clouds Grids Big Data 23:231
68. Hoi SC et al (2012) Online feature selection for mining big data. In: Proceedings of the 1st international workshop on big data, streams and heterogeneous source mining: algorithms, systems, programming models and applications
69. Hartigan JA, Wong MA (1979) Algorithm AS 136: a k-means clustering algorithm. In: Applied statistics, pp 100–108
70. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1):37–52
71. Azar AT, Hassanien AE (2014) Dimensionality reduction of medical big data using neural-fuzzy classifier. Soft Comput 19(4):1115–1127
72. Cichocki A (2014) Era of big data processing: a new approach via tensor networks and tensor decompositions. arXiv preprint arXiv:1403.2048
73. Dalessandro B (2013) Bring the noise: embracing randomness is the key to scaling up machine learning algorithms. Big Data 1(2):110–112
74. Zeng X-Q, Li G-Z (2014) Incremental partial least squares analysis of big streaming data. Pattern Recogn 47(11):3726–3735
75. Ruhe A (1984) Rational Krylov sequence methods for eigenvalue computation. Linear Algebra Appl 58:391–405
76. Tannahill BK, Jamshidi M (2014) System of systems and big data analytics–Bridging the gap. Comput Electr Eng 40(1):2–15
77. Liu Q et al (2014) Mining the big data: the critical feature dimension problem. In: 2014 IIAI 3rd international conference on advanced applied informatics (IIAIAAI)
78. Jiang P et al (2014) An intelligent information forwarder for healthcare big data systems with distributed wearable sensors. IEEE Syst J PP(99):1–9
79. Leung CK-S, MacKinnon RK, Jiang F (2014) Reducing the search space for big data mining for interesting patterns from uncertain data. In: 2014 IEEE international congress on big data (BigData congress)
80. Stateczny A, Wlodarczyk-Sielicka M (2014) Self-organizing artificial neural networks into hydrographic big data reduction process. In: Rough sets and intelligent systems paradigms. Springer, pp 335–342
81. Hinton GE, Osindero S, Teh Y-W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554
82. LeCun Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
83. Kavukcuoglu K et al (2009) Learning invariant features through topographic filter maps. In: 2009 IEEE conference on computer vision and pattern recognition, CVPR 2009
84. Dean J et al (2012) Large scale distributed deep networks. In: Advances in neural information processing systems
85. Martens J (2010) Deep learning via Hessian-free optimization. In: Proceedings of the 27th international conference on machine learning (ICML-10), June 21–24, Haifa, Israel