A survey on deep learning for big data

Information Fusion

Volumn 42, Issue , 2018, Pages 146-157

  Zhang, Qingchen ^a,b   Yang, Laurence T ^a,b   Chen, Zhikui ^c   Li, Peng ^c  

^a UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b ST FRANCIS XAVIER UNIVERSITY   (Canada)

^c DALIAN UNIVERSITY OF TECHNOLOGY   (China)

Author keywords

Big data; Convolutional neural networks; Deep belief networks; Deep learning; Recurrent neural networks; Stacked auto encoders

Indexed keywords

CHARACTER RECOGNITION; DATA MINING; DEEP LEARNING; LEARNING SYSTEMS; MEDICAL COMPUTING; NEURAL NETWORKS; PATTERN RECOGNITION; RECURRENT NEURAL NETWORKS; SENSOR NETWORKS; SPEECH RECOGNITION;

ANALYTIC SOLUTION; AUTO ENCODERS; COMMUNICATION TECHNOLOGIES; CONVOLUTIONAL NEURAL NETWORK; DEEP BELIEF NETWORKS; HIERARCHICAL ARCHITECTURES; INDUSTRIAL CONTROLS; MACHINE LEARNING TECHNIQUES;

BIG DATA;

EID: 85033462391     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2017.10.006     Document Type: Short Survey

References (102)

1. Kuang, L., Yang, L.T., Liao, Y., An integration framework on cloud for cyber physical social systems big data. IEEE Trans. Cloud Comput., 2015 10.1109/TCC.2015.2511766.
2. Li, Y., Qi, X., Keally, M., Ren, Z., Zhou, G., Xiao, D., Deng, S., Communication energy modeling and optimization through joint packet size analysis of BSN and wifi networks. IEEE Trans. Parallel Distrib. Syst. 24:9 (2013), 1741–1751.
3. Chen, C.L.P., Zhang, C., Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275 (2014), 314–347 2014.
4. Wu, X., Zhu, X., Wu, G.-Q., Ding, W., Data mining with big data. IEEE Trans. Knowl. Data Eng. 26:1 (2014), 97–107.
5. Orgaz, G.B., Jung, J.J., Camacho, D., Social big data: recent achievements and new challenges. Inf. Fusion 28 (2016), 45–59.
6. Samuel, A., Sarfraz, M.I., Haseeb, H., Basalamah, S., Ghafoor, A., A framework for composition and enforcement of privacy-aware and context-driven authorization mechanism for multimedia big data. IEEE Trans. Multimedia 17:9 (2015), 1484–1494.
7. Saha, B., Srivastava, D., Data quality: the other face of big data. Proceedings of IEEE International Conference on Data Engineering, 2014, 1294–1297 IEEE.
8. Chen, X., Lin, X., Big data deep learning: challenges and perspectives. IEEE Access 2 (2014), 514–525.
9. Bengio, Y., Courville, A., Vincent, P., Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1798–1828.
10. Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521(7553) 436–444.
11. Schmidhuber, J., Deep learning in neural networks: an overview. Neural Netw. 61 (2015), 85–117.
12. Hinton, G.E., Salakhutdinov, R.R., Reducing the dimensionality of data with neural networks. Science 313:5786 (2006), 504–507.
13. Raina, R., Madhavan, A., Ng, A., Large-scale deep unsupervised learning using graphics processors. Proceedings of International Conference on Machine Learning, 2009, 873 C880 ACM.
14. V. Sze, Y. Chen, T. Yang, J. Emer, Efficient processing of deep neual networks: a tutoria and survey, 2017, arXiv:1703.09039.
15. Islam, S.M.S., Rahman, S., Rahman, M.M., Dey, E.K., Shoyaib, M., Application of deep learning to compuer vision: a comprehensive study. Proceedings of International Conference on Informatics, Electronics and Vision, 2016, 592–597 IEEE.
16. Kruger, N., Janssen, P., Kalkan, S., Lappe, M., Leonardis, A., Piater, J., Rodriguez-Sanchez, A.J., Wiskott, L., Deep hierarchies in the primate visual cortex: what can we learn for computer vision. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1847–1871.
17. Deng, L., Hinton, G., Kingsbury, B., New types of deep neual network learning for speech recognition and related applications: an overview. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, 26–31 IEEE.
18. Chen, D., Mak, B.K., Multitask learning of deep neural networks for low-resource speech recognition. IEEE/ACM Trans. Audio Speech Lang. Process. 23:7 (2015), 1172–1183.
19. Majumder, N., Poria, S., Gelbukh, A., Cambria, E., Deep learning-based document modeling for personality detection from text. IEEE Intell. Syst. 32:2 (2017), 74–79.
20. Jiang, Z., Li, L., Huang, D., Lin, L., Training word embeddings for deep learning in biomedical text mining tasks. Proceedings of IEEE International Conference on Bioinformatics and Biomedicine, 2015, 625–628 IEEE.
21. Khumoyun, A., Cui, Y., Hanku, L., Spark based distributed deep learning framework for big data applications. Proceedings of International Conference on Information Science and Communications Technologies, 2016, 1–5 IEEE.
22. Alsheikh, M.A., Niyato, D., Lin, S., Tan, H., Han, Z., Mobile big data analytics using deep learning and apache spark. IEEE Netw. 30:3 (2016), 22–29.
23. Wilamowski, B.M., Wu, B., Korniak, J., Big data and deep learning. Proceedings of IEEE Jubilee International Conference on Intelligent Engineering Systems, 2016, 11–16 IEEE.
24. Gehring, J., Miao, Y., Metze, F., Waibel, A., Extracting deep bottleneck features using stacked auto-encoders. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, 26–31 IEEE.
25. Weng, R., Lu, J., Tan, Y., Zhou, J., Learning cascaded deep auto-encoder networks for face alignment. IEEE Trans. Multimedia 18:10 (2016), 2066–2078.
26. Sun, M., Zhang, X., Hmme, H.V., Zheng, T.F., Unseen noise estimation using separable deep auto encoder for speech enhancement. IEEE/ACM Trans. Audio Speech Lang. Process. 24:1 (2016), 93–104.
27. Goodfellow, I., Le, Q., Saxe, A., Ng, A., Measuring invariances in deep networks. Proceeding of Neural Information and Processing System, 2009, 646–654 MIT.
28. A. Ng, Sparse autoencoder, CS294A Lecture notes 72(2011) 1–19.
29. G.E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18(7) 1527–1554.
30. Larochelle, H., Mandel, M., Pascanu, R., Bengio, Y., Learning algorithms for the classification restricted boltzmann machine. J. Mach. Learn. Res. 13 (2012), 643–669.
31. Salakhutdinov, R., Mnih, A., Hinton, G., Restricted boltzmann machines for collaborative filtering. Proceedings of International Conference on Machine learning, 2007, 791–798 ACM.
32. Cho, K., Ilin, A., Raiko, T., Improved learning of gaussian-bernoulli restricted boltzmann machines. Proceedings of International Conference in Artificial Neural Networks, 2011, 10C17 MIT.
33. Li, T., Zhang, J., Zhang, Y., Classification of hyperspectral image based on deep belief networks. Proceedings of IEEE International Conference on Image Processing, 2014, 5132–5136 IEEE.
34. Niu, J., Bu, X., Li, Z., Wang, Y., An improved bilinear deep belief network algorithm for image classification. Proceedings of International Conference on Computational Intelligence and Security, 2014, 189–192 IEEE.
35. Mohamed, A., Dahl, G.E., Hinton, G., Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20:1 (2012), 14–22.
36. Zhang, X., Wu, J., Deep belief networks based voice activity detection. IEEE Trans. Audio Speech Lang. Process. 21:4 (2013), 697–710.
37. Huang, W., Song, G., Hong, H., Xie, K., Deep architecture for traffic flow prediction: deep belief networks with multitask learning. IEEE Trans. Intell. Transp. Syst. 15:5 (2014), 2191–2201.
38. Sarikaya, R., Hinton, G.E., Deoras, A., Application of deep belief networks for natural language understanding. IEEE/ACM Trans. Audio Speech Lang. Process. 22:4 (2014), 778–784.
39. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Li, F., Large-scale video classification with convolutional neural networks. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2014, 1725–1732 IEEE.
40. Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P., Convolutional neural networks for large-scale remote-sensing image classification. IEEE Trans. Geosci. Remote Sens. 55:2 (2017), 645–657.
41. Han, Y., Kim, J., Lee, K., Deep convolutional neural networks for predominant instrument recognition in polyphonic music. IEEE/ACM Trans. Audio Speech Lang. Process. 25:1 (2017), 208–221.
42. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Proceedings of Advances in Neural Information Processing Systems, 2012, 1097–1105 MIT.
43. K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, 2014, arXiv:1409.1556.
44. R. Zhao, R. Yan, Z. Chen, K. Mao, P. Wang, R.X. Gao, Deep learning and its applications to machine health monitoring: a survey, 2016, arXiv:1612.07640.
45. N. Kalchbrenner, E. Grefenstette, P. Blunsom, A convolutional neural network for modeling sentences, 2014, arXiv:1404.2188.
46. Shi, H., Ushio, T., Endo, M., Yamagami, K., Horii, N., A multichannel convolutional neural network for cross-language dialog state tracking. Proceedings of IEEE Spoken Language Technology Workshop, 2016, 559–564 IEEE.
47. Abdel-Hamid, O., Mohamed, A., Jiang, H., Penn, G., Applying convolutional neural networks concepts to gybrid NN-HMM model for speech recognition. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2012, 4277–4280 IEEE.
48. Qian, Y., Bi, M., Tan, T., Yu, K., Very deep convolutional neural networks for noise robust speech recognition. IEEE/ACM Trans. Audio Speech Lang. Process. 24:12 (2016), 2263–2276.
49. Abdel-Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., Yu, D., Convolutional neural networks for speech recognition. IEEE/ACM Trans. Audio Speech Lang. Process. 22:10 (2014), 1533–1545.
50. Swietojanski, P., Ghoshal, A., Renals, S., Convolutional neural networks for distant speech recognition. IEEE Signal Process. Lett. 21:9 (2014), 1120–1124.
51. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Comput. 9:8 (1997), 1725–1780.
52. Gers, F.A., Schmidhuber, J., Cummins, F., Learning to torget: continual prediction with istm. Neural Comput. 12:10 (2000), 2451–2471.
53. Gers, F.A., Schraudolph, N.N., Schmidhuber, J., Learning precise timing with istm recurrent networks. J. Mach. Learn. Res. 3 (2002), 115–143.
54. J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014, 1412.3555.
55. Chen, X., Liu, X., Qian, Y., Gales, M.J.F., Woodland, P.C., CUED-RNNLM-an open-source toolkit for efficient training and evaluation of recurrent neural network language models. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2016, IEEE, 6000–6004.
56. Fu, T., Han, Y., Li, X., Liu, Y., Wu, X., Integrating prosodic information into recurrent neual network language model for speech recognition. Proceedings of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 2015, IEEE, 1194–1197.
57. Hermanto, A., Adji, T.B., Setiawan, N.A., Recurrent neural network language model for english-indonesian machine translation: Experimental study. Proceedings of International Conference on Science in Information Technology, 2015, IEEE, 132–136.
58. Zhang, X., Xie, G., Liu, C., Bengio, Y., End-to-end online writer identification with recurrent neural network. IEEE Trans. Hum. Mach. Syst. 47:2 (2017), 285–292.
59. Chien, J., Ku, Y., Bayesian recurrent neural network for language modeling. IEEE Trans. Neural Netw. Learn. Syst. 27:2 (2016), 361–374.
60. Deng, L., Yu, D., Platt, J., Scalable stacking and learning for building deep architectures. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2012, IEEE 2133 C2136.
61. Hutchinson, B., Deng, L., Yu, D., Tensor deep stacking networks. IEEE Trans. Pattern Anal. Mach. Intell. 35:8 (2013), 1944–1957.
62. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q.V., Ng, A.Y., Large scale distributed deep networks. Proceedings of Advances in Neural Information Processing Systems, 2012, MIT, 1223–1231.
63. Heigold, G., Vanhoucke, V., Senior, A., Nguyen, P., Ranzato, M., Devin, M., Dean, J., Multilingual acoustic models using distributed deep neural networks. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, IEEE, 8619–8623.
64. Heigold, G., McDermott, E., Vanhoucke, V., Senior, A., Bacchiani, M., Asynchronous stochastic optimization for sequence training of deep neural networks. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2014, IEEE, 5587–5591.
65. Vanhoucke, V., Senior, A., Mao, M., Improving the speed of neural networks on CPUs. Proceedings of Deep Learning and Unsupervised Feature Learning Workshop, 2011, MIT, 1–8.
66. Geman, S., Geman, D., Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. 6:6 (1984), 721–741.
67. Casella, G., George, E., Explaining the gibbs sampler. Am. Stat. 46:3 (1992), 167–174.
68. Coats, A., Huval, B., Wng, T., Wu, D., Wu, A., Deep learning with COTS HPC systems. J. Mach. Learn. Res. 28 (2013), 1337–1345.
69. Zhang, C., Li, P., Sun, G., Guan, Y., Xiao, B., Cong, J., Optimizing FPGA-based accelerator design for deep convolutional neural networks. Proceedings of ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, 2015, ACM, 161–170.
70. Wang, C., Gong, L., Yu, Q., Li, X., Xie, Y., Zhou, X., DLAU: a scalable deep learning accelerator unit on FPGA. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 36:3 (2017), 513–517.
71. Denil, M., Shakibi, B., Dinh, L., Ranzato, M., de Freitas, N., Predicting parameters in deep learning. Proceedings of Advances in Neural Information Processing Systems, 2013, MIT, 2148–2156.
72. Sainath, T.N., Kingsbury, B., Sindhwani, V., Arisoy, E., Ramabhadran, B., Low-rank matrix factorization for deep neural network training with high-dimensional output targets. Proceedings of IEEE International Conference of Acoustics, Speech, and Signal Processing, 2013, IEEE Pp. 6655 C6659.
73. Xue, J., Li, J., Gong, Y., Restructuring of deep neural network acoustic models with singular value decomposition. Proceedings of Conference of the International Speech Communication Association, 2013, Springer Pp. 2365 C2369.
74. Chen, W., Wilson, J.T., Tyree, S., Weinberger, K.Q., Chen, Y., Compressing neural networks with the hashing trick. Proceedings of International Conference on Machine Learning, 2015, ACM Pp. 2285 C2294.
75. Novikov, A., Podoprikhin, D., Osokin, A., Vetrov, D., Tensorizing neural netwroks. Proceedings of Advances in Neural Information Processing Systems, 2015, MIT, 442–450.
76. V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, V. Lempitsky, Speeding-up convolutional neural networks ssing fine-tuned CP-decomposition, 2014, 1412.6553.
77. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y., Multimodal deep learning. Proceedings of International Conference on Machine Learning, 2011, ACM, 689–696.
78. Srivastava, N., Salakhutdinov, R., Multimodal learning with deep boltzmann machines. Proceedings of Advances in Neural Information Processing Systems, 2012, MIT, 2222–2230.
79. Ouyang, W., Chu, X., Wang, X., Multi-source deep learning for human pose estimation. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2014, IEEE, 2337–2344.
80. Zhou, Y., Hu, Q., Liu, J., Jia, Y., Combining heterogeneous deep neural networks with conditional random fields for chinese dialogue act recognition. Neurocomputing 168 (2015), 408–417.
81. Zhao, L., Hu, Q., Wang, W., Heterogeneous feature selection with multi-modal deep neural networks and sparse group LASSO. IEEE Trans. Multimedia 17:11 (2015), 1936–1948.
82. Wang, A., Lu, J., Cai, J., Cham, T., Wang, G., Large-margin multi-modal deep learning for RGB-d object recognition. IEEE Trans. Multimedia 17:11 (2015), 1887–1898.
83. Neverova, N., Wolf, C., Taylor, G., Nebout, F., Moddrop: adaptive multi-modal gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38:8 (2016), 1692–1706.
84. Rastegar, S., Baghshah, M.S., Rabiee, H.R., Shojaee, S.M., MDL-CW: a multimodal deep learning framework with crossweights. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016, IEEE, 2601–2609.
85. Zhang, Q., Yang, L.T., Chen, Z., Deep computation model for unsupervised feature learning on big data. IEEE Trans. Serv. Comput. 9:1 (2016), 161–171.
86. Zhang, Q., Yang, L.T., Chen, Z., Privacy preserving deep computation model on cloud for big data feature learning. IEEE Trans. Comput. 65:5 (2016), 1351–1362.
87. Fu, L.M., Huang, H., Principe, J.C., Incremental backpropagation learning networks. IEEE Trans. Neural Netw. 7:3 (1996), 757–761.
88. Wan, S., Banta, L.E., Parameter incremental learning algorithm for neural networks. IEEE Trans. Neural Netw. 17:6 (2006), 1424–1438.
89. Shalev-Shwartz, S., Online learning and online convex optimization. Found. Trends Mach. Learn. 4:2 (2012), 107–194.
90. Zhao, L., Chen, J., Chen, F., Wang, W., Lu, C., Ramakrishnan, N., Simnest: social media nested epidemic simulation via online semi-supervised deep learning. Proceedings of IEEE International Conference on Data Mining, 2015, IEEE, 639–648.
91. Yu, L., Chen, H., Dou, Q., Qin, J., Heng, P.A., Integrating online and offline three-dimensional deep learning for automated polyp detection in colonoscopy videos. IEEE J. Biomed. Health Inf. 21:1 (2017), 65–75.
92. Bu, F., Chen, Z., Zhang, Q., Incremental updating method for big data feature learning. Comput. Eng. Appl. 51:12 (2015), 21–26.
93. Elwell, R., Polikar, R., Incremental learning in nonstationary environments with controlled forgetting. Proceedings of International Joint Conference on Neural Networks, 2009, IEEE, 771–778.
94. Chen, K., Huo, Q., Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2016, IEEE, 5880–5884.
95. Bizer, C., Boncz, P., Brodie, M.L., Erling, O., The meaningful use of big data: four perspectives – four challenges. ACM SIGMOD Rec. 40:4 (2012), 56–60.
96. Kwon, O., Lee, N., Shi, B., Data quality management, data usage experience and acquisition intention of big data analytics. Int. J. Inf. Manage. 34:3 (2014), 387–394.
97. Becker, D., King, T.D., McMullen, B., Big data, big data quality problem. Proceedings of IEEE International Conference on Big Data, 2015, IEEE, 2644–2653.
98. Ciancarini, P., Poggi, F., Russo, D., Big data quality: a roadmap for open data. Proceedings of IEEE International Conference on Big Data Computing Service and Applications, 2016, IEEE, 210–215.
99. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P., Extracting and composing robust features with denoising autoencoders. Proceedings of International Conference on Machine learning, 2008, ACM, 1096–1103.
100. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P., Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11 (2010), 3371–3408.
101. Bu, F., Chen, Z., Zhang, Q., Incomplete big data impputation algorithm based on deep learning. Microelectr. Comput. 31:12 (2014), 173–176.
102. R, W., Tao, D., Non-local auto-encoder with collaborative stabilization for image restoration. IEEE Trans. Image Process. 25:5 (2016), 2117–2129.