Review of Convolutional Neural Network

Jisuanji Xuebao/Chinese Journal of Computers

Volumn 40, Issue 6, 2017, Pages 1229-1251

  Zhou, Fei Yan ^a,b   Jin, Lin Peng ^a,b   Dong, Jun ^a  

^a SUZHOU INSTITUTE OF NANO TECH AND NANO BIONICS   (China)

^b UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

Convolutional neural network; Deep learning; Domain data; Network structure; Training method

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION; CONVOLUTION; DEEP LEARNING; DEEP NEURAL NETWORKS; FACE RECOGNITION; FAULT TOLERANCE; FORMAL LANGUAGES; LEARNING ALGORITHMS; LEARNING SYSTEMS; LINEAR TRANSFORMATIONS; MATHEMATICAL TRANSFORMATIONS; NATURAL LANGUAGE PROCESSING SYSTEMS; NEURAL NETWORKS; OBJECT DETECTION; OBJECT RECOGNITION; RECURRENT NEURAL NETWORKS; SCALES (WEIGHING INSTRUMENTS); SEMANTICS; SPEECH RECOGNITION; UNSUPERVISED LEARNING;

CONVOLUTIONAL NEURAL NETWORK; DOMAIN DATA; FULLY CONNECTED NEURAL NETWORK; MACHINE LEARNING APPROACHES; NETWORK STRUCTURES; SIGNAL AND INFORMATION PROCESSING; TRAINING METHODS; UNSUPERVISED LEARNING METHOD;

NETWORK ARCHITECTURE;

EID: 85029554489     PISSN: 02544164     EISSN: None     Source Type: Journal    
DOI: 10.11897/SP.J.1016.2017.01229     Document Type: Review

References (107)

1. McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 1943, 5(4): 115-133.
2. Rosenblatt F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 1958, 65(6): 386-408.
3. Rosenblatt F. Principles of Neurodinamics: Preceptron and Theory of Brain Mechanisms. Washington, USA: Spartan Books, 1962.
4. Rumelhart D E, Hinton G, Williams R J. Learning representations by back-propagating errors. Nature, 1986, 323(6088): 533-536.
5. Cortes C, Vapnik V. Support vector networks. Machine Learning, 1995, 20(3): 273-297.
6. Hinton G, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786): 504-507.
7. Yu Kai, Jia Lei, Chen Yu-Qiang, et al. Deep learning: Yesterday, today, and tomorrow. Journal of Computer Research and Development, 2013, 50(9): 1799-1804 (in Chinese).
8. Bengio Y, Lamblin P, Popovici D, et al. Greedy layer-wise training of deep networks// Proceedings of the 2007 Advances in Neural Information Processing Systems. Vancouver, Canada, 2007: 153-160.
9. Ranzato M A, Poultney C, Chopra S, et al. Efficient learning of sparse representations with an energy-based model// Proceedings of the 2007 Advances in Neural Information Processing Systems. Vancouver, Canada, 2007: 1137-1144.
10. Erhan D, Bengio Y, Courville A, et al. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 2010, 11(3): 625-660.
11. Lecun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436-444.
12. Bengio Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2009, 2(1): 1-127.
13. Hinton G, Osindero S, Teh Y-W. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18(7): 1527-1554.
14. Salakhutdinov R, Hinton G. Deep Boltzmann machines. Journal of Machine Learning Research-Proceedings Track, 2009, 9(1): 448-455.
15. Salakhutdinov R, Hinton G. An efficient learning procedure for deep Boltzmann machines. Neural Computation, 2012, 24(8): 1967-2006.
16. Liu Jian-Wei, Liu Yuan, Luo Xiong-Lin. Research and development on Boltzmann machine. Journal of Computer Research and Development, 2014, 51(1): 1-16 (in Chinese).
17. Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders// Proceedings of the 25th international Conference on Machine Learning. Helsinki, Finland, 2008: 1096-1103.
18. Vincent P, Larochelle H, Bengio Y, et al. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 2010, 11(12): 3371-3408.
19. LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
20. LeCun Y, Boser B, Denker J S, et al. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989, 11(4): 541-551.
21. David S, Huang A, Maddidon C J. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529(7587): 484-489.
22. Lawrence S, Giles C L, Tsoi A C, et al. Face recognition: A convolutional neural-network approach. IEEE Transactions on Neural Networks, 1997, 8(1): 98-113.
23. Neubauer C. Evaluation of convolutional neural networks for visual recognition. IEEE Transactions on Neural Networks, 1998, 9(4): 685-696.
24. Krizhevsky A, Sutskever II, Hinton G. Imagenet classification with deep convolutional neural networks// Proceedings of the Advances in Neural Information Processing Systems. Lake Tahoe, USA, 2012: 1097-1105.
25. Nair V, Hinton G E, Farabet C. Rectified linear units improve restricted Boltzmann machines// Proceedings of the 27th International Conference on Machine Learning. Haifa, Israel, 2010: 807-814.
26. Hagan M T, Demuth H B, Beale M H. Neural Network Design. Translated by Dai Kui, Beijing: China Machine Press, 2002 (in Chinese).
27. Kiranyaz S, Ince T, Gabbouj M. Real-time patient-specific ECG classification by 1D convolutional neural networks. IEEE Transactions on Biomedical Engineering, 2016, 63(3): 664-675.
28. Hubel D H, Wiesel T N. Receptive fields binocular interaction, and functional architecture in the cat's visual cortex. Journal of Physiology, 1962, 160(1): 106-154.
29. Fukushima K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 1980, 36(4): 193-202.
30. Yoo H-J. Deep convolution neural networks in computer vision: A review. IEIE Transactions on Smart Processing and Computing, 2015, 4(1): 35-43.
31. Gao Li-Gang, Chen Pai-Yu, Yu Shi-Meng. Demonstration of convolution kernel operation on resistive cross-point array. IEEE Electron Device Letters, 2016, 37(7): 870-873.
32. Jin Lin-Peng. Study on Approach of Electrocardiogram Classification for Clinical Application[Ph.D. dissertation]. Suzhou Institute of Nano-tech and Nano-bionics, Chinese Academy of Sciences, Suzhou, 2016 (in Chinese).
33. Xu Bing, Wang Nai-Yan, Chen Tian-Qi, et al. Empirical evaluation of rectified activations in convolution network. arXiv:1505.00853v2, 2015.
34. Jarrett K, Kavukcuoglu K, Marc'Aurelio Ranzato, et al. What is the best multi-stage architecture for object recognition?// Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. Kyoto, Japan, 2009: 2146-2153.
35. Bengio Y, Courville A, Vincent P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1798-1828.
36. He Kai-Ming, Sun Jian. Convolutional neural networks at constrained time cost// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA, 2015: 5353-5360.
37. Gu Jiu-Xiang, Wang Zhen-Hua, Jason Kuen, et al. Recent advances in convolutional neural networks. arXiv:1512.07108v5, 2017.
38. Boureau Y-L, Roux N L, Bach F, et al. Ask the locals: Multi-way local pooling for image recognition// Proceedings of the 2011 International Conference on Computer Vision. Barcelona, Spain, 2011: 2651-2658.
39. Zeiler M D, Fergus R. Stochastic pooling for regularization of deep convolutional neural networks. arXiv:1301.3557v1, 2013.
40. Boureau Y-L, Ponce J, LeCun Y. A theoretical analysis of feature pooling in visual recognition. International Conference on Machine Learning, 2010, 32(4): 111-118.
41. Boureau Y-L, Bach F, LeCun Y, et al. Learning mid-level features for recognition// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, USA, 2010: 2559-2566.
42. Sainath T N, Mohamed A, Kingsbury B, et al. Deep convolutional neural networks for LVCSR// Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Vancouver, Canada, 2013: 8614-8618.
43. O'Shea K, Nash R. An introduction to convolutional neural networks. arXiv:1511.08458v2, 2015.
44. Srivastava N, Hinton G, Krizhevsky A, et al. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014, 15(6): 1929-1958.
45. Sainatha T N, Kingsburya B, Saona G, et al. Deep convolutional neural networks for large-scale speech tasks. Neural Networks, 2015, 64(Special Issue): 39-48.
46. Chu J L, Krzyzak A. Analysis of feature maps selection in supervised learning using convolutional neural networks// Proceedings of the 27th Canadian Conference on Artificial Intelligence. Montreal, Canada, 2014: 59-70.
47. Cao K, Jain A K. Latent orientation field estimation via convolutional neural network// Proceedings of the 2015 International Conference on Biometrics. Phuket, Thailand, 2015: 349-356.
48. Jin Lin-Peng, Dong Jun. Deep learning research on clinical electrocardiogram analysis. Science China: Information Sciences, 2015, 45(3): 398-416 (in Chinese).
49. Huang Jui-Ting, Li Jin-Yu, Gong Yi-Fan. An analysis of convolutional neural networks for speech recognition// Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). South Brisbane, Australia, 2015: 4989-4993.
50. Lin Min, Chen Qiang, Yan Shui-Cheng. Network in network. arXiv:1312.4400v3, 2013.
51. Xu Chun-Yan, Lu Can-Yi, Liang Xiao-Dan, et al. Multi-loss regularized deep neural network. IEEE Transactions on Circuits and Systems For Video Technology, 2015, 26(12): 2273-2283.
52. Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks. arXiv:1506.02025v3, 2016.
53. Zeiler M D, Krishnan D, Taylor G W, et al. Deconvolutional networks// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, USA, 2010: 2528-2535.
54. Zeiler M D. Visualizing and understanding convolutional networks// Proceedings of the 13th European Conference on Computer Vision (ECCV). Zurich, Switzerland, 2014: 818-833.
55. Zhao Jun-Bo, Mathieu M, Goroshin R, et al. Stacked what-where auto-encoders. arXiv:1506.02351v8, 2016.
56. Jiang Zong-Li. Introduction to Aritificial Neural Networks. Beijing: Higher Education Press, 2001 (in Chinese).
57. Fausett L. Fundamentals of Neural Networks: Architectures, Algorithms, and Applications. London: Prentice-Hall, 1994.
58. Ngiam J, Koh P W, Chen Z, et al. Sparse filtering// Proceedings of the Advances in Neural Information Processing Systems 24 (NIPS 2011). Granada, Spain, 2011: 1125-1133.
59. Dong Zhen, Pei Ming-Tao, He Yang, et al. Vehicle type classification using unsupervised convolutional neural network// Proceedings of the 22nd International Conference on Pattern Recognition. Stockholm, Sweden, 2014: 172-177.
60. Dong Zhen, Wu Yu-Wei, Pei Ming-Tao, et al. Vehicle type classification using a semisupervised convolutional neural network. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(4): 2247-2256.
61. Jin Lin-Peng, Dong Jun. Ensemble deep learning for biomedical time series classification. Computational Intelligence and Neuroscience, 2016, 2016(3): 1-13.
62. Jia Yang-Qing, Shelhamer E, Donahue J, et al. Caffe: Convolutional architecture for fast feature embedding// Proceedings of the ACM International Conference on Multimedia. Orlando, USA, 2014: 675-678.
63. Al-Rfou R, Alain G, Almahairi A, et al. Theano: A python framework for fast computation of mathematical expressions. arXiv:1605.02688v1, 2016.
64. Bahrampour S, Ramakrishnan N, Schott L, et al. Comparative study of deep learning software frameworks. arXiv:1511.06435v3, 2016.
65. Szegedy C, Liu Wei, Jia Yang-Qing, et al. Going deeper with convolutions// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA, 2015: 1-9.
66. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6, 2014.
67. He Kai-Ming, Zhang Xiang-Yu, Ren Shao-Qing, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1915.
68. Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.
69. He Kai-Ming, Zhang Xiang-Yu, Ren Shao-Qing, et al. Deep residual learning for image recognition// Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition. Las Vegas, USA, 2016: 770-778.
70. Wang Xiao-Gang. Deep learning in image recognition. Communications of the CCF, 2015, 11(8): 15-23 (in Chinese).
71. Mishkin D, Sergievskiy N, Matas J. Systematic evaluation of CNN advances on the imageNet. arXiv:1606.02228v2, 2016.
72. Taigman Y, Yang M, Ranzato M A, et al. DeepFace: Closing the gap to human-level performance in face verification// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Columbus, USA, 2014: 1701-1708.
73. Sun Yi, Wang Xiao-Gang, Tang Xiao-Ou. Deep learning face representation from predicting 10000 classes// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA, 2014: 1891-1898.
74. Sun Yi, Chen Yu-Heng, Wang Xiao-Gang, et al. Deep learning face representation by joint identification-verification// Proceedings of the Advances in Neural Information Processing Systems. Montreal, Canada, 2014: 1988-1996.
75. Sun Yi, Wang Xiao-Gang, Tang Xiao-Ou. Deeply learned face representations are sparse, selective, and Robust// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA, 2015: 2892-2900.
76. Sun Yi, Liang Ding, Wang Xiao-Gang, et al. DeepID3: Face recognition with very deep neural networks. arXiv:1502.00873v1, 2015.
77. Schroff F, Kalenichenko D, Philbin J. FaceNet: A unified embedding for face recognition and clustering// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA, 2015: 815-823.
78. Parkhi O M, Vedaldi A, Zisserman A. Deep face recognition// Proceedings of the British Machine Vision Conference (BMVC 2015). Swansea, England, 2015: 1-12.
79. Chang Liang, Deng Xiao-Ming, Zhou Ming-Quan, et al. Convolutional neural networks in image understanding. Acta Automatica Sinica, 2016, 42(9): 1300-1312 (in Chinese).
80. Abdel-Hamid Ossama, Mohamed Abdel-rahman, Jiang Hui, et al. Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition// Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Kyoto, Japan, 2012: 4277-4280.
81. Abdel-Hamid Ossama, Mohamed Abdel-rahman, Jiang Hui, et al. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014, 22(10): 1533-1545.
82. Thomas S, Ganapathy S, Saon G, et al. Analyzing convolutional neural networks for speech activity detection in mismatched acoustic conditions// Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Florence, Italy, 2014: 2519-2523.
83. Hinton G, Deng Li, Yu Dong, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012, 29(6): 82-97.
84. Huang Jui-Ting, Li Jin-Yu, Gong Yi-Fan. An analysis of convolutional neural networks for speech recognition// Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brisbane, Australia, 2015: 4989-4993.
85. Dong Jun, Zhang Jia-Wei, Zhu Hong-Hai, et al. Wearable ECG monitors and its remote diagnosis service platform. IEEE Intelligent Systems, 2012, 27(6): 36-43.
86. Kadi I, Idri A, Fernandez-Aleman J L. Knowledge discovery in cardiology: A systematic literature review. International Journal of Medical Informatics, 2017, 97: 12-32.
87. Zhu Hong-Hai. Key Algorithms on Computer-Aided Electrocardiogram Analysis and Development of Remote Multi-Signs Monitoring System[Ph.D. dissertation]. Suzhou Institute of Nano-tech and Nano-bionics, Chinese Academy of Sciences, Suzhou, 2013 (in Chinese).
88. Zhang Jia-Wei, Liu Xia, Dong Jun. CCDD: An enhanced standard ECG database with its management and annotation tools. International Journal on Article Intelligence Tools, 2012, 21(5): 1-26.
89. Fayyaz-ul-Afsar Amir Minhas, Muhammad Arif. Robust electrocardiogram beat classification using discrete wavelet transform. Physiological Measurement, 2008, 29(5): 555-570.
90. Martis R J, Chakraborty C, Ray A K. A two-stage mechanism for registration and classification of ECG using Gaussian mixture model. Pattern Recognition, 2009, 42(11): 2979-2988.
91. Hakacova N, Tragardh-Johansson E, Wagner G S, et al. Computer-based rhythm diagnosis and its possible influence on nonexpert electrocardiogram readers. Journal of Electrocardiology, 2012, 45(1): 18-22.
92. Vogl T P, Mangis J K, Rigler A K, et al. Accelerating the convergence of the back-propagation method. Biological Cybernetics, 1988, 59(4): 257-263.
93. Wang Li-Ping. Study on Approach of ECG Classification with Domain Knowledge[Ph.D. dissertation]. East China Normal University, Shanghai, 2012 (in Chinese).
94. Zhou Fei-Yan, Jin Lin-Peng, Dong Jun. PVC recognition algorithm based on ensemble learning. Acta Electronica Sinica, 2017, 45(2): 501-507 (in Chinese).
95. Zheng Yi, Liu Qi, Chen En-Hong, et al. Time series classification using multi-channels deep convolutional neural networks// Proceedings of the 15th International Conference on Web-Age Information Management (WAIM). Macau, China, 2014: 298-310.
96. Zheng Yi, Liu Qi, Chen En-Hong, et al. Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Frontiers of Computer Science, 2015, 10(1): 96-112.
97. Cui Zhi-Cheng, Chen Wen-Lin, Chen Yi-Xin. Multi-scale convolutional neural networks for time series classification. arXiv:1603.06995, 2016.
98. Zheng Yi, Liu Qi, Chen En-Hong, et al. Convolutional nonlinear neighbourhood components analysis for time series classification// Proceedings of the 19th Pacific-Asia Conference on Knowledge Discovery and Data Mining. Ho Chi Minh, Vietnam, 2015: 534-546.
99. Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection. arXiv:1506.02640v5, 2015.
100. Ren Shao-Qing, He Kai-Ming, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, doi:10.1109/TPAMI.2016.2577031.
101. Girshick R. Fast R-CNN// Proceedings of the 2015 IEEE International Conference on Computer Vision. Santiago, Chile, 2015: 1440-1448.
102. Liu Wei, Anguelov D, Erhan D, et al. SSD: Single shot multibox detector// Proceedings of the 14th European Conference on Computer Vision. Amsterdam, Netherlands, 2016: 21-37.
103. Xu Jia-Ming, Wang Peng, Tian Guan-Hua, et al. Short text clustering via convolutional neural networks// Proceedings of the NAACL-HLT 2015. Denver, USA, 2015: 62-69.
104. Gao Jun-Yu, Yang Xiao-Shan, Zhang Tian-Zhu, et al. Robust visual tracking method via deep learning. Chinese Journal of Computers, 2016, 39(7): 1419-1434 (in Chinese).
105. Li Hong, Liu Fang, Yang Shu-Yuan, et al. Remote sensing image fusion based on deep support value learning networks. Chinese Journal of Computers, 2016, 39(8): 1583-1596 (in Chinese).
106. Hu Xiao-Juan. The Research on Signal Perception and Computer Aided Recognition of Traditional Chinese Medicine Pulse Diagnosis[Ph.D. dissertation]. East China Normal University, Shanghai, 2013 (in Chinese).
107. Guo Yan-Ming, Liu Yu, Ard Oerlemans, et al. Deep learning for visual understanding: A review. Neurocomputing, 2016, 187(Special Issue): 27-48.