Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems

Neurocomputing

Volumn 216, Issue , 2016, Pages 718-734

  Liew, Shan Sung ^a   Khalil Hani, Mohamed ^a   Bakhteri, Rabia ^b  

^a UNIVERSITI TEKNOLOGI MALAYSIA   (Malaysia)

^b Machine Learning Developer Group   (Canada)

Author keywords

Activation function; Convolutional neural network; Deep neural network; Generalization performance; Output boundary; Training stability

Indexed keywords

CHEMICAL ACTIVATION; COMPUTATION THEORY; COMPUTATIONAL EFFICIENCY; CONVERGENCE OF NUMERICAL METHODS; CONVOLUTION; MEAN SQUARE ERROR; NEURAL NETWORKS; PATTERN RECOGNITION; STABILITY;

ACTIVATION FUNCTIONS; CONVOLUTIONAL NEURAL NETWORK; DEEP NEURAL NETWORKS; GENERALIZATION PERFORMANCE; OUTPUT BOUNDARY;

HYPERBOLIC FUNCTIONS;

ALGORITHM; ARTICLE; CONTROLLED STUDY; DATA BASE; DELAY DISCOUNTING; FACIAL EXPRESSION; HUMAN; LEARNING; LEARNING ALGORITHM; NERVE CELL NETWORK; PATTERN RECOGNITION; PERCEPTRON; PRIORITY JOURNAL; PROBABILITY; TASK PERFORMANCE; TRAINING; VISUAL DISCRIMINATION;

EID: 84994477344     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2016.08.037     Document Type: Article

References (64)

1. [1] Sodhi, S.S., Chandra, P., Bi-modal derivative activation function for sigmoidal feedforward networks. Neurocomputing 143:0 (2014), 182–196, 10.1016/j.neucom.2014.06.007.
2. [2] Hornik, K., Approximation capabilities of multilayer feedforward networks. Neural Netw. 4:2 (1991), 251–257, 10.1016/0893-6080(91)90009-T.
3. [3] I. Belič, Neural networks and modelling in vacuum science, Vacuum, 80(10), 2006, pp. 1107–1122. The World Energy Crisis: Some Vacuum based Solutions 〈http://dx.doi.org/10.1016/j.vacuum.2006.02.017〉.
4. [4] Wang, W., Yang, X., Ooi, B., Zhang, D., Zhuang, Y., Effective deep learning-based multi-modal retrieval. VLDB J., 2015, 1–23, 10.1007/s00778-015-0391-4.
5. [5] Sadrmomtazi, A., Sobhani, J., Mirgozar, M., Modeling compressive strength of {EPS} lightweight concrete using regression, neural network and {ANFIS}. Constr. Build. Mater. 42 (2013), 205–216, 10.1016/j.conbuildmat.2013.01.016.
6. [6] Wen, L., Yang, B., Cui, C., You, L., Zhao, M., Ultrasound-assisted extraction of phenolics from longan (Dimocarpus longan lour.) fruit seed with artificial neural network and their antioxidant activity. Food Anal. Methods 5:6 (2012), 1244–1251, 10.1007/s12161-012-9370-1.
7. [7] T. Desell, S. Clachar, J. Higgins, B. Wild, Evolving neural network weights for time-series prediction of general aviation flight data, in: Parallel Problem Solving from Nature, Vol. 8672 of Lecture Notes in Computer Science, Springer International Publishing, 2014, pp. 771–781.
8. [8] Y. Ioannou, D.P. Robertson, J. Shotton, R. Cipolla, A. Criminisi, Training cnns with low-rank filters for efficient image classification, CoRR abs/1511.06744.
9. [9] Mamalet, F., Roux, S., Garcia, C., Real-time video convolutional face finder on embedded platforms. EURASIP J. Embed. Syst. 2007:1 (2007), 1–8, 10.1155/2007/21724.
10. [10] Li, J.-C., Ng, W., Yeung, D., Chan, P., Bi-firing deep neural networks. Int. J. Mach. Learn. Cybern. 5:1 (2014), 73–83, 10.1007/s13042-013-0198-9.
11. [11] X. Liu, S. Li, M. Kan, J. Zhang, S. Wu, W. Liu, H. Han, S. Shan, X. Chen, Agenet: deeply learned regressor and classifier for robust apparent age estimation, 2015.
12. [12] L. Zhang, L. Lin, X. Wu, S. Ding, L. Zhang, End-to-end photo-sketch generation via fully convolutional representation learning, CoRR abs/1501.07180.
13. [13] P.H.O. Pinheiro, R. Collobert, Weakly supervised object segmentation with convolutional neural networks, Idiap-RR Idiap-RR-13-2014, Idiap (8 2014).
14. [14] Tompson, J.J., Jain, A., LeCun, Y., Bregler, C., Joint training of a convolutional network and a graphical model for human pose estimation. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K., (eds.) Advances in Neural Information Processing Systems, 27, 2014, Curran Associates, Inc., United States, 1799–1807.
15. [15] I.J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, V.D. Shet, Multi-digit number recognition from street view imagery using deep convolutional neural networks, CoRR abs/1312.6082.
16. [16] Y. Miao, M. Gowayyed, F. Metze, EESEN: end-to-end speech recognition using deep RNN models and wfst-based decoding, CoRR abs/1507.08240.
17. [17] A. Graves, Generating sequences with recurrent neural networks, CoRR abs/1308.0850.
18. [18] K. Kurach, M. Andrychowicz, I. Sutskever, Neural random-access machines, CoRR abs/1511.06392.
19. [19] Bian, W., Chen, X., Neural network for nonsmooth, nonconvex constrained minimization via smooth approximation. IEEE Trans. Neural Netw. Learn. Syst. 25:3 (2014), 545–556, 10.1109/TNNLS.2013.2278427.
20. [20] Jain, S., Singh, S., Low-order dominant harmonic estimation using adaptive wavelet neural network. IEEE Trans. Ind. Electron. 61:1 (2014), 428–435, 10.1109/TIE.2013.2242414.
21. [21] N. Wang, J. Melchior, L. Wiskott, Gaussian-binary restricted boltzmann machines on modeling natural image statistics, CoRR abs/1401.5900.
22. [22] Jain, S., Singh, S., Fast harmonic estimation of stationary and time-varying signals using ea-awnn. IEEE Trans. Instrum. Meas. 62:2 (2013), 335–343, 10.1109/TIM.2012.2217637.
23. [23] Nambiar, V.P., Hani, M.K., Sahnoun, R., Marsono, M.N., Hardware implementation of evolvable block-based neural networks utilizing a cost efficient sigmoid-like activation function. Neurocomputing 140 (2014), 228–241, 10.1016/j.neucom.2014.03.018.
24. [24] M. Puheim, L. Nyulaszi, L. Madarasz, V. Gaspar, On practical constraints of approximation using neural networks on current digital computers, in: Proceedings of the 2014 18th International Conference onIntelligent Engineering Systems (INES), 2014, pp. 257–262. 〈 http://dx.doi.org/10.1109/INES.2014.6909379〉.
25. [25] Hornik, K., Stinchcombe, M., White, H., Multilayer feedforward networks are universal approximators. Neural Netw. 2:5 (1989), 359–366, 10.1016/0893–6080(89)90020-8.
26. [26] X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: G.J. Gordon, D.B. Dunson (Eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS-11), Vol. 15, J. Mach. Learn. Res. Workshop and Conference Proceedings, 2011, pp. 315–323.
27. [27] Wen, C., Ma, X., A max-piecewise-linear neural network for function approximation. Neurocomputing 71:4 (2008), 843–852.
28. [28] P. Chandra, U. Ghose, A. Sood, A non-sigmoidal activation function for feedforward artificial neural networks, in: Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN), 2015, pp. 1–8 〈 http://dx.doi.org/10.1109/IJCNN.2015.7280440〉.
29. [29] Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, in: Proceedings of the IEEE, 86(11), 1998, pp. 2278–2324 〈 http://dx.doi.org/10.1109/5.726791〉.
30. [30] J.A. Bonnell, Implementation of a new sigmoid function in backpropagation neural networks, Master's thesis, East Tennessee State University (8 2011).
31. [31] da, G., Gomes, S., Ludermir, T., Lima, L., Comparison of new activation functions in neural network for forecasting financial time series. Neural Comput. Appl. 20:3 (2011), 417–439, 10.1007/s00521-010-0407-3.
32. [32] P. Chandra, S. Sodhi, A skewed derivative activation function for sffanns, in: Recent Advances and Innovations in Engineering (ICRAIE), 2014, 2014, pp. 1–6. http://dx.doi.org/10.1109/ICRAIE.2014.6909324.
33. [33] Chandra, P., Singh, Y., A case for the self-adaptation of activation functions in {FFANNs}. Neurocomputing 56 (2004), 447–454, 10.1016/j.neucom.2003.08.005.
34. [34] P. Chandra, Y. Singh, An activation function adapting training algorithm for sigmoidal feedforward networks, Neurocomputing 61 (2004) 429 – 437, hybrid Neurocomputing: Selected Papers from the 2nd International Conference on Hybrid Intelligent Systems. http://dx.doi.org/10.1016/j.neucom.2004.04.001.
35. [35] Singh, Y., Chandra, P., A class +1 sigmoidal activation functions for ffanns. J. Econ. Dyn. Control 28:1 (2003), 183–187.
36. [36] van den Bout, D., Franzon, P., Paulos, J., Miller, T., Snyder, W., Nagle, T., Liu, W., Scalable vlsi implementations for neural networks. J. VLSI Signal Process. Syst. Signal, Image Video Technol. 1:4 (1990), 367–385, 10.1007/BF00929928.
37. [37] X. Nie J. Cao S. Fei Multistability and instability of delayed competitive neural networks with nondecreasing piecewise linear activation functions Neurocomputing 119 (2013) 281–291, 10.1016/j.neucom.2013.03.030. (intelligent Processing Techniques for Semantic-based Image and Video Retrieval).
38. [38] Yam, J.Y., Chow, T.W., A weight initialization method for improving training speed in feedforward neural network. Neurocomputing 30:1 (2000), 219–232.
39. [39] Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Pereira, F., Burges, C., Bottou, L., Weinberger, K., (eds.) Advances in Neural Information Processing Systems 25, 2012, Curran Associates, Inc., 1097–1105.
40. [40] A.L. Maas, A.Y. Hannun, A.Y. Ng, Rectifier nonlinearities improve neural network acoustic models, in: Proceedings of ICML, vol. 30, 2013.
41. [41] S.G. andAnkur Agrawal, K. Gopalakrishnan, P. Narayanan, Deep learning with limited numerical precision, CoRR abs/1502.02551.
42. [42] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I.J. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, Y. Bengio, Theano: new features and speed improvements, CoRR abs/1211.5590.
43. [43] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in: Proceedings of the 22nd ACM International Conference on Multimedia, MM'14, ACM, New York, NY, USA, 2014, pp. 675–678. 〈 http://dx.doi.org/10.1145/2647868.2654889〉.
44. [44] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, E. Shelhamer, cudnn: Efficient primitives for deep learning, CoRR abs/1410.0759.
45. [45] M. Peemen, R. Shi, S. Lal, B. Juurlink, B. Mesman, H. Corporaal, The neuro vector engine: Flexibility to improve convolutional net efficiency for wearable vision, in: Proceedings of the 2016 Design, Automation Test in Europe Conference Exhibition (DATE), 2016, pp. 1604–1609.
46. [46] Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, O. Temam, Shidiannao: Shifting vision processing closer to the sensor, in: Proceedings of the 42nd Annual International Symposium on Computer Architecture, ISCA'15, ACM, New York, NY, USA, 2015, pp. 92–104. 〈 http://dx.doi.org/10.1145/2749469.2750389〉.
47. [47] D. Liu, T. Chen, S. Liu, J. Zhou, S. Zhou, O. Teman, X. Feng, X. Zhou, Y. Chen, Pudiannao: A polyvalent machine learning accelerator, in: Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'15, ACM, New York, NY, USA, 2015, pp. 369–381. 〈 http://dx.doi.org/10.1145/2694344.2694358〉.
48. [48] Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, O. Temam, Dadiannao: A machine-learning supercomputer, in: Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO-47, IEEE Computer Society, Washington, DC, USA, 2014, pp. 609–622. 〈 http://dx.doi.org/10.1109/MICRO.2014.58〉.
49. [49] T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, O. Temam, Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning, in: Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'14, ACM, New York, NY, USA, 2014, pp. 269–284. 〈 http://dx.doi.org/10.1145/2541940.2541967〉.
50. [50] T. Moreau, J. Nelson, A. Sampson, H. Esmaeilzadeh, L. Ceze, Approximate computing on programmable socs via neural acceleration, Technical report, 2014.
51. [51] Ma, J., Sheridan, R.P., Liaw, A., Dahl, G.E., Svetnik, V., Deep neural nets as a method for quantitative structure-activity relationships. J. Chem. Inf. Model. 55:2 (2015), 263–274.
52. [52] X. Zhang, J. Trmal, D. Povey, S. Khudanpur, Improving deep neural network acoustic models using generalized maxout networks, in: Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 215–219. http://dx.doi.org/10.1109/ICASSP.2014.6853589〉.
53. [53] H. Larochelle, D. Erhan, A. Courville, J. Bergstra, Y. Bengio, An empirical evaluation of deep architectures on problems with many factors of variation, in: Proceedings of the 24th International Conference on Machine Learning, ICML ’07, ACM, New York, NY, USA, 2007, pp. 473–480. 〈 http://dx.doi.org/10.1145/1273496.1273556〉.
54. [54] Martinez, A.M., Kak, A., Pca versus lda. IEEE Trans. Pattern Anal. Mach. Intell. 23:2 (2001), 228–233, 10.1109/34.908974.
55. [55] S. Chopra, R. Hadsell, Y. LeCun, Learning a similarity metric discriminatively, with application to face verification, in: Proceeedings of the Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, 2005, pp. 539–546 . 〈 http://dx.doi.org/10.1109/CVPR.2005.202〉.
56. [56] Liew, S.S., Khalil-Hani, M., Syafeeza, A., Bakhteri, R., Gender classification: a convolutional neural network approach. Turk. J. Elec. Eng. 24:3 (2016), 1248–1264.
57. [57] Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., Lew, M.S., Deep learning for visual understanding: a review. Neurocomputing, 2015, 10.1016/j.neucom.2015.09.116.
58. [58] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS) 9 (2010) 249–256. 〈 http://dx.doi.org/10.1.1/207.2059〉.
59. [59] M. Milakov, Convolutional Neural Networks in Galaxy Zoo Challenge, April 2014, pp. 1–7.
60. [60] Dong, Y., Liu, Y., Lian, S., Automatic age estimation based on deep learning algorithm. Neurocomputing, 2015, 10.1016/j.neucom.2015.09.115.
61. [61] Y. Taigman, M. Yang, M. Ranzato, L. Wolf, Deepface: Closing the gap to human-level performance in face verification, in: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1701–1708. 〈 http://dx.doi.org/10.1109/CVPR.2014.220〉.
62. [62] Negnevitsky, M., Artificial Intelligence: A Guide to Intelligent Systems. 1st ed., 2001, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
63. [63] S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32nd International Conference on Machine Learning, 2015, pp. 448–456.
64. [64] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: surpassing human-level performance on imagenet classification, CoRR abs/1502.01852.