Deep learning with S-shaped rectified linear activation units

30th AAAI Conference on Artificial Intelligence, AAAI 2016

Volumn , Issue , 2016, Pages 1737-1743

  Jin, Xiaojie ^a   Xu, Chunyan ^c   Feng, Jiashi ^a   Wei, Yunchao ^a   Xiong, Junjun ^b   Yan, Shuicheng ^a  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^b Beijing Samsung Telecom R and D Center   (China)

^c NANJING UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BACKPROPAGATION; FUNCTIONS; PIECEWISE LINEAR TECHNIQUES;

ACTIVATION FUNCTIONS; COMPUTATION COSTS; CONVOLUTIONAL NETWORKS; DIFFERENT LAYERS; MULTIPLE FUNCTION; NONCONVEX FUNCTIONS; PIECE-WISE LINEAR FUNCTIONS; STATE OF THE ART;

CHEMICAL ACTIVATION;

EID: 85007251559     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Agostinelli, F.; Hoffman, M.; Sadowski, P. J.; and Baldi, P. 2014. Learning activation functions to improve deep neural networks. CoRR abs/1412.6830.
2. Cireşan, D. C.; Meier, U.; Gambardella, L. M.; and Schmidhuber, J. 2011. Convolutional neural network committees for handwritten character classification. In Document Analysis and Recognition (ICDAR), 2011 International Conference on, 1135-1139. IEEE.
3. Dayan, P., and Abbott, L. F. 2001. Theoretical neuroscience, volume 806. Cambridge, MA: MIT Press.
4. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei- Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 248-255. IEEE.
5. Fechner, G. 1965. Elements of psychophysics.
6. Girshick, R. B.; Donahue, J.; Darrell, T.; and Malik, J. 2013. Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR abs/1311.2524.
7. Goodfellow, I. J.; Warde-Farley, D.; Mirza, M.; Courville, A.; and Bengio, Y. 2013. Maxout networks. arXiv preprint arXiv:1302.4389.
8. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR abs/1502.01852.
9. Jia, Y.; Shelhamer, E.; Donahue, J.; Karayev, S.; Long, J.; Girshick, R. B.; Guadarrama, S.; and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. CoRR abs/1408.5093.
10. Johnson, K. O.; Hsiao, S. S.; and Yoshioka, T. 2002. Book review: neural coding and the basic law of psychophysics. The Neuroscientist 8(2):111-121.
11. Krizhevsky, A., and Hinton, G. 2009. Learning multiple layers of features from tiny images.
12. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097-1105.
13. LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278-2324.
14. Lee, C.-Y.; Xie, S.; Gallagher, P.; Zhang, Z.; and Tu, Z. 2014. Deeply-supervised nets. arXiv preprint arXiv:1409.5185.
15. Lin, M.; Chen, Q.; and Yan, S. 2013. Network in network. CoRR abs/1312.4400.
16. Maas, A. L.; Hannun, A. Y.; and Ng, A. Y. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML, volume 30.
17. Nair, V., and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), 807-814.
18. Nieder, A. 2005. Counting on neurons: The neurobiology of numerical competence. Nature Reviews Neuroscience 6(3):177-190.
19. Randall, D.; Burggren, W. W.; French, K.; and Eckert, R. 2002. Eckert animal physiology. Macmillan.
20. Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 1-42.
21. Springenberg, J. T., and Riedmiller, M. 2013. Improving deep neural networks with probabilistic maxout units. arXiv preprint arXiv:1312.6116.
22. Srivastava, N., and Salakhutdinov, R. R. 2013. Discriminative transfer learning with tree-based priors. In Burges, C.; Bottou, L.; Welling, M.; Ghahramani, Z.; and Weinberger, K., eds., Advances in Neural Information Processing Systems 26. Curran Associates, Inc. 2094-2102.
23. Stevens, S. S. 1957. On the psychophysical law. Psychological review 64(3):153.
24. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2014. Going deeper with convolutions. arXiv preprint arXiv:1409.4842.
25. Weber, E. 1851. Annotationes anatomicae et physiologicae [anatomical and physiological annotations]. Leipzig: CF Koehler.