On the importance of normalisation layers in deep learning with piecewise linear activation units

2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016

Volumn , Issue , 2016, Pages

  Liao, Zhibin ^a   Carneiro, Gustavo ^a  

^a UNIVERSITY OF ADELAIDE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION; CLASSIFICATION (OF INFORMATION); COMPLEX NETWORKS; COMPUTER VISION; DIGITAL STORAGE; FEEDFORWARD NEURAL NETWORKS;

ACTIVATION FUNCTIONS; CLASSIFICATION RESULTS; COMPLEX FUNCTIONS; PIECEWISE LINEAR ACTIVATIONS; PRE-CONDITIONING; REGULARIZATION SCHEMES; STATE OF THE ART; TRAINING PROCESS;

PIECEWISE LINEAR TECHNIQUES;

EID: 84977615889     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WACV.2016.7477624     Document Type: Conference Paper

References (21)

1. Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio, "On the number of linear regions of deep neural networks, " in Advances in Neural Information Processing Systems (NIPS), 2014, pp. 2924-2932.
2. Vinod Nair and Geoffrey E Hinton, "Rectified linear units improve restricted boltzmann machines, " in Proceedings of the 27th International Conference on Machine Learning (ICML), 2010, pp. 807-814.
3. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng, "Rectifier nonlinearities improve neural network acoustic models, " in Proc. ICML, 2013, vol. 30.
4. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, " IEEE International Conference on Computer Vision (ICCV), 2015.
5. Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmidhuber, "Compete to compute, " in Advances in Neural Information Processing Systems (NIPS), 2013, pp. 2310-2318.
6. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio, "Maxout networks, " The 30th International Conference on Machine Learning (ICML), 2013.
7. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, "Imagenet classification with deep convolutional neural networks, " in Advances in neural information processing systems (NIPS), 2012, pp. 1097-1105.
8. Alex Krizhevsky and Geoffrey Hinton, "Learning multiple layers of features from tiny images, " Computer Science Department, University of Toronto, Tech. Rep, vol. 1, no. 4, pp. 7, 2009.
9. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner, "Gradient-based learning applied to document recognition, " Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
10. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, BoWu, and Andrew Y Ng, "Reading digits in natural images with unsupervised feature learning, " in NIPS workshop on deep learning and unsupervised feature learning. Granada, Spain, 2011, vol. 2011, p. 5.
11. Rupesh Kumar Srivastava, Jonathan Masci, Faustino Gomez, and Jürgen Schmidhuber, "Understanding locally competitive networks, " International Conference on Learning Representations (ICLR), 2015.
12. Sergey Ioffe and Christian Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift, " International Conference on Machine Learning (ICML), 2015.
13. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting, " The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
14. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams, "Learning representations by back-propagating errors, " Cognitive modeling, vol. 5, 1988.
15. Min Lin, Qiang Chen, and Shuicheng Yan, "Network in network, " International Conference on Learning Representations (ICLR), 2013.
16. A. Vedaldi and K. Lenc, "Matconvnet-convolutional neural networks for matlab, " CoRR, vol. Abs/1412. 4564, 2014.
17. Matthew D Zeiler and Rob Fergus, "Stochastic pooling for regularization of deep convolutional neural networks, " International Conference on Learning Representations (ICLR), 2013.
18. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu, "Deeply-supervised nets, " In Proceedings of AISTATS, 2015.
19. Ming Liang and Xiaolin Hu, "Recurrent convolutional neural network for object recognition, " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3367-3375.
20. Nitish Srivastava and Ruslan R Salakhutdinov, "Discriminative transfer learning with tree-based priors, " in Advances in Neural Information Processing Systems (NIPS), 2013, pp. 2094-2102.
21. Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus, "Regularization of neural networks using dropconnect, " in Proceedings of the 30th International Conference on Machine Learning (ICML), 2013, pp. 1058-1066.