Fast and accurate deep network learning by exponential linear units (ELUs)

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Clevert, Djork Arné ^a   Unterthiner, Thomas ^a   Hochreiter, Sepp ^a  

^a JOHANNES KEPLER UNIVERSITY LINZ   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION;

ACTIVATION FUNCTIONS; CLASSIFICATION ACCURACY; CLASSIFICATION ERRORS; GENERALIZATION PERFORMANCE; NATURAL GRADIENT; NEGATIVE VALUES; NETWORK LEARNING; VANISHING GRADIENT;

DEEP NEURAL NETWORKS;

EID: 85083953568     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. Amari, S.-I. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998.
2. Clevert, D.-A., Unterthiner, T., Mayr, A., and Hochreiter, S. Rectified factor networks. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 28. Curran Associates, Inc., 2015.
3. Desjardins, G., Simonyan, K., Pascanu, R., and Kavukcuoglu, K. Natural neural networks. CoRR, abs/1507.00210, 2015. URL http://arxiv.org/abs/1507.00210.
4. Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Gordon, G., Dunson, D., and Dudk, M. (eds.), JMLR W&CP: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2011), volume 15, pp. 315–323, 2011.
5. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout networks. ArXiv e-prints, 2013.
6. Graham, Benjamin. Fractional max-pooling. CoRR, abs/1412.6071, 2014. URL http://arxiv.org/abs/1412.6071.
7. Grosse, R. and Salakhudinov, R. Scaling up natural gradient by sparsely factorizing the inverse Fisher matrix. Journal of Machine Learning Research, 37:2304–2313, 2015. URL http://jmlr.org/proceedings/papers/v37/grosse15.pdf. Proceedings of the 32nd International Conference on Machine Learning (ICML15).
8. He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision (ICCV), 2015.
9. Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(2):107–116, 1998.
10. Hochreiter, S. and Schmidhuber, J. Feature extraction through LOCOCODE. Neural Computation, 11(3): 679–714, 1999.
11. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Kremer and Kolen (eds.), A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
12. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Journal of Machine Learning Research, 37:448–456, 2015. URL http://jmlr.org/proceedings/papers/v37/ioffe15.pdf. Proceedings of the 32nd International Conference on Machine Learning (ICML15).
13. Jia, Yangqing. Learning Semantic Image Representations at a Large Scale. PhD thesis, EECS Department, University of California, Berkeley, May 2014. URL http://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-93.html.
14. Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc., 2012.
15. Kurita, T. Iterative weighted least squares algorithms for neural networks classifiers. In Proceedings of the Third Workshop on Algorithmic Learning Theory (ALT92), volume 743 of Lecture Notes in Computer Science, pp. 77–86. Springer, 1993.
16. LeCun, Y., Kanter, I., and Solla, S. A. Eigenvalues of covariance matrices: Application to neural-network learning. Physical Review Letters, 66(18):2396–2399, 1991.
17. LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Orr, G. B. and Müller, K.-R. (eds.), Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pp. 9–50. Springer, 1998.
18. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick W., Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. In AISTATS, 2015.
19. LeRoux, N., Manzagol, P.-A., and Bengio, Y. Topmoumoute online natural gradient algorithm. In Platt, J. C., Koller, D., Singer, Y., and Roweis, S. T. (eds.), Advances in Neural Information Processing Systems 20 (NIPS), pp. 849–856, 2008.
20. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. CoRR, abs/1312.4400, 2013. URL http://arxiv.org/abs/1312.4400.
21. Maas, A. L., Hannun, A. Y., and Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference on Machine Learning (ICML13), 2013.
22. Martens, J. Deep learning via Hessian-free optimization. In Fürnkranz, J. and Joachims, T. (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML10), pp. 735–742, 2010.
23. Mayr, A., Klambauer, G., Unterthiner, T., and Hochreiter, S. DeepTox: Toxicity prediction using deep learning. Front. Environ. Sci., 3(80), 2015. doi: 10.3389/fenvs.2015.00080. URL http://journal.frontiersin.org/article/10.3389/fenvs.2015.00080.
24. Nair, V. and Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Fürnkranz, J. and Joachims, T. (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML10), pp. 807–814, 2010.
25. Olivier, Y. Riemannian metrics for neural networks i: feedforward networks. CoRR, abs/1303.0818, 2013. URL http://arxiv.org/abs/1303.0818.
26. Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks. In International Conference on Learning Representations 2014, 2014. URL http://arxiv.org/abs/1301.3584.arXiv:1301.3584.
27. Raiko, T., Valpola, H., and LeCun, Y. Deep learning made easier by linear transformations in perceptrons. In Lawrence, N. D. and Girolami, M. A. (eds.), Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS12), volume 22, pp. 924–932, 2012.
28. Schraudolph, N. N. Centering neural network gradient factor. In Orr, G. B. and Müller, K.-R. (eds.), Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pp. 207–226. Springer, 1998.
29. Schraudolph, Nicol N. A Fast, Compact Approximation of the Exponential Function. Neural Computation, 11: 853–862, 1999.
30. Springenberg, Jost Tobias, Dosovitskiy, Alexey, Brox, Thomas, and Riedmiller, Martin A. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014. URL http://arxiv.org/abs/1412.6806.
31. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jürgen. Training very deep networks. CoRR, abs/1507.06228, 2015. URL http://arxiv.org/abs/1507.06228.
32. Unterthiner, T., Mayr, A., Klambauer, G., and Hochreiter, S. Toxicity prediction using deep learning. CoRR, abs/1503.01445, 2015. URL http://arxiv.org/abs/1503.01445.
33. Vinyals, O. and Povey, D. Krylov subspace descent for deep learning. In AISTATS, 2012. URL http://arxiv.org/pdf/1111.4259v1. arXiv:1111.4259.
34. Xu, B., Wang, N., Chen, T., and Li, M. Empirical evaluation of rectified activations in convolutional network. CoRR, abs/1505.00853, 2015. URL http://arxiv.org/abs/1505.00853.
35. Yang, H. H. and Amari, S.-I. Complexity issues in natural gradient descent method for training multilayer perceptrons. Neural Computation, 10(8), 1998.