Self-normalizing neural networks

Advances in Neural Information Processing Systems

Volumn 2017-December, Issue , 2017, Pages 972-981

  Klambauer, Günter ^a   Unterthiner, Thomas ^a   Mayr, Andreas ^a   Hochreiter, Sepp ^a  

^a JOHANNES KEPLER UNIVERSITY LINZ   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION; DECISION TREES; DEEP LEARNING; FIXED POINT ARITHMETIC; LEARNING ALGORITHMS; NATURAL LANGUAGE PROCESSING SYSTEMS; NETWORK LAYERS;

ABSTRACT REPRESENTATION; BANACH FIXED-POINT THEOREM; CONVERGENCE PROPERTIES; CONVOLUTIONAL NEURAL NETWORK; MACHINE LEARNING METHODS; RECURRENT NEURAL NETWORK (RNNS); UCI MACHINE LEARNING REPOSITORY; UPPER AND LOWER BOUNDS;

RECURRENT NEURAL NETWORKS;

EID: 85039743275     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Ba, J. L., and Kiros, J. R., and Hinton, G. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.
2. Bengio, Y. (2013). Deep learning of representations: Looking forward. In Proceedings of the First International Conference on Statistical Language and Speech Processing, pages 1-37, Berlin, Heidelberg.
3. Bradley, R. C. (1981). Central limit theorems under weak dependence. Journal of Multivariate Analysis, 11(1):1-16.
4. Cireşan, D. and Meier, U. (2015). Multi-column deep neural networks for offline handwritten chinese character classification. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1-6. IEEE.
5. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2015). Fast and accurate deep network learning by exponential linear units (ELUs). 5th International Conference on Learning Representations, arXiv:1511.07289.
6. Desjardins, G., Simonyan, K., Pascanu, R., et al (2015). Natural neural networks. In Advances in Neural Information Processing Systems, pages 2071-2079.
7. Dugan, P., Clark, C., LeCun, Y., and Van Parijs, S. (2016). Phase 4: Dcl system using deep learning approaches for land-based or ship-based real-time recognition and localization of marine mammals-distributed processing and big data applications. arXiv preprint arXiv:1605.00982.
8. Esteva, A., Kuprel, B., Novoa, R., Ko, J., Swetter, S., Blau, H., and Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115-118.
9. Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems. Journal of Machine Learning Research, 15(1):3133-3181.
10. Graves, A., Mohamed, A., and Hinton, G. (2013). Speech recognition with deep recurrent neural networks. In IEEE International conference on acoustics, speech and signal processing (ICASSP), pages 6645-6649.
11. Graves, A. and Schmidhuber, J. (2009). Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in neural information processing systems, pages 545-552.
12. Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22):2402-2410.
13. He, K., Zhang, X., Ren, S., and Sun, J. (2015a). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
14. He, K., Zhang, X., Ren, S., and Sun, J. (2015b). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026-1034.
15. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8):1735-1780.
16. Huval, B., Wang, T., Tandon, S., et al (2015). An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716.
17. Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning, pages 448-456.
18. Korolev, V. and Shevtsova, I. (2012). An improvement of the Berry-Esseen inequality with applications to Poisson and mixed Poisson random sums. Scandinavian Actuarial Journal, 2012(2):81-105.
19. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105.
20. LeCun, Y. and Bengio, Y. (1995). Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995.
21. LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature, 521(7553):436-444.
22. Lyon, R., Stappers, B., Cooper, S., Brooke, J., and Knowles, J. (2016). Fifty years of pulsar candidate selection: From simple filters to a new principled real-time classification approach. Monthly Notices of the Royal Astronomical Society, 459(1):1104-1123.
23. Mayr, A., Klambauer, G., Unterthiner, T., and Hochreiter, S. (2016). DeepTox: Toxicity prediction using deep learning. Frontiers in Environmental Science, 3:80.
24. Sak, H., Senior, A., Rao, K., and Beaufays, F. (2015). Fast and accurate recurrent neural network acoustic models for speech recognition. arXiv preprint arXiv:1507.06947.
25. Salimans, T. and Kingma, D. P. (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901-909.
26. Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61:85-117.
27. Silver, D., Huang, A., Maddison, C., et al (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489.
28. Srivastava, R. K., Greff, K., and Schmidhuber, J. (2015). Training very deep networks. In Advances in Neural Information Processing Systems, pages 2377-2385.
29. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112.
30. Wainberg, M., Alipanahi, B., and Frey, B. J. (2016). Are random forests truly the best classifiers? Journal of Machine Learning Research, 17(110):1-5.