All you need is a good init

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Mishkin, Dmytro ^a   Matas, Jiri ^a  

^a CZECH TECHNICAL UNIVERSITY IN PRAGUE   (Czech Republic)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVATION FUNCTIONS; COMPLEX SCHEMES; INNER PRODUCT; ORTHONORMAL; SIMPLE METHOD; STATE OF THE ART; TEST ACCURACY; WEIGHT INITIALIZATION;

DEEP LEARNING;

EID: 85083951894     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. Bengio, Yoshua, Lamblin, Pascal, Popovici, Dan, and Larochelle, Hugo. Greedy layer-wise training of deep networks. In Schölkopf, B., Platt, J.C., and Hoffman, T. (eds.), Advances in Neural Information Processing Systems 19, pp. 153–160. MIT Press, 2007. URL http://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf.
2. Chang, J.-R. and Chen, Y.-S. Batch-normalized Maxout Network in Network. ArXiv e-prints, November 2015. URL http://arxiv.org/abs/1511.02583.
3. Dieleman, Sander. Classifying plankton with deep neural networks, 2015. URL http://benanne.github.io/2015/03/17/plankton.html.
4. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10). Society for Artificial Intelligence and Statistics, 2010.
5. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier neural networks. In Gordon, Geoffrey J. and Dunson, David B. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS-11), volume 15, pp. 315–323. Journal of Machine Learning Research - Workshop and Conference Proceedings, 2011. URL http://www.jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf.
6. Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron C., and Bengio, Yoshua. Maxout networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 1319–1327, 2013. URL http://jmlr.org/proceedings/papers/v28/goodfellow13.html.
7. Graham, Ben. Fractional Max-Pooling. ArXiv e-prints, December 2014a. URL http://arxiv.org/abs/1412.6071.
8. Graham, Ben. Train you very own deep convolutional network, 2014b. URL https://www.kaggle.com/c/cifar-10/forums/t/10493/train-you-very-own-deep-convolutional-network.
9. Graham, Ben. Spatially-sparse convolutional neural networks. ArXiv e-prints, September 2014c.
10. He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. ArXiv e-prints, December 2015. URL http://arxiv.org/abs/1512/03385.
11. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision (ICCV), 2015. URL http://arxiv.org/abs/1502.01852.
12. Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the Knowledge in a Neural Network. In Proceedings of Deep Learning and Representation Learning Workshop: NIPS 2014, 2014.
13. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Blei, David and Bach, Francis (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 448–456. JMLR Workshop and Conference Proceedings, 2015. URL http://jmlr.org/proceedings/papers/v37/ioffe15.pdf.
14. Jia, Yangqing, Shelhamer, Evan, Donahue, Jeff, Karayev, Sergey, Long, Jonathan, Girshick, Ross, Guadarrama, Sergio, and Darrell, Trevor. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
15. Krizhevsky, Alex. Learning Multiple Layers of Features from Tiny Images. Master’s thesis, 2009. URL http://www.cs.toronto.edu/{}kriz/learning-features-2009-TR.pdf.
16. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C.J.C., Bottou, L., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc., 2012. URL http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks. pdf.
17. Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, Nov 1998. ISSN 0018-9219. doi: 10.1109/5.726791.
18. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick W., Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2015, San Diego, California, USA, May 9-12, 2015, 2015. URL http://jmlr.org/proceedings/papers/v38/lee15a.html.
19. Maas, Andrew L, Hannun, Awni Y, and Ng, Andrew Y. Rectifier nonlinearities improve neural network acoustic models. Proc. ICML, 30, 2013.
20. Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. Fitnets: Hints for thin deep nets. In Proceedings of ICLR, May 2015. URL http://arxiv.org/abs/1412.6550.
21. Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), pp. 1–42, April 2015. doi: 10.1007/s11263-015-0816-y.
22. Saxe, Andrew M., McClelland, James L., and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of ICLR, 2014. URL http://arxiv.org/abs/1312.6120.
23. Sercu, T., Puhrsch, C., Kingsbury, B., and LeCun, Y. Very Deep Multilingual Convolutional Neural Networks for LVCSR. ArXiv e-prints, September 2015. URL http://arxiv.org/abs/1509/08967.
24. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale visual recognition. In Proceedings of ICLR, May 2015. URL http://arxiv.org/abs/1409.1556.
25. Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. Striving for Simplicity: The All Convolutional Net. In Proceedings of ICLR Workshop, December 2014. URL http://arxiv.org/abs/1412.6806.
26. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jrgen. Training Very Deep Networks. In Proceedings of NIPS, 2015. URL http://arxiv.org/abs/1507.06228.
27. Sussillo, David and Abbott, L. F. Random Walk Initialization for Training Very Deep Feedforward Networks. ArXiv e-prints, December 2014. URL http://arxiv.org/abs/1412.6558.
28. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. In CVPR 2015, 2015. URL http://arxiv.org/abs/1409.4842.
29. Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5: RMSProp – Divide the gradient by a running average of its recent magnitude. In COURSERA: Neural Networks for Machine Learning. 2012.