Understanding the difficulty of training deep feedforward neural networks

Journal of Machine Learning Research

Volumn 9, Issue , 2010, Pages 249-256

  Glorot, Xavier ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

FASTER CONVERGENCE; GRADIENT DESCENT; HIDDEN LAYERS; JACOBIANS; MEAN VALUES; NON-LINEAR ACTIVATION; NON-LINEARITY; SINGULAR VALUES;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; FEEDFORWARD NEURAL NETWORKS; MULTILAYER NEURAL NETWORKS;

CHEMICAL ACTIVATION;

EID: 84862277874     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (20)

1. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2, 1-127. Also published as a book. Now Publishers, 2009.
2. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. NIPS 19 (pp. 153-160). MIT Press.
3. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5, 157-166.
4. Bergstra, J., Desjardins, G., Lamblin, P., & Bengio, Y. (2009). Quadratic polynomials learn better image features (Technical Report 1337). Département d'Informatique et de Recherche Opérationnelle, Université de Montréal.
5. Bradley, D. (2009). Learning in modular systems. Doctoral dissertation, The Robotics Institute, Carnegie Mellon University.
6. Collobert, R., &Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML 2008.
7. Erhan, D., Manzagol, P.-A., Bengio, Y., Bengio, S., & Vincent, P. (2009). The difficulty of training deep architectures and the effect of unsupervised pre-training. AISTATS'2009 (pp. 153-160).
8. Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527-1554.
9. Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images (Technical Report). University of Toronto.
10. Larochelle, H., Bengio, Y., Louradour, J., & Lamblin, P. (2009). Exploring strategies for training deep neural networks. The Journal of Machine Learning Research, 10, 1-40.
11. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. ICML 2007.
12. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998a). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278-2324.
13. LeCun, Y., Bottou, L., Orr, G. B., & Müller, K.-R. (1998b). Efficient backprop. In Neural networks, tricks of the trade, Lecture Notes in Computer Science LNCS 1524. Springer Verlag.
14. Mnih, A., & Hinton, G. E. (2009). A scalable hierarchical distributed language model. NIPS 21 (pp. 1081-1088).
15. Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. NIPS 19.
16. Rumelhart, D. E., Hinton, G. E., &Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536.
17. Solla, S. A., Levin, E., & Fleisher, M. (1988). Accelerated learning in layered neural networks. Complex Systems, 2, 625-639.
18. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. ICML 2008.
19. Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. ICML 2008 (pp. 1168-1175). New York, NY, USA: ACM.
20. Zhu, L., Chen, Y., & Yuille, A. (2009). Unsupervised learning of probabilistic grammar-markov models for object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 114-128.