Deep learning made easier by linear transformations in perceptrons

Journal of Machine Learning Research

Volumn 22, Issue , 2012, Pages 924-932

  Raiko, Tapani ^a   Valpola, Harri ^a   LeCun, Yann ^b  

^a AALTO UNIVERSITY   (Finland)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; FISHER INFORMATION MATRIX; LEARNING ALGORITHMS; MATHEMATICAL TRANSFORMATIONS; MATRIX ALGEBRA; NETWORK LAYERS; STOCHASTIC SYSTEMS;

DIAGONAL MATRICES; INPUT-OUTPUT MAPPING; LINEAR DEPENDENCY; LOW-DIMENSIONAL REPRESENTATION; MULTI LAYER PERCEPTRON NETWORKS; NATURAL GRADIENT; SHORT-CUT CONNECTION; STOCHASTIC GRADIENT;

LINEAR TRANSFORMATIONS;

EID: 84893409634     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (15)

1. S. Amari. Natural gradient works efficiently in learning. Neural Computation, 10 (2): 251-276, 1998.
2. R. Battiti. First-and second-order methods for learning: Between steepest descent and Newton's method. Neural Computation, 4 (2): 141-166, 1992.
3. D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets excel on handwritten digit recognition. CoRR, abs/1003.0358, 2010.
4. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pages 249-256, 2010.
5. G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313 (5786): 504-507, 2006.
6. A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, 2009.
7. A. Krogh and J. A. Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems 4 (NIPS 1991), pages 950-957, 1992.
8. N. Le Roux, P. A. Manzagol, and Y. Bengio. Topmoumoute online natural gradient algorithm. In Advances in Neural Information Processing Systems 20 (NIPS∗2007), 2008.
9. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pages 2278-2324, 1998.
10. Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In Neural Networks: tricks of the trade. Springer-Verlag, 1998.
11. J. Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning (ICML), 2010.
12. S. Rifai, X. Glorot, Y. Bengio, and P. Vincent. Adding noise to the input of a model trained with a regularized objective. Technical Report 1359, Université de Montréal, Montréal (QC), H3C 3J7, Canada, April 2011.
13. N. N. Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical Report IDSIA-33-98, Istituto Dalle Molle di Studi sull'Intelligenza Artificiale, 1998.
14. N. N. Schraudolph. Centering neural network gradient factors. In Genevieve Orr and Klaus-Robert Mller, editors, Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pages 548-548. Springer Berlin / Heidelberg, 1998.
15. P. Vincent, H. Larochelle, Y. Bengio, and P. A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML08), pages 1096-1103, 2008.