Natural neural networks

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 2071-2079

  Desjardins, Guillaume ^a   Simonyan, Karen ^a   Pascanu, Razvan ^a   Kavukcuoglu, Koray ^a  

^a DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION SCIENCE; LEARNING ALGORITHMS;

FEED FORWARD; INTERNAL REPRESENTATION; NATURAL GRADIENT; NETWORK WEIGHTS; ONLINE LEARNING ALGORITHMS; REPARAMETRIZATION; SPEED UP;

ALGORITHMS;

EID: 84965130201     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 1998.
2. Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS. 2014.
3. Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett., 2003.
4. P. L. Combettes and J.-C. Pesquet. Proximal Splitting Methods in Signal Processing. ArXiv e-prints, December 2009.
5. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In JMLR. 2011.
6. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, May 2010.
7. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
8. Roger Grosse James Martens. Optimizing neural networks with kronecker-factored approximate curvature. In ICML, June 2015.
9. Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
10. Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. In Neural Networks, Tricks of the Trade, Lecture Notes in Computer Science LNCS 1524. Springer Verlag, 1998.
11. Yann Lecun, Lon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278-2324, 1998.
12. James Martens. Deep learning via Hessian-free optimization. In ICML, June 2010.
13. K.-R. Müller and G. Montavon. Deep boltzmann machines and the centering trick. In K.-R. Müller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade. Springer, 2013.
14. Yann Ollivier. Riemannian metrics for neural networks. arXiv, abs/1303.0818, 2013.
15. Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In ICLR, 2014.
16. Daniel Povey, Xiaohui Zhang, and Sanjeev Khudanpur. Parallel training of deep neural networks with natural gradient and parameter averaging. ICLR workshop, 2015.
17. T. Raiko, H. Valpola, and Y. LeCun. Deep learning made easier by linear transformations in perceptrons. In AISTATS, 2012.
18. G. Raskutti and S. Mukherjee. The Information Geometry of Mirror Descent. arXiv, October 2013.
19. Ruslan Salakhutdinov Roger B. Grosse. Scaling up natural gradient by sparsely factorizing the inverse fisher matrix. In ICML, June 2015.
20. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
21. Nicol N. Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical Report IDSIA-33-98, Istituto Dalle Molle di Studi sull'Intelligenza Artificiale, 1998.
22. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
23. Jascha Sohl-Dickstein. The natural gradient by analogy to signal whitening, and recipes and tricks for its use. arXiv, 2012.
24. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.
25. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv, 2014.
26. Philip S Thomas, William C Dabney, Stephen Giguere, and Sridhar Mahadevan. Projected natural actorcritic. In Advances in Neural Information Processing Systems 26. 2013.
27. Tijmen Tieleman and Geoffrey Hinton. Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. 2012.
28. Tommi Vatanen, Tapani Raiko, Harri Valpola, and Yann LeCun. Pushing stochastic gradient towards second-order methods - backpropagation learning with transformations in nonlinearities. ICONIP, 2013.