A kronecker-factored approximate fisher matrix for convolution layers

33rd International Conference on Machine Learning, ICML 2016

Volumn 2, Issue , 2016, Pages 851-874

  Grosse, Roger ^a   Martens, James ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CURVE FITTING; ITERATIVE METHODS; LEARNING SYSTEMS; PROBABILITY DISTRIBUTIONS; STOCHASTIC SYSTEMS;

CONVOLUTIONAL NETWORKS; CRUDE APPROXIMATIONS; CURVATURE INFORMATION; KRONECKER PRODUCT; NATURAL GRADIENT; PROBABILISTIC MODELING; SECOND ORDER OPTIMIZATION; STOCHASTIC GRADIENT DESCENT;

CONVOLUTION;

EID: 84998893215     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (43)

1. Amari, Shun-Ichi. Natural gradient works efficiently in learning. Neural Computation, 10(2):251-276, 1998.
2. Chapelle, O. and Erhan, D. Improved preconditioner for Hessianfree optimization. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
3. Chellapilla, K., Puri, S., and Simard, P. High performance convolutional neural networks for document processing. In International Workshop on Frontiers in Handwriting Recognition, 2006.
4. Cho, K., Raiko, t., and Ilin, A. Enhanced gradient for training restricted Boltzmann machines. Neural Computation, 25:805-813, 2013.
5. Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A., Tucker, P., Yang, K., and Ng, A. Y. Large scale distributed deep networks. In Neural Information Processing Systems, 2012.
6. Demmel, J. W. Applied Numerical Linear Algebra. SIAM, 1997.
7. Desjardins, G., Simonyan, K., Pascanu, R., and Kavukcuoglu, K. Natural neural networks. arXiv. 1507.00210, 2015.
8. Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
9. Grosse, Roger and Salakhutdinov, Ruslan. Scaling up natural gradient by sparsely factorizing the inverse Fisher matrix. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
10. Heskes, Tom. On "natural" learning and pruning in multilayered perceptrons. Neural Computation, 12(4):881-901, 2000.
11. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9:1735-1780, 1997.
12. Ioffe, S. and Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
13. Kingma, D. P. and Ba, J. L. Adam: a method for stochastic optimization. In International Conference on Learning Representations, 2015.
14. Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
15. Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Neural Information Processing Systems, 2012.
16. Le Roux, Nicolas, Manzagol, Pierre-antoine, and Bengio, Yoshua. Topmoumoute online natural gradient algorithm. In Advances in Neural Information Processing Systems 20, pp. 849-856. MIT Press, 2008.
17. LeCun, Y, Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541-551, 1989.
18. LeCun, Y, Bottou, L., Orr, G., and Müller, K. Efficient backprop. Neural networks: Tricks of the trade, pp. 546-546, 1998.
19. Martens, J. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning (ICML), 2010.
20. Martens, J. New insights and perspectives on the natural gradient method, 2014.
21. Martens, J. and Grosse, R. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning, 2015.
22. Mnih, V. CUDAMat: A CUDA-based matrix class for Python. Technical Report 004, University of Toronto, 2009.
23. Moré, J. J. The Levenberg-Marquardt algorithm: implementation and theory. Numerical analysis, pp. 105-116, 1978.
24. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems Deep Learning and Unsupervised Feature Learning Workshop, 2011.
25. Nocedal, Jorge and Wright, Stephen J. Numerical optimization. Springer, 2. ed. edition, 2006.
26. Ollivier, Y. Riemannian metrics for neural networks I: feedforward networks. Information and Inference, 4(2):108-153, 2015.
27. Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, 2013.
28. Pascanu, R., Dauphin, Y. N., Ganguli, S., and Bengio, Y. On the saddle point problem for non-convex optimization. arXiv: 1405.4604, 2014.
29. Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM Journal of Control and Optimization, 30(4):838-855, 1992.
30. Povey, Daniel, Zhang, Xiaohui, and Khudanpur, Sanjeev. Parallel training of DNNs with natural gradient and parameter averaging. In International Conference on Learning Representations: Workshop track, 2015.
31. Ranzato, M. and Hinton, G. E. Modeling pixel means and covariances using factorized third-order Boltzmann machines. In Computer Vision and Pattern Recognition, 2010.
32. Schraudolph, Nicol N. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14, 2002.
33. Simoncelli, E. P. and Olshausen, B. A. Natural image statistics and neural representation. Annual Review of Neuroscience, 24:1193-1216, 2001.
34. Sohl-Dickstein, J., Poole, B., and Ganguli, S. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In International Conference on Machine Learning, 2014.
35. Srivastava, N. Improving neural networks with dropout. Master's thesis, University of Toronto, 2013.
36. Srivastava, N. Toronto Deep Learning ConvNet. https://github.com/TorontoDeepLearning/convnet/, 2015.
37. Sutskever, I., Vinyals, O., and Le, Q. V. V. Sequence to sequence learning with neural networks. In Neural Information Processing Systems, 2014.
38. Swersky, K., Chen, Bo, Marlin, B., and de Freitas, N. A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets. In Information Theory and Applications Workshop (ITA), 2010, pp. 1-10, Jan 2010.
39. Tieleman, T. and Hinton, G. Lecture 6.5, RMSProp. In Coursera course Neural Networks for Machine Learning, 2012.
40. Vatanen, Tommi, Raiko, Tapani, Valpola, Harri, and LeCun, Yann. Pushing stochastic gradient towards second-order methods - backpropagation learning with transformations in nonlinearities. 2013.
41. Vinyals, O. and Povey, D. Krylov subspace descent for deep learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2012.
42. Yang, Z., Moczulski, M., Denil, M., de Freitas, N., Smola, A., Song, L., and Wang, Z. Deep fried convnets. arXiv: 1412.7149, 2014.
43. Zeiler, Matthew D. ADADELTA: An adaptive learning rate method. 2013.