Tensorizing neural networks

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 442-450

  Novikov, Alexander ^a,d   Podoprikhin, Dmitry ^a   Osokin, Anton ^b   Vetrov, Dmitry ^a,c  

^a SKOLKOVO INSTITUTE OF SCIENCE AND TECHNOLOGY   (Russian Federation)

^b INRIA   (France)

^c NATIONAL RESEARCH UNIVERSITY HIGHER SCHOOL OF ECONOMICS   (Russian Federation)

^d INSTITUTE OF NUMERICAL MATHEMATICS   (Russian Federation)

Author keywords

[No Author keywords available]

Indexed keywords

COMPRESSION FACTOR; COMPUTATIONAL RESOURCES; DEEP NEURAL NETWORKS; EXPRESSIVE POWER; LARGE AMOUNTS; STATE-OF-THE-ART PERFORMANCE; TENSOR TRAINS; WEIGHT MATRICES;

INFORMATION SCIENCE;

EID: 84965128773     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. K. Asanovi and N. Morgan, "Experimental determination of precision requirements for back-propagation training of artificial neural networks", International Computer Science Institute, Tech. Rep., 1991.
2. J. Ba and R. Caruana, "Do deep nets really need to be deep?" in Advances in Neural Information Processing Systems 27 (NIPS), 2014, pp. 2654-2662.
3. J. D. Caroll and J. J. Chang, "Analysis of individual differences in multidimensional scaling via n-way generalization of Eckart-Young decomposition", Psychometrika, vol. 35, pp. 283-319, 1970.
4. W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, "Compressing neural networks with the hashing trick", in International Conference on Machine Learning (ICML), 2015, pp. 2285-2294.
5. G. Cybenko, "Approximation by superpositions of a sigmoidal function", Mathematics of control, signals and systems, pp. 303-314, 1989.
6. M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas, "Predicting parameters in deep learning", in Advances in Neural Information Processing Systems 26 (NIPS), 2013, pp. 2148-2156.
7. E. Denton, W. Zaremba, J. Bruna, Y. Le Cun, and R. Fergus, "Exploiting linear structure within convolutional networks for efficient evaluation", in Advances in Neural Information Processing Systems 27 (NIPS), 2014, pp. 1269-1277.
8. E. Gilboa, Y. Saati, and J. P. Cunningham, "Scaling multidimensional inference for structured gaussian processes", arXiv preprint, no. 1209.4120, 2012.
9. Y. Gong, L. Liu, M. Yang, and L. Bourdev, "Compressing deep convolutional networks using vector quantization", arXiv preprint, no. 1412.6115, 2014.
10. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, "Maxout networks", in International Conference on Machine Learning (ICML), 2013, pp. 1319-1327.
11. W. Hackbusch and S. Kühn, "A new scheme for the tensor representation", J. Fourier Anal. Appl., vol. 15, pp. 706-722, 2009.
12. A. Krizhevsky, "Learning multiple layers of features from tiny images", Master's thesis, Computer Science Department, University of Toronto, 2009.
13. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks", in Advances in Neural Information Processing Systems 25 (NIPS), 2012, pp. 1097-1105.
14. V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, "Speeding-up convolutional neural networks using fine-tuned CP-decomposition", In International Conference on Learning Representations (ICLR), 2014.
15. Y. Le Cun, C. Cortes, and C. J. C. Burges, "The MNIST database of handwritten digits", 1998.
16. A. Novikov, A. Rodomanov, A. Osokin, and D. Vetrov, "Putting MRFs on a Tensor Train", in International Conference on Machine Learning (ICML), 2014, pp. 811-819.
17. I. V. Oseledets, "Tensor-Train decomposition", SIAM J. Scientific Computing, vol. 33, no. 5, pp. 2295-2317, 2011.
18. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors", Nature, vol. 323, no. 6088, pp. 533-536, 1986.
19. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "Imagenet large scale visual recognition challenge", International Journal of Computer Vision (IJCV), 2015.
20. T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, "Low-rank matrix factorization for deep neural network training with high-dimensional output targets", in International Conference of Acoustics, Speech, and Signal Processing (ICASSP), 2013, pp. 6655-6659.
21. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition", In International Conference on Learning Representations (ICLR), 2015.
22. J. Snoek, H. Larochelle, and R. P. Adams, "Practical bayesian optimization of machine learning algorithms", in Advances in Neural Information Processing Systems 25 (NIPS), 2012, pp. 2951-2959.
23. L. R. Tucker, "Some mathematical notes on three-mode factor analysis", Psychometrika, vol. 31, no. 3, pp. 279-311, 1966.
24. A. Vedaldi and K. Lenc, "Matconvnet-convolutional neural networks for MATLAB", in Proceeding of the ACM Int. Conf. on Multimedia.
25. J. Xue, J. Li, and Y. Gong, "Restructuring of deep neural network acoustic models with singular value decomposition", in Interspeech, 2013, pp. 2365-2369.
26. Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang, "Deep fried convnets", arXiv preprint, no. 1412.7149, 2014.
27. Z. Zhang, X. Yang, I. V. Oseledets, G. E. Karniadakis, and L. Daniel, "Enabling high-dimensional hierarchical uncertainty quantification by ANOVA and tensor-train decomposition", Computer-Aided Design of Integrated Circuits and Systems, IEEE Transactions on, pp. 63-76, 2014.