Learning the number of neurons in deep networks

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 2270-2278

  Alvarez, Jose M ^a   Salzmann, Mathieu ^b  

^a CSIRO   (Australia)

^b EPFL   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

COMPACT ARCHITECTURE; COMPUTATION COSTS; GROUP SPARSITIES; NETWORK ACCURACY; NUMBER OF LAYERS; OVER-COMPLETE; REDUNDANT PARAMETERS; SINGLE NEURON;

NEURONS;

EID: 85018888114     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. J.M. Alvarez and L. Petersson. Decomposeme: Simplifying convnets for end-to-end learning. CoRR, abs/1606.05426, 2016.
2. T. Ash. Dynamic node creation in backpropagation networks. Connection Science, 1(4):365-375, 1989.
3. P. L. Bartlett. For valid generalization the size of the weights is more important than the size of the network. In NIPS, 1996.
4. M. G. Bello. Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks. IEEE Transactions on Neural Networks, 3(6):864-875, Nov 1992.
5. Yu Cheng, Felix X. Yu, Rogério Schmidt Feris, Sanjiv Kumar, Alok N. Choudhary, and Shih-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV, 2015.
6. M. D. Collins and P. Kohli. Memory Bounded Deep Convolutional Networks. CoRR, abs/1412.1442, 2014.
7. R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011.
8. M. Denil, B. Shakibi, L. Dinh, M.A. Ranzato, and N. de Freitas. Predicting parameters in deep learning. CoRR, abs/1306.0543, 2013.
9. E. L Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS. 2014.
10. Y. Gong, L. Liu, M. Yang, and L. D. Bourdev. Compressing deep convolutional networks using vector quantization. In CoRR, volume abs/1412.6115, 2014.
11. I. J. Goodfellow, D. Warde-farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In ICML, 2013.
12. B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In ICNN, 1993.
13. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CoRR, volume abs/1512.03385, 2015.
14. G. E. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In arXiv, 2014.
15. M. Jaderberg, A. Vedaldi, and A. Zisserman. Deep features for text spotting. In ECCV, 2014a.
16. M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014b.
17. C. Ji, R. R. Snapp, and D. Psaltis. Generalizing smoothness constraints from discrete samples. Neural Computation, 2(2):188-197, June 1990. ISSN 0899-7667.
18. A. Krogh and J. A. Hertz. A simple weight decay can improve generalization. In NIPS, 1992.
19. Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In NIPS, 1990.
20. B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Penksy. Sparse convolutional neural networks. In CVPR, 2015.
21. G. F Montufar, R. Pascanu, K. Cho, and Y. Bengio. On the number of linear regions of deep neural networks. In NIPS. 2014.
22. M. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In NIPS, 1988.
23. K. Murray and D. Chiang. Auto-sizing neural networks: With applications to n-gram language models. CoRR, abs/1508.05051, 2015.
24. N. Parikh and S. Boyd. Proximal algorithms. Found. Trends Optim., 1(3):127-239, January 2014.
25. R. Reed. Pruning algorithms - a survey. IEEE Transactions on Neural Networks, 4(5):740-747, Sep 1993.
26. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
27. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
28. J. Wang X. Huang, X. Zhang S. Tu, Y. Xue. Learning block group sparse representation combined with convolutional neural networks for rgb-d object recognition. Journal of Fiber Bioengineering and Informatics, 7(4):603, 2014.
29. N. Simon, J. Friedman, T. Hastie, and R. Tibshirani. A sparse-group lasso. Journal of Computational and Graphical Statistics, 2013.
30. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.
31. R. K Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In NIPS, 2015.
32. S. Theodoridis. Machine Learning, A Bayesian and Optimization Perspective, Volume 8. Elsevier, 2015.
33. A. S. Weigend, D. Rumelhart, and B. A. Huberman. Generalization by weight-elimination with application to forecasting. In NIPS, 1991.
34. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, Series B, 68(1):49-67, 2007.
35. B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva. Places: An image database for deep scene understanding. 2015.
36. H. Zhou, J. M. Alvarez, and F. Porikli. Less is more: Towards compact CNNs. In ECCV, 2016.