An exploration of parameter redundancy in deep networks with circulant projections

Proceedings of the IEEE International Conference on Computer Vision

Volumn 2015 International Conference on Computer Vision, ICCV 2015, Issue , 2015, Pages 2857-2865

  Cheng, Yu ^a,c   Yu, Felix X ^b,d   Feris, Rogerio S ^a   Kumar, Sanjiv ^b   Choudhary, Alok ^c   Chang, Shi Fu ^d  

^a IBM RESEARCH   (United States)

^b GOOGLE INC   (United States)

^c NORTHWESTERN UNIVERSITY   (United States)

^d Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; COMPUTER VISION; FAST FOURIER TRANSFORMS; NETWORK ARCHITECTURE; NEURAL NETWORKS; REDUNDANCY;

CONVOLUTIONAL NEURAL NETWORK; DEEP NEURAL NETWORKS; FULLY CONNECTED NEURAL NETWORK; GRADIENT COMPUTATION; LINEAR PROJECTIONS; NETWORK PARAMETERS; PARAMETER REDUNDANCY; SPACE COMPLEXITY;

NETWORK LAYERS;

EID: 84973890879     PISSN: 15505499     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICCV.2015.327     Document Type: Conference Paper

References (45)

1. N. Ailon and B. Chazelle. Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. In ACM Symposium on Theory of Computing, 2006.
2. A. Bissacco, M. Cummins, Y. Netzer, and H. Neven. Photoocr: Reading text in uncontrolled conditions. In ICCV, 2013.
3. A. Collins and P. Kohli. Memory bounded deep convolutional networks. In http: //arxiv. org/abs/1412. 1442, 2014.
4. A. Dasgupta, R. Kumar, and T. Sarlós. Fast locality-sensitive hashing. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2011.
5. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In NIPS, 2012.
6. M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. Freitas. Predicting Parameters in Deep Learning. In NIPS, 2013.
7. E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS, pages 1269-1277, 2014.
8. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv: 1310. 1531, 2013.
9. D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. In CVPR, 2014.
10. C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello. Hardware accelerated convolutional neural networks for synthetic vision systems. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 257-260. IEEE, 2010.
11. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
12. X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier networks. In International Conference on Artificial Intelligence and Statistics, 2011.
13. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. arXiv preprint arXiv: 1302. 4389, 2013.
14. J. Henriques, J. Carreira, R. Caseiro, and J. Batista. Beyond hard negative mining: Efficient detector learning via blockcirculant decomposition. In ICCV, 2013.
15. J. Henriques, R. Caseiro, P. Martins, and J. Batista. Exploiting the circulant structure of tracking-by-detection with kernels. In ECCV, 2012.
16. A. Hinrichs and J. Vybíral. Johnson-Lindenstrauss lemma for circulant matrices. Random Structures & Algorithms, 39 (3): 391-398, 2011.
17. G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing coadaptation of feature detectors. In ArXiv e-prints, 2012.
18. G. Hinton, O. Vinyals, and J. Dean. Distilling knowledge in a neural network. In Deep Learning and Representation Learning Workshop, NIPS, 2014.
19. P.-S. Huang, H. Avron, T. N. Sainath, V. Sindhwani, and B. Ramabhadran. Kernel methods match deep neural networks on timit. In ICASSP. IEEE, 2014.
20. M. Jaderberg, A. Vedaldi, and A. Zisserman. Deep features for text spotting. In ECCV, 2014.
21. M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014.
22. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv: 1408. 5093, 2014.
23. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
24. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. In Proceedings of the IEEE, pages 2278-2324, 1998.
25. M. Lin, Q. Chen, and S. Yan. Network in network. In ICLR, 2014.
26. M. Mathieu, M. Henaff, and Y. LeCun. Fast training of convolutional networks through FFTs. arXiv preprint arXiv: 1312. 5851, 2013.
27. M. D. McDonnell, M. D. Tissera, A. van Schaik, and J. Tapson. Fast, simple and accurate handwritten digit classification using extreme learning machines with shaped inputweights. arXiv preprint arXiv: 1412. 8307, 2014.
28. M. D. McDonnell and T. Vladusich. Enhanced image classification with a fast-learning shallow convolutional neural network. arXiv preprint arXiv: 1503. 04596, 2015.
29. A. V. Oppenheim, R. W. Schafer, J. R. Buck, et al. Discretetime signal processing, volume 5. Prentice Hall Upper Saddle River, 1999.
30. M. Prezioso, F. Merrikh-Bayat, B. Hoskins, G. Adam, K. K. Likharev, and D. B. Strukov. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature, 521 (7550): 61-64, 2015.
31. A. Romero, N. Ballas, S. Kahou, A. Chassang, C. Gatta, and Y. Bengio. FitNets: Hints for thin deep nets. In ICLR, 2015.
32. T. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran. Low-Rank Matrix Factorization for Deep Neural Network Training with High-Dimensional Output Targets. In ICASSP, 2013.
33. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR, 2014.
34. H. Soltau, G. Saon, and T. Sainath. Joint training of convoutional and non-convoutional neural networks. In ICASSP, 2014.
35. Y. Sun, X. Wang, and X. Tang. Deeply learned face representations are sparse, selective, and robust. In CVPR, 2015.
36. Y. Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10, 000 classes. In CVPR, 2014.
37. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In arXiv preprint arXiv: 1409. 4842, 2014.
38. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014.
39. K. van de Sande, J. Uijlings, T. Gevers, and A. Smeulders. Segmentation as selective search for object recognition. In ICCV, 2011.
40. V. Vanhoucke, A. Senior, and M. Z. Mao. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011.
41. J. Vybíral. A variant of the johnson-lindenstrauss lemma for circulant matrices. Journal of Functional Analysis, 260 (4): 1096-1105, 2011.
42. Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang. Deep fried convnets. arXiv preprint arXiv: 1412. 7149, 2014.
43. F. X. Yu, S. Kumar, Y. Gong, and S.-F. Chang. Circulant binary embedding. In International Conference on Machine Learning, 2014.
44. F. X. Yu, S. Kumar, H. Rowley, and S.-F. Chang. Compact nonlinear maps and circulant extensions. arXiv preprint arXiv: 1503. 03893, 2015.
45. M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.