On the depth of deep neural networks: A theoretical view

30th AAAI Conference on Artificial Intelligence, AAAI 2016

Volumn , Issue , 2016, Pages 2066-2072

  Sun, Shizhao ^a   Chen, Wei ^b   Wang, Liwei ^c   Liu, Xiaoguang ^a   Liu, Tie Yan ^b  

^a NANKAI UNIVERSITY   (China)

^b MICROSOFT RESEARCH   (United States)

^c Peking University   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; ERRORS;

BETTI NUMBERS; CROSS ENTROPY; DEEP NEURAL NETWORKS; HIDDEN UNITS; MARGIN BOUNDS; PENALTY TERM; REPRESENTATION POWER; TEST PERFORMANCE;

COMPLEX NETWORKS;

EID: 84999110931     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. Bartlett, P. L., and Mendelson, S. 2003. Rademacher and Gaussian complexities: Risk bounds and structural results. The Journal of Machine Learning Research 3:463-482.
2. Bartlett, P. L.; Maiorov, V.; and Meir, R. 1998. Almost linear vc-dimension bounds for piecewise polynomial networks. Neural computation 10(8):2159-2173.
3. Bartlett, P. L. 1998. The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Transactions on Information Theory 44(2):525-536.
4. Bianchini, M., and Scarselli, F. 2014. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks.
5. Ciresan, D.; Meier, U.; and Schmidhuber, J. 2012. Multicolumn deep neural networks for image classification. In Computer Vision and Pattern Recognition, 3642-3649.
6. Delalleau, O., and Bengio, Y. 2011. Shallow vs. deep sumproduct networks. In Advances in Neural Information Processing Systems, 666-674.
7. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei- Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 248-255. IEEE.
8. Gabrielov, A., and Vorobjov, N. 2004. Complexity of computations with pfaffian and noetherian functions. Normal forms, bifurcations and finiteness problems in differential equations 211-250.
9. Glorot, X., and Bengio, Y. 2010. Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics, 249-256.
10. Goldberg, P. W., and Jerrum, M. R. 1995. Bounding the vapnik-chervonenkis dimension of concept classes parameterized by real numbers. Machine Learning 18(2-3):131- 148.
11. Hastad, J. 1986. Almost optimal lower bounds for small depth circuits. In Proceedings of the eighteenth annual ACM symposium on Theory of computing, 6-20. ACM.
12. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv:1502.01852.
13. Hinton, G. E., and Salakhutdinov, R. R. 2006. Reducing the dimensionality of data with neural networks. Science 313(5786):504-507.
14. Hinton, G. E.; Deng, L.; Yu, D.; Dahl, G. E.; Mohamed, A.- r.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen, P.; Sainath, T. N.; et al. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29(6):82-97.
15. Jia, Y.; Shelhamer, E.; Donahue, J.; Karayev, S.; Long, J.; Girshick, R.; Guadarrama, S.; and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093.
16. Karpinski, M., and Macintyre, A. 1995. Polynomial bounds for vc dimension of sigmoidal neural networks. In Proceedings of the twenty-seventh annual ACM symposium on Theory of computing, 200-208. ACM.
17. Koltchinskii, V., and Panchenko, D. 2002. Empirical margin distributions and bounding the generalization error of combined classifiers. Annals of Statistics 1-50.
18. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 1097-1105.
19. Krizhevsky, A. 2009. Learning multiple layers of features from tiny images. Technical report, University of Toronto.
20. LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278-2324.
21. Lee, C.-Y.; Xie, S.; Gallagher, P.; Zhang, Z.; and Tu, Z. 2014. Deeply-supervised nets. arXiv preprint arXiv:1409.5185.
22. Li, C.; Zhu, J.; Shi, T.; and Zhang, B. 2015. Max-margin deep generative models. In Advances in neural information processing systems.
23. Montufar, G. F.; Pascanu, R.; Cho, K.; and Bengio, Y. 2014. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems, 2924- 2932.
24. Neyshabur, B.; Tomioka, R.; and Srebro, N. 2015. Normbased capacity control in neural networks. arXiv preprint arXiv:1503.00036.
25. Romero, A.; Ballas, N.; Kahou, S. E.; Chassang, A.; Gatta, C.; and Bengio, Y. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550.
26. Rosset, S.; Zhu, J.; and Hastie, T. J. 2003. Margin maximizing loss functions. In Advances in neural information processing systems, 1237-1244.
27. Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
28. Srivastava, R. K.; Greff, K.; and Schmidhuber, J. 2015. Training very deep networks. In Advances in neural information processing systems.
29. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2014. Going deeper with convolutions. arXiv preprint arXiv:1409.4842.
30. Weston, J.; Ratle, F.; Mobahi, H.; and Collobert, R. 2012. Deep learning via semi-supervised embedding. In Neural Networks: Tricks of the Trade. Springer. 639-655.
31. Zell, T. 1999. Betti numbers of semi-pfaffian sets. Journal of Pure and Applied Algebra 139(1):323-338.