On the number of response regions of deep feedforward networks with piecewise linear activations

2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings

Volumn , Issue , 2014, Pages

  Pascanu, Razvan ^a   Montúfar, Guido ^b   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b MAX PLANCK INSTITUTE FOR MATHEMATICS IN THE SCIENCES   (Germany)

Author keywords

Artificial neural network; Deep learning; Hyperplane arrangement; Rectifier unit; Representational power

Indexed keywords

CHEMICAL ACTIVATION; COMPUTATIONAL GEOMETRY; DEEP LEARNING; DEEP NEURAL NETWORKS; ELECTRIC RECTIFIERS; NETWORK ARCHITECTURE; NEURAL NETWORKS; PIECEWISE LINEAR TECHNIQUES;

FEED-FORWARD NETWORK; HYPERPLANE ARRANGEMENTS; MATHEMATICAL TOOLS; MULTI LAYER PERCEPTRON; PIECE-WISE LINEAR FUNCTIONS; PIECEWISE LINEAR ACTIVATIONS; RECTIFIER UNIT; REPRESENTATIONAL POWER;

COMPLEX NETWORKS;

EID: 85083950242     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Y. Bengio. Learning deep architectures for AI. Foundations and trendsR in Machine Learning, 2(1):1–127, 2009.
2. Y. Bengio and O. Delalleau. On the expressive power of deep architectures. In J. Kivinen, C. Szepesvri, E. Ukkonen, and T. Zeugmann, editors, Algorithmic Learning Theory, volume 6925 of Lecture Notes in Computer Science, pages 18–36. Springer Berlin Heidelberg, 2011.
3. X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011.
4. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In ICML’2013, 2013.
5. A. Hajnal, W. Maass, P. Pudlk, M. Szegedy, and G. Turn. Threshold circuits of bounded depth. Journal of Computer and System Sciences, 46(2):129–154, 1993.
6. J. Håstad. Almost optimal lower bounds for small depth circuits. In Proceedings of the 18th annual ACM Symposium on Theory of Computing, pages 6–20, Berkeley, California, 1986. ACM Press.
7. J. Håstad and M. Goldmann. On the power of small-depth threshold circuits. Computational Complexity, 1: 113–129, 1991.
8. G. Hinton, L. Deng, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012a.
9. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinv. Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580, 2012b.
10. N. Le Roux and Y. Bengio. Deep belief networks are compact universal approximators. Neural Computation, 22(8):2192–2207, 2010.
11. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Montreal (QC), Canada, 2009.
12. J. Martens, A. Chattopadhya, T. Pitassi, and R. Zemel. On the expressive power of restricted boltzmann machines. In Advances in Neural Information Processing Systems 26, pages 2877–2885. 2013.
13. G. Montúfar and N. Ay. Refinements of universal approximation results for deep belief networks and restricted Boltzmann machines. Neural Computation, 23(5):1306–1319, 2011.
14. G. Montúfar and J. Morton. When does a mixture of products contain a product of mixtures? arXiv preprint arXiv:1206.0387, 2012.
15. G. Montúfar, J. Rauh, and N. Ay. Expressive power and approximation errors of restricted Boltzmann machines. Advances in Neural Information Processing Systems, 24:415–423, 2011.
16. V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. pages 807–814, 2010.
17. H. Poon and P. Domingos. Sum-product networks: A new deep architecture. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 689–690, 2011.
18. R. Stanley. An introduction to hyperplane arrangements. In Lect. notes, IAS/Park City Math. Inst., 2004.
19. I. Sutskever and G. E. Hinton. Deep, narrow sigmoid belief networks are universal approximators. Neural Computation, 20(11):2629–2636, 2008.
20. T. Zaslavsky. Facing Up to Arrangements: Face-Count Formulas for Partitions of Space by Hyperplanes. Number no. 154 in Memoirs of the American Mathematical Society. American Mathematical Society, 1975.
21. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. Technical report, arXiv:1311.2901, 2013.