Deep vs. shallow networks: An approximation theory perspective

Analysis and Applications

Volumn 14, Issue 6, 2016, Pages 829-848

  Mhaskar, H N ^a,b   Poggio, T ^c  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b CLAREMONT GRADUATE UNIVERSITY   (United States)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

blessed representation; Deep and shallow networks; Gaussian networks; ReLU networks

Indexed keywords

EID: 84991247403     PISSN: 02195305     EISSN: 17936861     Source Type: Journal    
DOI: 10.1142/S0219530516400042     Document Type: Article

References (27)

1. G. E. Andrews, R. Askey and R. Roy, Special Functions, Vol. 71 (Cambridge University Press, 1999).
2. C. K. Chui, An Introduction to Wavelets (Academic Press, 1992).
3. C. K. Chui, X. Li and H. N. Mhaskar, Neural networks for localized approximation, Math. Comput. 63(208) (1994) 607-623.
4. -, Limitations of the approximation capabilities of neural networks with one hidden layer, Adv. Comput. Math. 5(1) (1996) 233-243.
5. R. A. DeVore, R. Howard and C. A. Micchelli, Optimal nonlinear approximation, Manuscripta Math. 63(4) (1989) 469-478.
6. D. L. Donoho, High-dimensional data analysis: The curses and blessings of dimensionality, Lecture, AMS Math Challenges Lecture Conference, Los Angeles, USA (2000) 32 pp.
7. R. Eldan and O. Shamir, The power of depth for feedforward neural networks, preprint (2015); arXiv:1512.03965.
8. G. Freud. On direct and converse theorems in the theory of weighted polynomial approximation, Math. Z. 126(2) (1972) 123-134.
9. K. Fukushima, Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position, Biol. Cybern. 36(4) (1980) 193-202.
10. J. T. Hastad, Computational Limitations for Small Depth Circuits (MIT Press, 1987).
11. K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, preprint (2015); arXiv:1512.03385v1 [cs.CV].
12. Y. LeCun, Y. Bengio and G. Hinton, Deep learning, Nature 521(7553) (2015) 436-444.
13. H. N. Mhaskar, Neural networks for optimal approximation of smooth and analytic functions, Neural Comput. 8(1) (1996) 164-177.
14. -, On the degree of approximation in multivariate weighted approximation, in Advanced Problems in Constructive Approximation (Springer, 2003), pp. 129-141.
15. -, When is approximation by Gaussian networks necessarily a linear process? Neural Netw. 17(7) (2004) 989-1001.
16. -, A Markov-Bernstein inequality for Gaussian networks, in Trends and Applications in Constructive Approximation (Springer, 2005), pp. 165-180.
17. -, Weighted quadrature formulas and approximation by zonal function networks on the sphere, J. Complex. 22(3) (2006) 348-370.
18. -, Eignets for function approximation on manifolds, Appl. Comput. Harmon. Anal. 29(1) (2010) 63-87.
19. -, Local approximation using Hermite functions, to appear in Progress in Approximation Theory and Applicable Complex Analysis - In the Memory of Q. I. Rahman (Springer); arXiv:1608.01959.
20. H. N. Mhaskar, Q. Liao and T. Poggio, Learning real and boolean functions: When is deep better than shallow, preprint (2016); arXiv:1603.00988.
21. T. Poggio, F. Anselmi and L. Rosasco, I-theory on depth vs width: Hierarchical function composition, CBMM memo 041 (2015).
22. T. Poggio and S. Smale, The mathematics of learning: Dealing with data, Not. Amer. Mathe. Soc. 50(5) (2003) 537-544.
23. M. Riesenhuber and T. Poggio, Hierarchical models of object recognition in cortex, Nature Neurosci. 2(11) (1999) 1019-1025.
24. D. L. Ruderman, Origins of scaling in natural images, Vision Res. 37(23) (1997) 3385-3398.
25. E. M. Stein, Singular Integrals and Differentiability Properties of Functions, Princeton Mathematical Series, No. 30 (Princeton University Press, 1970).
26. G. Szegö, Orthogonal Polynomials, Colloquium Publications, Vol. 23 (American Mathematical Society, Providence, 1975).
27. M. Telgarsky, Representation benefits of deep feedforward networks, preprint (2015); arXiv:1509.08101v2 [cs.LG].