Neural network with unbounded activation functions is universal approximator

Applied and Computational Harmonic Analysis

Volumn 43, Issue 2, 2017, Pages 233-268

  Sonoda, Sho ^a   Murata, Noboru ^a  

^a WASEDA UNIVERSITY   (Japan)

Author keywords

Admissibility condition; Backprojection filter; Bounded extension to L2; Integral representation; Lizorkin distribution; Neural network; Radon transform; Rectified linear unit (ReLU); Ridgelet transform; Universal approximation

Indexed keywords

BACKPROPAGATION; CHEMICAL ACTIVATION; LOW PASS FILTERS; NEURAL NETWORKS;

ADMISSIBILITY CONDITION; BACK-PROJECTION; BOUNDED EXTENSION TO L2; INTEGRAL REPRESENTATION; LINEAR UNITS; LIZORKIN DISTRIBUTION; RADON TRANSFORM; RIDGELET TRANSFORMS; UNIVERSAL APPROXIMATION;

MATHEMATICAL TRANSFORMATIONS;

EID: 84960887035     PISSN: 10635203     EISSN: 1096603X     Source Type: Journal    
DOI: 10.1016/j.acha.2015.12.005     Document Type: Article

References (43)

1. Murata, N., An integral representation of functions using three-layered betworks and their approximation bounds. Neural Netw. 9:6 (1996), 947–956, 10.1016/0893-6080(96)00000-7.
2. Sonoda, S., Murata, N., Sampling hidden parameters from oracle distribution. 24th Int. Conf. Artif. Neural Networks LNCS, vol. 8681, 2014, Springer International Publishing, 539–546, 10.1007/978-3-319-11179-7_68.
3. Glorot, X., Bordes, A., Bengio, Y., Deep sparse rectifier neural networks. 14th Int. Conf. Artif. Intell. Stat., vol. 15, JMLR W&CP, 2011, 315–323.
4. Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y., Maxout networks. 30th Int. Conf. Mach. Learn., vol. 28, JMLR W&CP, 2013, 1319–1327.
5. Dahl, G.E., Sainath, T.N., Hinton, G.E., Improving deep neural networks for LVCSR using rectified linear units and dropout. 2013 IEEE Int. Conf., Acoust. Speech Signal Process, 2013, IEEE, 8609–8613, 10.1109/ICASSP.2013.6639346.
6. Maas, A.L., Hannun, A.Y., Ng, A.Y., Rectifier nonlinearities improve neural network acoustic models. ICML 2013 Work. Deep Learn., Audio, Speech, Lang. Process, 2013.
7. Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y., What is the best multi-stage architecture for object recognition?. 2009 IEEE 12th Int. Conf. Comput. Vision, 2009, IEEE, 2146–2153, 10.1109/ICCV.2009.5459469.
8. Krizhevsky, A., Sutskever, I., Hinton, G.E., ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst., vol. 25, 2012, Curran Associates, Inc., 1097–1105.
9. Zeiler, M.D., Ranzato, M., Monga, R., Mao, M.Z., Yang, K., Le Viet, Q., Nguyen, P., Senior, A.W., Vanhoucke, V., Dean, J., Hinton, G.E., On rectified linear units for speech processing. 2013 IEEE Int. Conf. Acoust. Speech Signal Process., 2013, IEEE, 3517–3521, 10.1109/ICASSP.2013.6638312.
10. Mhaskar, H., Micchelli, C.A., Approximation by superposition of sigmoidal and radial basis functions. Adv. in Appl. Math. 13:3 (1992), 350–373, 10.1016/0196-8858(92)90016-P.
11. Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S., Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw. 6:6 (1993), 861–867, 10.1016/S0893-6080(05)80131-5.
12. Pinkus, A., Approximation theory of the MLP model in neural networks. Acta Numer. 8 (1999), 143–195, 10.1017/S0962492900002919.
13. Ito, Y., Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory. Neural Netw. 4:3 (1991), 385–394, 10.1016/0893-6080(91)90075-G.
14. Kainen, P.C., Kůrková, V., Vogt, A., A Sobolev-type upper bound for rates of approximation by linear combinations of Heaviside plane waves. J. Approx. Theory 147:1 (2007), 1–10, 10.1016/j.jat.2006.12.009.
15. Kůrková, V., Complexity estimates based on integral transforms induced by computational units. Neural Netw. 33 (2012), 160–167, 10.1016/j.neunet.2012.05.002.
16. Carroll, S.M., Dickinson, B.W., Construction of neural nets using the Radon transform. Int. Jt. Conf. Neural Networks, vol. 1, 1989, IEEE, 607–611, 10.1109/IJCNN.1989.118639.
17. Helgason, S., Integral Geometry and Radon Transforms. 2011, Springer-Verlag, New York, 10.1007/978-1-4419-6055-9.
18. Irie, B., Miyake, S., Capabilities of three-layered perceptrons. IEEE Int. Conf. Neural Networks, 1988, IEEE, 641–648, 10.1109/ICNN.1988.23901.
19. Funahashi, K.-I., On the approximate realization of continuous mappings by neural networks. Neural Netw. 2:3 (1989), 183–192, 10.1016/0893-6080(89)90003-8.
20. Jones, L.K., A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20:1 (1992), 608–613, 10.1214/aos/1176348546.
21. Barron, A.R., Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39:3 (1993), 930–945, 10.1109/18.256500.
22. Kainen, P.C., Kůrková, V., Sanguineti, M., Approximating multivariable functions by feedforward neural nets. Handb. Neural Inf. Process., vol. 49, 2013, Springer, Berlin, Heidelberg, 143–181, 10.1007/978-3-642-36657-4.
23. Candès, E.J., Harmonic analysis of neural networks. Appl. Comput. Harmon. Anal. 6:2 (1999), 197–218, 10.1006/acha.1998.0248.
24. Candès, E.J., Ridgelets: theory and applications. Ph.D. thesis, 1998, Standford University.
25. p-spaces. J. Fourier Anal. Appl. 4:2 (1998), 175–197, 10.1007/BF02475988.
26. n. Proc. Natl. Acad. Sci. USA 96:5 (1999), 1828–1833, 10.1073/pnas.96.5.1828.
27. Donoho, D.L., Ridge functions and orthonormal ridgelets. J. Approx. Theory 111:2 (2001), 143–179, 10.1006/jath.2001.3568.
28. Starck, J.-L., Murtagh, F., Fadili, J.M., The ridgelet and curvelet transforms. Sparse Image Signal Process. Wavelets, Curvelets, Morphol. Divers., 2010, Cambridge University Press, 89–118, 10.1017/CBO9780511730344.006.
29. Rubin, B., Convolution backprojection method for the k-plane transform, and Calderón's identity for ridgelet transforms. Appl. Comput. Harmon. Anal. 16:3 (2004), 231–242, 10.1016/j.acha.2004.03.003.
30. Kostadinova, S., Pilipović, S., Saneva, K., Vindas, J., The ridgelet transform of distributions. Integral Transforms Spec. Funct. 25:5 (2014), 344–358, 10.1080/10652469.2013.853057.
31. Kostadinova, S., Pilipović, S., Saneva, K., Vindas, J., The ridgelet transform and quasiasymptotic behavior of distributions. Oper. Theory Adv. Appl. 245 (2015), 185–197, 10.1007/978-3-319-14618-8_13.
32. Schwartz, L., Théorie des Distributions. nouvelle edition, 1966, Hermann, Paris.
33. Trèves, F., Tological Vector Spaces, Distributions and Kernels. 1967, Academic Press.
34. Rudin, W., Functional Analysis. 2nd edition Higher Mathematics Series, 1991, McGraw–Hill Education.
35. Brezis, H., Functional Analysis, Sobolev Spaces and Partial Differential Equations. 1st edition Universitext, 2011, Springer-Verlag, New York, 10.1007/978-0-387-70914-7.
36. Yosida, K., Functional Analysis. 6th edition, 1995, Springer-Verlag, Berlin, Heidelberg, 10.1007/978-3-642-61859-8.
37. Yuan, W., Sickel, W., Yang, D., Morrey and Campanato Meet Besov, Lizorkin and Triebel. Lecture Notes in Mathematics, 2010, Springer, Berlin, Heidelberg, 10.1007/978-3-642-14606-0.
38. Holschneider, M., Wavelets: An Analysis Tool. Oxford Mathematical Monographs, 1995, The Clarendon Press.
39. Hertle, A., Continuity of the radon transform and its inverse on Euclidean space. Math. Z. 184:2 (1983), 165–192, 10.1007/BF01252856.
40. Grafakos, L., Classical Fourier Analysis. 2nd edition Graduate Texts in Mathematics, 2008, Springer, New York, 10.1007/978-0-387-09432-8.
41. Gel'fand, I.M., Shilov, G.E., Properties and Operations, Generalized Functions, vol. 1. 1964, Academic Press, New York.
42. Shepp, L.A., Logan, B.F., The Fourier reconstruction of a head section. IEEE Trans. Nucl. Sci. 21:3 (1974), 21–43, 10.1109/TNS.1974.6499235.
43. Stein, E.M., Singular Integrals and Differentiability Properties of Functions. Princeton Mathematical Series (PMS), 1970, Princeton University Press.