Networks for Approximation arid Learning

Proceedings of the IEEE

Volumn 78, Issue 9, 1990, Pages 1481-1497

  Poggio, Tomaso ^a   Girosi, Federico ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; MATHEMATICAL TECHNIQUES--APPROXIMATION THEORY;

NONLINEAR MAPPING; REGULARIZATION THEORY;

NEURAL NETWORKS;

EID: 0025490985     PISSN: 00189219     EISSN: 15582256     Source Type: Journal    
DOI: 10.1109/5.58326     Document Type: Article

References (82)

1. S. Omohundro, “Efficient algorithms with neural network behavior,” Complex Systems, vol. 1, p. 273, 1987.
2. A. R. Barron and R. L. Barron, “Statistical learning networks: A unifying view,” in Symposium on the Interface: Statistics and Computing Science, Reston, Virginia, April 1988.
3. J. Schwartz, “On the effectiveness of backpropagation learning in trainable N2 nets, and of other related form of discrimination learning, preprint, 1988.
4. T. Poggio and the staff, “MIT progress in understanding images,” in Proceedings Image Understanding Workshop, Cambridge, MA, April 1988. Morgan Kaufmann, San Mateo, CA.
5. T. Poggio and the staff, “MIT progress in understanding images,” in Proceedings Image Understanding Workshop, pages 56–74, Palo Alto, CA, May 1989. Morgan Kaufmann, San Mateo, CA.
6. J. Y. Aloimonos, “Unification and integration of visual modules: An extension of the Marr paradigm,” in Proceedings Image Understanding Workshop, pages 507–551, Palo Alto, CA, May 1989. Morgan Kaufmann, San Mateo, CA.
7. J. Moody and C. Darken, “Fast learning in networks of locally-tuned processing units,” Neural Computation, vol. 1, no. 2, pp. 281–294, 1989.
8. T. Poggio, “On optimal nonlinear associative recall,” Biological Cybernetics, vol. 19, pp. 201–209, 1975.
9. L. G. Valiant, “A theory of learnable,” Proc. of the 1984 STOC, pp. 436–445, 1984.
10. J. R. Rice, The Approximation of Functions, Vol. 1. Addison-Wesley, Reading, MA, 1964.
11. T. Poggio and F. Cirosi, “A theory of networks for approximation and learning,” A.I. Memo No. 1140, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1989.
12. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing, ch. 8, pp. 318–362. MIT Press, Cambridge, MA, 1986.
13. G. Cybenko, “Approximation by superposition of a sigmoidal function,” Math. Control Systems Signals, in press, 1989.
14. K. Funahashi, “On the approximate realization of continuous mappings by neural networks,” Neural Networks, vol. 2, pp. 183–192, 1989.
15. G. Cybenko, “Continuous valued neural networks with two hidden layers are sufficient,” Technical report, Dept. of Computer Sciences, Tufts Univ., Medford, MA, 1988.
16. B. Moore and T. Poggio, “Representations properties of multilayer feedforward networks,” in Abstracts of the First Annual INNS Meeting, p. 502, New York, 1988, Pergamon Press.
17. A. N. Kolmogorov, “On the representation of continuous functions of several variables by superposition of continuous functions of one variable and addition,” Dokl. Akad. Nauk SSSR, vol. 114, pp. 953–956, 1957.
18. T. Poggio, “Visual algorithms,” in O. J. Braddick and A. C. Sleigh, editors, Physical and Biological Processing of Images, pp. 128–153, Springer-Verlag, Berlin, 1982.
19. F. Girosi and T. Poggio, “Representation properties of networks: Kolmogorov's theorem is irrelevant,” Neural Computation, vol. 1, no. 4, pp. 465–469, 1989.
20. J. Hadamard, La theorie des equations aux derivees partielles. Editions Scientifiques, Pekin, 1964.
21. A. N. Tikhonov and V. Y. Arsenin, Solutions of Ill-posed Problems. W. H. Winston, Washington, D.C., 1977.
22. C. J. Stone, “Optimal global rates of convergence for non-parametric regression,” Ann. Stat., vol. 10, pp. 1040–1053, 1982.
23. M. Bertero, “Regularizations methods for linear inverse problems,” in C. G. Talenti, editor, Inverse Problems. Springer-Verlag, Berlin, 1986.
24. W. E. L. Crimson, From Images to Surfaces. MIT Press, Cambridge, Mass., 1981.
25. M. Bertero, T. Poggio, and V. Torre, “Ill–posed problems in early vision,” Proceedings of the IEEE, vol. 76, pp. 869–889, 1988.
26. J. L. Marroquin, S. Mitter, and T. Poggio, “Probabilistic solution of ill-posed problems in computational vision,” J. Amer. Stat. Assoc., vol. 82, pp. 76–89, 1987.
27. J. Rissanen, “Modeling by shortest data description,” Automatica, vol. 14, pp. 465–471, 1978.
28. R. J. Solomonoff, “Complexity-based induction systems: Comparison and convergence theorems,” IEEE Transactions on Information Theory, p. 24, 1978.
29. J. G. Harris, “An analog VLSI chip for thin-plate surface interpolation,” in D. S. Touretzky, editor, Advances in Neural Information Processing Systems I. Morgan Kaufmann Publishers, Carnegie Mellon University, 1989.
30. M.J.D. Powell, “Radial basis functions for multivariable interpolation: A review,” in J. C. Mason and M. G. Cox, editors, Algorithms for Approximation. Clarendon Press, Oxford, 1987.
31. R. L. Hardy, “Multiquadric equations of topography and other irregular surfaces,” J. Geophys. Res., vol. 76, pp. 1905–1915, 1971.
32. R. Franke, “Scattered data interpolation: Tests of some method,” Math. Comp., vol. 38(5), pp. 181–200, 1982.
33. E. J. Kansa, “Multiquadrics—a scattered data approximation scheme with applications to computational fluid dynamics—I,” Computers Math. Appl., vol. 19, no. 8/9, pp. 127–145, 1990.
34. D. S. Broomhead and D. Lowe, “Multivariable functional interpolation and adaptive networks,” Complex Systems, vol. 2, pp. 321–355, 1988.
35. S. Renals and R. Rohwer, “Phoneme classification experiments using radial basis functions,” in Proceedings of the International Joint Conference on Neural Networks, pp. 1-461–1-467, Washington, D.C., June 1989, IEEE TAB Neural Network Committee.
36. M. Casdagli, “Nonlinear prediction of chaotic time-series,” Physica D, vol. 35, pp. 335–356, 1989.
37. A. N. Tikhonov, “Solution of incorrectly formulated problems and the regularization method,” Soviet Math. Dokl., vol. 4, pp. 1035–1038, 1963.
38. V. A. Morozov, Methods for Solving Incorrectly Posed Problems. Springer-Verlag, Berlin, 1984.
39. R. Courant and D. Hubert, Methods of Mathematical Physics, Vol. 2. Interscience, London, England, 1962.
40. G. Wahba, Splines Models for Observational Data, Series in Applied Mathematics, Vol. 59, SIAM, Philadelphia, 1990.
41. C. A. Micchelli, “Interpolation of scattered data: Distance matrices and conditionally positive definite functions,” Constr. Approx., vol. 2, pp. 11–22, 1986.
42. N. Dyn, “Interpolation of scattered data by radial functions,” in C. K. Chui, L. L. Schumaker,and F. I. Utreras, editors, Topics in Multivariate Approximation. Academic Press, New York, 1987.
43. W. R. Madych and S. A. Nelson, “Multivariate interpolation and conditionally positive definite functions, II,” Mathematics of Computation, vol. 54, no. 189, pp. 211–230, Jan. 1990.
44. J. Duchon, “Spline minimizing rotation-invariant semi-norms in Sobolev spaces,” in W. Schempp and K. Zeller, editors, Constructive Theory of Functions of Several Variables, Lecture Notes in Mathematics, 571. Springer-Verlag, Berlin, 1977.
45. J. Meinguet, “Multivariate interpolation at arbitrary points made simple,” J. Appl. Math. Phys., vol. 30, pp. 292-304, 1979.
46. R. L. Harder and R. M. Desmarais, “Interpolation using surface splines,” J. Aircraft, vol. 9, pp. 189–191, 1972.
47. S. Bochner, “Vorlesungen ueber Fouriersche Integrale,” in Akademische Verlagsgesellschaft, Leipzig, 1932.
48. A. Yuille and N. Grzywacz, “The motion coherence theory,” in Proceedings of the International Conference on Computer Vision, pp. 344–354, Washington, D.C., December 1988, IEEE Computer Society Press.
49. I. Stakgold, Green's Functions and Boundary Problems. John Wiley and Sons, New York, 1979.
50. F. Girosi and T. Poggio, “Networks and the best approximation property,” Biological Cybernetics, vol. 63, pp. 169–176, 1990.
51. M. Stinchcombe and H. White, “Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions,” in Proceedings of the International Joint Conference on Neural Networks, pp. 1-607–1-611, Washington, D.C., June 1989, IEEE TAB Neural Network Committee.
52. H. L. Resnikoff, “On the psychophysical function,” J. Math. Biol., vol. 2, pp. 265–276, 1975.
53. J. Demmel, “The geometry of ill-conditioning,” J. Complexity, vol. 3, pp. 201–229, 1987.
54. S. G. Mikhlin, The Problem of the Minimum of a Quadratic Functional. Holden-Day, San Francisco, CA, 1965.
55. D. Geiger and F. Girosi, “Parallel and deterministic algorithms for MRFs: surface reconstruction and integration,” in Lecture Notes in Computer Science, Vol. 427: Computer Vision—ECCV 90, O. Faugeras, Ed. Berlin: Springer-Verlag, 1990.
56. F. Girosi, T. Poggio, and B. Caprile, “Extensions of a theory of networks for approximation and learning: Outliers and negative examples,” A.I. Memo 1220, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1990.
57. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, vol. 220, pp. 219–227, 1983.
58. A. Albert, Regression and the Moore-Penrose Pseudoinverse. Academic Press, New York, 1972.
59. T. Poggio and F. Girosi, “Extension of a theory of networks for approximation and learning: Dimensionality reduction and clustering,” A.I. Memo 1167, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1990.
60. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. Wiley, New York, 1973.
61. T. Poggio and S. Edelman, “A network that learns to recognize 3D objects,” Nature, vol. 343, pp. 263–266, 1990.
62. S. Edelman and T. Poggio, “Bringing the grandmother back into the picture: A memory-based view of object recognition,” A. I. Memo 1181, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1990.
63. J. D. Farmer and J. J. Sidorowich, “Exploiting chaos to predict the future and reduce noise,” Technical report, Los Alamos National Laboratory, New Mexico, 1988.
64. A. Lapedes and R. Farber, “Nonlinear signal processing using neural networks: Prediction and system modelling,” Los Alamos National Laboratory LA-UR-87-2662, 1987, submitted to Proc. IEEE.
65. T. Flash and N. Hogan, “The coordination of arm movements: An experiment confirmed mathematical model,” The Journal of Neuroscience, vol. 5(7), pp. 1688–1703, 1985.
66. P. Kanerva, Sparse Distributed Memory, MIT Press, Cambridge, MA, 1988.
67. D. Marr, “A theory of cerebellar cortex,” J. Physiology, vol. 202, pp. 437–470, 1969.
68. J. S. Albus, “A theory of cerebellar functions,” Math. Bio., vol. 10, pp. 25-61, 1971.
69. J. D. Keeler, “Comparison between Kanerva's SDM and Hopfield–type neural networks,” Cognitive Science, vol. 12, pp. 299–329, 1988.
70. T. Kohonen, “Self organized formation of topologically correct feature maps in the brain,” Biological Cybernetics, vol. 43, pp. 59–69, 1982.
71. J. MacQueen, “Some methods of classification and analysis of multivariate observations,” in L. M. LeCam and J. Neyman, editors, Proc. 5th Berkeley Symposium on Math., Stat., and Prob., p. 281. U. California Press, Berkeley, CA, 1967.
72. C. Stanfill and D. Waltz, “Towards memory-based reasoning,” Comm. of the ACM, vol. 29, pp. 1213–1228, 1986.
73. G. G. Lorentz, Approximation of Functions. Chelsea Publishing Co., New York, 1986.
74. C. J. Stone, “Additive regression and other nonparametric models,” Ann. Stat., vol. 13, pp. 689–705, 1985.
75. E. B. Baum, “On the capabilities of multilayer perceptrons,” J. Complexity, vol. 4, pp. 193–215, 1988.
76. E. B. Baum and D. Haussler, “What size net gives valid generalization?” in D. S. Touretzky, editor, Advances in Neural Information Processing Systems I, pp. 81–90. Morgan Kaufmann Publishers, Carnegie Mellon University, 1989.
77. F. Girosi and T. Poggio, “Networks for learning: A view from the theory of approximation of functions,” in Proc. of the Genoa Summer School on Neural Networks and Their Applications, Genoa, Italy, June 1989, Prentice-Hall (in press).
78. G. Wahba, “Practical approximate solutions to linear operator equations when the data are noisy,” SIAM J. Numer. Anal., p. 14, 1977.
79. P. Craven and G. Wahba, “Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross validation,” Numer. Math, vol. 31, pp. 377–403, 1979.
80. I. M. Gelfand and N. Ya. Vilenkin, Generalized Functions. Vol. 4: Applications of Harmonic Analysis, Academic Press, New York, 1964.
81. I. J. Schoenberg, “Metric spaces and positive definite function,” Ann. of Math., vol. 44, pp. 522–536, 1938.
82. J. Stewart, “Positive definite functions and generalizations, an historical survey,” Rocky Mountain J. Math., vol. 6, pp. 409–434, 1976.