Error bounds for functional approximation and estimation using mixtures of experts

IEEE Transactions on Information Theory

Volumn 44, Issue 3, 1998, Pages 1010-1025

  Zeevi, Assaf J ^a   Meir, Ron ^b   Maiorov, Vitaly ^b  

^a Stanford University   (United States)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Approximation error; Estimation error; Mixture of experts

Indexed keywords

APPROXIMATION THEORY; ERRORS; ESTIMATION; EXPERT SYSTEMS; FUNCTIONS; MATHEMATICAL MODELS; PROBABILITY; REGRESSION ANALYSIS; VECTORS;

APPROXIMATION ERROR; ESTIMATION ERROR; MIXTURE OF EXPERT MODEL; STOCHASTIC ERROR;

NEURAL NETWORKS;

EID: 0032072087     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.669150     Document Type: Article

References (27)

1. R. A. Adams, Sobolev Spaces. New York: Academic, 1975.
2. H. Akakike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, no. 6, pp. 716-723, 1974.
3. S. I. Amari and N. Murata, "Statistical theory of learning curves under entropie loss criterion," Neural Comp., vol. 5, pp. 140-153, 1993.
4. A. R. Barron and T. M. Cover, "Minimum complexity density estimation," IEEE Trans. Inform. Theory, vol. 37, pp. 1034-1054, July 1991.
5. _, "Universal approximation bound for superpositions of a sigmoidal function," IEEE Trans. Inform. Theory, vol. 39, pp. 930-945, 1993.
6. A. R. Barron, "Approximation and estimation bounds for artificial neural networks," Machine Learning, vol. 4, pp. 115-133, 1994.
7. P. Billingsley, Probability and Measure. New York: Wiley, 1979.
8. R. DeVore, R. Howard, and C. A. Micchelli, "Optimal non-linear approximation," Manuscripta Mathematica, vol. 63, pp. 469-478, 1989.
9. S. Geman and C. R. Hwang, "Nonparameteric maximum likelihood estimation by the method of sieves," Ann. Statist., vol. 10, no. 2, pp. 401-414, 1982.
10. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixtures of local experts,"Neural Comp., vol. 3, pp. 79-87, 1991.
11. M. I. Jordan and R. A. Jacobs, "Hierarchical mixtures of experts and the EM algorithm," Neural Comp., vol. 6, pp. 181-214, 1994.
12. N. L. Johnson and S. Kotz, Distributions in Statistics. Continuous Univariate Distributions-2. New York: Wiley, 1972.
13. M. Leshno, V. Lin, A. Pinkus, and S. Schocken, "Multilayer feedforward networks with a non-polynomial activation function can approximate any function," Neural Net., vol. 6, pp. 861-867, 1993.
14. H. Mhaskar, "Neural networks for optimal approximation of smooth and analytic functions," Neural Comp., vol. 8, no. 1, pp. 164-177, 1996.
15. N. Murata, S. Yoshizawa, and S. I. Amari, "Network information criterion - Determining the number of hidden units for an artificial neural network," IEEE Trans. Neural Networks, vol. 5, no. 6, pp. 865-872, 1994.
16. R. A. Redner and H. F. Walker, "Mixture densities, maximum likelihood and the EM algorithm," SIAM Rev., vol. 26, pp. 195-239, 1984.
17. J. Rissanen, "Universal coding, information prediction and estimation," IEEE Trans. Inform. Theory, vol. IT-30, pp. 629-636, July 1984.
18. A. F. Timan, Theory of Approximation of Functions of a Real Variable. New York: Macmillan, 1963.
19. D. M. Titterington, A. F. M. Smith, and U. E. Makov, Statistical Analysis of Finite Mixture Distributions New York: Wiley, 1985.
20. V. Vapnik, Estimation of Dependences Based on Empirical Data. New York: Springer-Verlag, 1982.
21. H. White, Estimation, Inference and Specification Analysis. Cambridge, U.K.: Cambridge Univ. Press, 1994.
22. _, "Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings," Neural Net., vol. 3, pp. 535-549, 1991.
23. _, "Consequences and detection of misspecified nonlinear regression models," J. Amer. Statist. Assoc., vol. 76, pp. 419-433, 1981.
24. J. E. Yukich, M. B. Stinchcombe, and H. White, "Sup-norm approximation bounds for networks through probabilistic methods," IEEE Trans. Inform. Theory, vol. 41, pp. 1021-1027, July 1995.
25. A. J. Zeevi, "A consistent learning property of mixtures of experts," submitted for publication, 1997.
26. A. J. Zeevi, R. Meir, and R. Adler, "Time series prediction using mixtures of experts," in Advances in Neural Information Processing Systems 9, M. Jordan, Ed. Boston, MA: MIT Press, 1997.
27. D. Zwillinger, SRC Standard Mathematical Tables and Formulae, 30th Ed. Boca Raton, FL: CRC, 1996.