On overfitting, generalization, and randomly expanded training sets

IEEE Transactions on Neural Networks

Volumn 11, Issue 5, 2000, Pages 1050-1057

  Karystinos, George N ^a,b   Pados, Dimitris A ^a,b  

^a IEEE   (United States)

^b the State University of New York   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLUSTERING METHODS; GAUSSIAN DISTRIBUTIONS; STOCHASTIC APPROXIMATION METHODS;

APPROXIMATION THEORY; BACKPROPAGATION; CURVE FITTING; DATA STRUCTURES; PROBABILITY DENSITY FUNCTION; RANDOM PROCESSES; SPURIOUS SIGNAL NOISE;

MULTILAYER NEURAL NETWORKS;

EID: 0034271110     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.870038     Document Type: Article

References (39)

1. P. J. Werbos, "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences," Ph.D. dissertation, Harvard Univ., Cambridge, MA, Aug. 1974.
2. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Cambridge, MA: MIT Press, 1986, vol. I, ch. 8.
3. H. White, "Some asymptotic results for learning in single hidden-layer feedforward network models," J. Amer. Statist. Assoc., vol. 84, no. 408, pp. 1003-1013, Dec. 1989.
4. D. A. Pados and P. Papantoni-Kazakos, "New nonleast-squares neural network learning algorithms for hypothesis testing," IEEE Trans. Neural Networks, vol. 6, pp. 596-609, May 1995.
5. F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Washington, DC: Spartan, 1962.
6. H. Robbins and S. Monro, "A stochastic approximation method," Ann. Math. Statist., vol. 22, pp. 400-407, 1951.
7. J. Kiefer and J. Wolfowitz, "Stochastic estimation of the maximum of a regression function," Ann. Math. Statist., vol. 23, pp. 462-466, 1952.
8. D. J. Sakrison, "Stochastic approximation: A recursive method for solving regression problems," in Advances in Communication Systems 2. New York: Academic, 1966, vol. 2, pp. 51-106.
9. J. R. Blum, "Approximation methods which converge with probability one," Ann. Math. Statist., vol. 25, pp. 382-386, 1954.
10. S. J. Farlow, Self Organizing Methods in Modeling: GMDH-Type Algorithms. New York: Marcel Dekker, 1984.
11. M. G. Bello, "Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks," IEEE Trans. Neural Networks, vol. 3, pp. 864-875, Nov. 1992.
12. H. Akaike, "Statistical predictor identification," Ann. Inst. Statist. Math., vol. 22, no. 2, pp. 203-217, 1970.
13. J. Rissannen, "Modeling by shortest data description," Automatica, vol. 14, no. 5, pp. 465-471, Sept. 1978.
14. G. Schwarz, "Estimating the dimension of a model," Ann. Statist., vol. 6, no. 2, pp. 461-464, Mar. 1977.
15. A. R. Barron and R. Barron, "Statistical learning networks: A unifying view," in Proc. 20th Symp. Interface, E. J. Wegman, D. I. Gantz, and J. J. Miller, Eds., 1989, pp. 192-202.
16. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Networks, vol. 4, pp. 740-747, Sept. 1993.
17. Y. le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, "Backpropagation applied to handwritten zip code recognition," Neural Comput., vol. 1, no. 4, pp. 541-551, 1989.
18. S. M. Peeling, R. K. Moore, and M. J. Tomlinson, "The multilayer perceptron as a tool for speech pattern processing research," in Proc. IoA Autumn Conf. Speech Hearing, 1986.
19. D. C. Plaut, S. J. Nowlan, and G. E. Hinton, "Experiments on learning by backpropagation," Carnegie-Mellon Univ., Pittsburgh, PA, Tech. Rep., 1986.
20. L. Holmstrom and P. Koistinen, "Using additive noise in backpropagation training," IEEE Trans. Neural Networks, vol. 3, pp. 24-38, Jan. 1992.
21. E. Parzen, "On estimation of a probability density function and mode," Ann. Math. Statist., vol. 33, pp. 1065-1076, 1962.
22. M. Rosenblatt, "Remarks on some nonparametric estimates of a density function," Ann. Math. Statist., vol. 27, pp. 832-837, 1956.
23. T. Cacoullos, "Estimation of a multivariate density," Ann. Inst. Statist. Math., vol. 18, no. 2, pp. 179-189, 1966.
24. C. M. Bishop, "Training with noise is equivalent to Tikhonov regularization," Neural Comput., vol. 7, no. 1, pp. 108-116, 1995.
25. K. Matsuoka, "Noise injection into inputs in backpropagation learning," IEEE Trans. Syst., Man, Cybern., vol. 22, pp. 436-440, May 1992.
26. J. Van Ness, "On the dominance of nonparametric Bayes rule discriminant algorithms in high dimensions," Pattern Recognit., vol. 12, no. 6, pp. 355-368, 1980.
27. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
28. A. Papoulis, Probability, Random Variables, and Stochastic Processes. New York: McGraw-Hill, 1991.
29. E. J. Kelly, "An adaptive detection algorithm," IEEE Trans. Aerospace Electron. Syst., vol. AE-22, pp. 115-127, Mar. 1986.
30. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc., vol. 39, pp. 1-38, 1977.
31. R. P. W. Duin, "On the choice of smoothing parameters for Parzen estimators of probability density functions," IEEE Trans. Comput., vol. 25, pp. 1175-1179, Nov. 1976.
32. Y. S. Chow, S. Geman, and L. D. Wu, "Consistent cross-validated density estimation," Ann. Statist., vol. 11, no. 1, pp. 25-38, Mar. 1983.
33. J. S. Marron, "An asymptotically efficient solution to the bandwidth problem of kernel density estimation," Ann. Statist., vol. 13, no. 3, pp. 1011-1023, Sept. 1985.
34. J. H. Friedman, "Multivariate adaptive regression splines,"Ann. Statist., vol. 19, no. 1, pp. 1-141, Mar. 1991.
35. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
36. J. Makhoul, S. Roucos, and H. Gish, "Vector quantization in speech coding," Proc. IEEE, vol. 73, Nov. 1985.
37. M. R. Anderberg, Cluster Analysis for Applications. New York: Academic, 1973.
38. P. Danos and E. A. Imhoff Jr., Introduction to Financial Accounting. Boston, MA: Irwin, 1994.
39. Z. Bodie, A. Kane, and A. J. Marcus, Essentials of Investments. Boston, MA: Irwin, 1995.