Objective functions for training new hidden units in constructive neural networks

IEEE Transactions on Neural Networks

Volumn 8, Issue 5, 1997, Pages 1131-1148

  Kwok, Tin Yau ^b   Yeung, Dit Yan ^a,b  

^a IEEE

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Cascade correlation; Constructive algorithms; Convergence; Input weight freezing; Quickprop

Indexed keywords

COMPUTATIONAL COMPLEXITY; CONVERGENCE OF NUMERICAL METHODS; FUNCTIONS; LEARNING ALGORITHMS; OPTIMIZATION; PROBLEM SOLVING; REGRESSION ANALYSIS;

CONSTRUCTIVE ALGORITHMS; INPUT WEIGHT FREEZING;

FEEDFORWARD NEURAL NETWORKS;

EID: 0031236099     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.623214     Document Type: Article

References (73)

1. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Networks, vol. 4, pp. 740-747, Sept. 1993.
2. E. Fiesler, "Comparative bibliography of ontogenic neural networks," in Proc. Int. Conf. Artificial Neural Networks, vol. 1, Sorrento, Italy, May 1994, pp. 793-796.
3. T. Y. Kwok and D. Y. Yeung, "Constructive algorithms for structure learning in feedforward neural networks for regression problems," IEEE Trans. Neural Networks, vol. 7, pp. 1168-1183, Sept. 1996.
4. D. E. Nelson and S. K. Rogers, "A taxonomy of neural-network optimality," in Proc. IEEE Nat. Aerospace and Electron. Conf., vol. 3, Dayton, OH, May 1992, pp. 894-899.
5. Y. Chauvin, "A backpropagation algorithm with optimal use of hidden units," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 519-526.
6. S. J. Hanson and L. Y. Pratt, "Comparing biases for minimal network construction with backpropagation," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 177-185.
7. A. S. Weigend, D. E. Rumelhart, and B. A. Huberman, "Generalization by weight-elimination with application to forecasting," in Advances in Neural Information Processing Systems 3, R. Lippmann, J. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 875-882.
8. G. H. Golub, M. Heath, and G. Wahba, "Generalized cross-validation as a method for choosing a good ridge parameter," Technometrics, vol. 21, no. 2, pp. 215-223, 1979.
9. W. L. Buntine and A. S. Weigend, "Bayesian backpropagation," Complex Syst., vol. 5, pp. 603-643, 1991.
10. D. J. C. MacKay, "Bayesian interpolation," Neural Computa., vol. 4, no. 3, pp. 415-447, May 1992.
11. R. M. Neal, "Bayesian learning for neural networks," Ph.D. dissertation, Dept. Comput. Sci., Univ. Toronto, Ont., Canada, 1995.
12. H. H. Thodberg, "A review of Bayesian neural networks with an application to near infrared spectroscopy," IEEE Trans. Neural Networks, vol. 7, pp. 56-72, Jan. 1996.
13. P. M. Williams, "Bayesian regularization and pruning using a Laplace prior," Neural Computa., vol. 7, pp. 117-143, 1995.
14. D. J. C. MacKay, "A practical Bayesian framework for backpropagation networks," Neural Computa., vol. 4, no. 3, pp. 448-472, May 1992.
15. E. Alpaydin, "GAL: Networks that grow when they learn and shrink when they forget," Int. Comput. Sci. Instit., Berkeley, CA: TR 91-032, May 1991.
16. G. Deffuant, "Neural units recruitment algorithm for generation of decision trees," in Proc. 1990 IEEE Int. Joint Conf. Neural Networks, vol. 1, San Diego, CA, June 1990, pp. 637-642.
17. M. Frean, "The upstart algorithm: A method for constructing and training feedforward neural networks," Neural Computa., vol. 2, pp. 198-209, 1990.
18. M. Marchand, M. Golea, and P. Ruján, "A convergence theorem for sequential learning in two-layer perceptrons," Europhys. Lett., vol. 11, no. 6, pp. 487-492, 1990.
19. J. H. Friedman, "An overview of predictive learning and function approximation," in From Statistics to Neural Networks: Theory and Pattern Recognition Applications, J. H. Friedman and H. Wechsler, Eds. Berlin: Springer-Verlag, ASI Proc., Subseries F, 1994.
20. H. Akaike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, pp. 716-723, Dec. 1974.
21. A. Barron, "Predicted squared error: A criterion for automatic model selection," in Self-Organizing Methods in Modeling, S. Farlow, Ed. New York: Marcel Dekker, 1984.
22. P. Craven and G. Wahba, "Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation," Numer. Math., vol. 31, pp. 377-403, 1979.
23. J. E. Moody, "Note on generalization, regularization, and architecture selection in nonlinear learning systems," in Proc. 1991 IEEE Wkshp. Neural Networks for Signal Processing, B. H. Juang, S. Y. Kung, and C. A. Kamm, Eds. Princeton, NJ, Sept. 1991, pp. 1-10.
24. J. Rissanen, "Modeling by shortest data description," Automatica, vol. 14, pp. 465-471, 1975.
25. G. Schwartz, "Estimating the dimension of a model," Ann. Statist., vol. 6, pp. 461-464, 1978.
26. J. Moody, "Prediction risk and architecture selection for neural networks," in From Statistics to Neural Networks - Theory and Pattern Recognition Applications, V. Cherkassky, J. H. Friedman, and H. Wechsler, Eds. Springer-Verlag, 1994, vol. 136 of NATO ASI Series F, pp. 147-165.
27. M. Stone, "Cross-validatory choice and assessment of statistical predictions (with discussion)," J. Roy. Statist. Soc. Series B, vol. 36, pp. 111-147, 1974.
28. A. S. Weigend and B. LeBaron, "Evaluating neural-network predictors by bootstrapping," in Proc. Int. Conf. Neural Inform. Processing, vol. 2, Seoul, Korea, Oct. 1994, pp. 1207-1212.
29. B. D. Ripley, "Choosing network complexity," in Probabilistic Reasoning and Bayesian Belief Networks, A. Gammerman, Ed. Alfred Waller, 1995, pp. 97-108.
30. B. D. Ripley, "Statistical ideas for selecting network architectures," in Neural Networks: Artificial Intelligence and Industrial Applications, B. Kappen and S. Gielen, Eds. Berlin: Springer-Verlag, 1995, pp. 183-190.
31. T. Ash, "Dynamic node creation in backpropagation networks," Connection Sci., vol. 1, no. 4, pp. 365-375, 1989.
32. Y. Hirose, K. Yamashita, and S. Hijiya, "Backpropagation algorithm which varies the number of hidden units," Neural Networks, vol. 4, pp. 61-66, 1991.
33. S. E. Fahlman, "Faster learning variations on backpropagation: An empirical study," in Proc. 1988 Connectionist Models Summer School, D. S. Touretzky, G. E. Hinton, and T. J. Sejnowski, Eds. San Mateo, CA: Morgan Kaufmann, 1988, pp. 38-51.
34. A. N. Kolmogorov and S. V. Fomin, Introductory Real Analysis, Dover, 1975.
35. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Contr., Signials, and Syst., vol. 2, pp. 303-314, 1989.
36. K. Hornik, "Some new results on neural-network approximation," Neural Networks, vol. 6, pp. 1069-1072, 1993.
37. E. Hartman, J. Keeler, and J. Kowalski, "Layered neural networks with Gaussian hidden units as universal approximations," Neural Computa., vol. 2, pp. 210-215, 1990.
38. J. Park and I. W. Sandberg, "Approximation and radial basis function networks," Neural Computa., vol. 5, pp. 305-316, 1993.
39. S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 524-532.
40. J. N. Hwang, S. R. Lay, M. Maechler, D. Martin, and J. Schimert, "Regression modeling in backpropagation and projection pursuit learning," IEEE Trans. Neural Networks, vol. 5, pp. 342-353, May 1994.
41. E. Littmann and H. Ritter, "Cascade LLM networks," in Proc. Int. Conf. Artificial Neural Networks, vol. 1, Brighton, U.K., Sept. 1992, pp. 253-257.
42. A. Saha, C. L. Wu, and D. S. Tang, "Approximation, dimension reduction, and nonconvex optimization using linear superpositions of Gaussians," Trans. Comput., vol. 42, no. 10, pp. 1222-1233, Oct. 1993.
43. Y. Zhao and C. G. Atkeson, "Some approximation properties of projection pursuit learning networks," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 936-943.
44. J. L. Yuan and T. L. Fine, "Forecasting demand for electric power," in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 739-746.
45. J. H. Friedman, "Exploratory projection pursuit," J. Amer. Statist. Assoc., vol. 82, no. 397, pp. 249-266, Mar. 1987.
46. O. Fujita, "Optimization of the hidden unit function in feedforward neural networks," Neural Networks, vol. 5, pp. 755-764, 1992.
47. E. B. Baum and D. Haussler, "What size net gives valid generalization?," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 81-90.
48. P. Courrieu, "A convergent generator of neural networks," Neural Networks, vol. 6, no. 6, pp. 835-844, 1993.
49. K. Hornik, "Approximation capabilities of multilayer feedforward networks," Neural Networks, vol. 4, pp. 251-257, 1991.
50. J. Luo, "A bias architecture with rank-expanding algorithm for neural networks supervised learning problem," in Proc. World Congr. Neural Networks, vol. 3, San Diego, CA, June 1994, pp. 742-747.
51. L. O. Hall, A. M. Bensaid, L. P. Clarke, R. P. Velthmzen, M. S. Silbiger, and J. C. Bezdek, "A comparison of neural-network and fuzzy clustering techniques in segmenting magnetic resonance images of the brain," IEEE Trans. Neural Networks, vol. 3, pp. 672-682, Sept. 1992.
52. N. Karunanithi and D. Whitley, "Prediction of software reliability using feedforward and recurrent neural nets," in Proc. Int. Joint Conf. Neural Networks, vol. 1, Baltimore, MD, June 1992, pp. 800-805.
53. S. J. McKenna, I. W. Ricketts, A. Y. Cairns, and K. A. Hussein, "Cascade-correlation neural networks for the classification of cervical cells," in Inst. Elec. Eng. Colloquium on Neural Networks for Image Processing Applicat., London, U.K., Oct. 1992, pp. 5/1-4.
54. N. Simon, H. Corporaal, and E. Kerckhoffs, "Variations on the cascade-correlation learning architecture for fast convergence in robot control," in Proc. 5th Int. Conf. Neural Networks Applicat., Nimes, France, Nov. 1992, pp. 455-464.
55. T. R. Shultz and W. C. Schmidt, "A cascade-correlation model of balance scale phenomena," in Proc. 13th Annu. Conf. Cognitive Sci. Soc., Hillsdale, NJ, 1991, pp. 635-640.
56. D. G. Luenberger, Linear and Nonlinear Programming. Reading, MA: Addison-Wesley, 2nd ed., 1989.
57. J. Park and I. Sandberg, "Universal approximation using radial basis function networks," Neural Computa., vol. 3, pp. 246-257, 1991.
58. G. P. Drago and S. Ridella, "Convergence properties of cascade correlation in function approximation," Neural Comput. Applicat., vol. 2, pp. 142-147, 1994.
59. A. R. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inform. Theory, vol. 39, no. 3, pp. 930-945, May 1993.
60. L. K. Jones, "A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural-network training," Ann. Statist., vol. 20, no. 1, pp. 608-613, 1992.
61. _, "On a conjecture of Huber concerning the convergence of projection pursuit regression," Ann. Statist., vol. 15, no. 2, pp. 880-882, 1987.
62. C. Darken, M. Donahue, L. Gurvits, and E. Sontag, "Rate of approximation results motivated by robust neural-network learning," Tech. Rep., Siemens Corporate Res., Inc., Princeton, NJ, Apr. 1994.
63. V. Ku̇rková and B. Beliczynski, "Incremental approximation by one-hidden-layer neural networks," in Proc. Int. Conf. Artificial Neural Networks, vol. 1, Pans, France, Oct. 1995, pp. 505-510.
64. T. T. Jervis and W. J. Fitzgerald, "Optimization schemes for neural networks," Cambridge Univ. Eng. Dept., U.K., Tech. Rep. CUED/F-INFENG/TR 144, 1993.
65. C. S. Squires and J. W. Shavlik, "Experimental analysis of aspects of the cascade-correlation learning architecture," Comput. Sci. Dept., Univ. Wisconsin-Madison, Madison, WI, Machine Learning Res. Group Working Paper 91-1, 1991.
66. R. S. Crowder, "Cascor.c, C implementation of the cascade-correlation learning algorithm," 1990.
67. R. A Jacobs, "Increased rates of convergence through learning rate adaptation," Neural Networks, vol. 1, pp. 295-307, 1988.
68. T. Y. Kwok and D. Y. Yeung, "Constructive neural networks: Some practical considerations," in Pro. IEEE Int. Conf. Neural Networks, vol. 1, Orlando, FL, June 1994, pp. 198-203.
69. V. K. Rohatgi, Statistical Inference. New York: Wiley, 1984.
70. D. L. Donoho and I. M. Johnstone, "Projection-based approximation and a duality with kernel methods," Ann. Statist., vol. 17, no. 1, pp. 58-106, 1989.
71. P. J. Huber, Robust Statistics. New York: Wiley, 1981.
72. E. Kreyszig, Introductory Functional Analysis with Applications. New York: Wiley 1989.
73. D. H. Griffel, Applied Functional Analysis. Chichester, U.K.: Ellis Horwood, 1981.