An iterative pruning algorithm for feedforward neural networks

IEEE Transactions on Neural Networks

Volumn 8, Issue 3, 1997, Pages 519-531

  Castellano, Giovanna ^b   Fanelli, Anna Maria ^a,c   Pelillo, Marcello ^a,d  

^a IEEE

^b CNR   (Italy)

^c UNIVERSITY OF BARI   (Italy)

^d CA' FOSCARI UNIVERSITY OF VENICE   (Italy)

Author keywords

Feedforward neural networks; Generalization; Hidden neurons; Iterative methods; Least squares methods; Network pruning; Pattern recognition; Structure simplification

Indexed keywords

ALGORITHMS; ITERATIVE METHODS; LEAST SQUARES APPROXIMATIONS; PATTERN RECOGNITION; PROBLEM SOLVING;

CONJUGATE GRADIENT ALGORITHMS; NETWORK PRUNING;

FEEDFORWARD NEURAL NETWORKS;

EID: 0031142667     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.572092     Document Type: Article

References (55)

1. D. R. Hush and B. G. Horne, "Progress in supervised neural networks," IEEE Signal Processing Mag., vol. 10, pp. 8-39, 1993.
2. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing - Vol. 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, pp. 318-362.
3. S. Y. Kung and J. N. Hwang, "An algebraic projection analysis for optimal hidden units size and learning rates in backpropagation learning," in Proc. IEEE Int. Conf. Neural Networks, San Diego, CA, vol. 1, 1988, pp. 363-370.
4. D. C. Plaut and G. E. Hinton, "Learning sets of filters using backpropagation," Comput. Speech Language, vol. 2, pp. 35-61, 1987.
5. D. J. Burr, "Experiments on neural net recognition of spoken and written text," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-36, pp. 1162-1168, 1988.
6. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by backpropagating errors," Nature, vol. 323, pp. 533-536, 1986.
7. X.-H. Yu, "Can backpropagation error surface not have local minima," IEEE Trans. Neural Networks, vol. 3, pp. 1019-1021, 1992.
8. R. P. Lippmann, "An introduction to computing with neural nets," IEEE Acoust., Speech, Signal Processing Mag., vol. 4, pp. 4-22, 1987.
9. M. D. Emmerson and R. I. Damper, "Determining and improving the fault tolerance of multilayer perceptrons in a pattern-recognition application," IEEE Trans. Neural Networks, vol. 4, pp. 788-793, 1993.
10. E. B. Baum and D. Haussler, "What size net gives valid generalization?" Neural Computa., vol. 1, pp. 151-160, 1989.
11. J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield, "Large automatic learning, rule extraction, and generalization," Complex Syst., vol. 1, pp. 877-922, 1987.
12. Y. Le Cun, "Generalization and network design strategies," in Connectionism in Perspective, R. Pfeifer, Z. Schreter, F. Fogelman-Soulie, and L. Steels, Eds. Amsterdam: Elsevier, 1989, pp. 143-155.
13. Y. Chauvin, "Generalization performance of overtrained backpropagation networks," in Neural Networks - Proc. EURASIP Wkshp. 1990, L. B. Almeida and C. J. Wellekens, Eds. Berlin: Springer-Verlag, 1990, pp. 46-55.
14. G. G. Towell, M. K. Craven, and J. W. Shavlik, "Constructive induction in knowledge-based neural networks," in Proc. 8th Int. Wkshp. Machine Learning, L. A. Birnbaum and G. C. Collins, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 213-217.
15. J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation. Redwood City, CA: Addison-Wesley, 1991.
16. Y. H. Hu, "Configuration of feedforward multilayer perceptron network," unpublished manuscript, 1993.
17. S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 524-532.
18. S. I Gallant, "Optimal linear discriminants," in Proc. 8th Int. Conf. Pattern Recognition, Paris, France, 1986, pp. 849-852.
19. T. Ash, "Dynamic node creation in backpropagation networks," Connection Sci., vol. 1, no. 4, pp. 365-375, 1989.
20. M. Mézard and J.-P. Nadal, "Learning in feedforward layered networks: The Tiling algorithm," J. Phys. A, vol. 22, pp. 2191-2204, 1989.
21. R. Reed, "Pruning algorithms - A review," IEEE Trans. Neural Networks, vol. 4, pp. 740-747, 1993.
22. S. C. Huang and Y. F. Huang, "Bounds on the number of hidden neurons in multilayer perceptrons," IEEE Trans. Neural Networks, vol. 2, pp. 47-55, 1991.
23. G. E. Hinton, "Connectionist learning procedures, "Artificial Intell., vol. 40, no. 1, pp. 143-150, 1989.
24. Y. Chauvin, "A backpropagation algorithm with optimal use of hidden units," in Advances in Neural Information Processing Systems 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 519-526.
25. A. S. Weigend, D. E. Rumelhart, and B. A. Huberman, "Generalization by weight-elimination with application to forecasting," in Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 875-882.
26. J. K. Kruschke, "Creating local and distributed bottlenecks in hidden layers of backpropagation networks," in Proc. 1988 Connectionist Models Summer School, D. S. Touretzky, G. E. Hinton, and T. J. Sejnowski, Eds. San Mateo, CA: Morgan Kaufmann, 1988, pp. 120-126.
27. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees. Belmont, CA: Wadsworth, 1984.
28. J. Sietsma and R. J. F. Dow, "Neural net pruning: Why and how," in Proc. IEEE Int. Conf. Neural Networks, San Diego, CA, vol. 1, 1988, pp. 325-333.
29. _, "Creating artificial neural networks that generalize," Neural Networks, vol. 4, pp. 67-79, 1991.
30. Y. Le Cun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in Advances in Neural Information Processing Systems 2, D. S. Touretsky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 598-605.
31. B. Hassibi and D. G. Stork, "Second-order derivatives for network pruning: Optimal brain surgeon," in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 164-171.
32. M. C. Mozer and P. Smolensky, "Using relevance to reduce network size automatically," Connection Sci., vol. 1, no. 1, pp. 3-16, 1989.
33. E. D. Karnin, "A simple procedure for pruning backpropagation trained neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 239-242, 1990.
34. A. Burkitt, "Optimization of the architecture of feedforward neural networks with hidden layers by unit elimination," Complex Syst., vol. 5, pp. 371-380, 1991.
35. F. L. Chung and T. Lee, "A node pruning algorithm for backpropagation networks," Int. J. Neural Syst., vol. 3, no. 3, pp. 301-314, 1992.
36. G. Castellano, A. M. Fanelli, and M. Pelillo, "Pruning in recurrent neural networks," in Proc. Int. Conf. Artificial Neural Networks (ICANN'94), Sorrento, Italy, 1994, pp. 451-454.
37. S. Y. Kung and Y. H. Hu, "A Frobenius approximation reduction method (FARM) for determining optimal number of hidden units," in Proc. Int. J. Conf. Neural Networks, Seattle, WA, vol. 2, 1991, pp. 163-168.
38. Q. Xue, Y. H. Hu, and W. J. Tompkins, "Analyzes of the hidden units of backpropagation model by singular value decomposition (SVD)," in Proc. Int. J. Conf. Neural Networks, Washington, D.C., vol. 1, 1990, pp. 739-742.
39. Y. H. Hu, Q. Xue, and W. J. Tompkins, "Structural simplification of a feedforward multilayer perceptron artificial neural network," in Proc. Int. Conf. Acoust., Speech, Signal Processing, Toronto, Canada, 1991, pp. 1061-1064.
40. M. A. Sartori and P. J. Antsaklis, "A simple method to derive bounds on the size and to train multilayer neural networks," IEEE Trans. Neural Networks, vol. 2, pp. 467-471, 1991.
41. G. H. Golub and C. F. Van Loan, Matrix Computations. Baltimore, MD: Johns Hopkins Univ. Press, 1989.
42. A. Björck, "Methods for sparse linear least-squares problems," in Sparse Matrix Computations, J. R. Bunch and D. J. Rose, Eds. New York: Academic, 1976, pp. 177-199.
43. I. S. Duff, "A survey of sparse matrix research," Proc. IEEE, vol. 65, no. 4, pp. 500-535, 1977.
44. A. Björck and T. Elfving, "Accelerated projection methods for computing pseudoinverse solutions of systems of linear equations," BIT, vol. 19, pp. 145-163, 1979.
45. G. Orlandi, F. Piazza, A. Uncini, E. Luminari, and A. Ascone, "Dynamic pruning in artificial neural networks," in Parallel Architectures and Neural Networks. E. R. Caianello, Ed., Singapore: World Scientific, 1991, pp. 199-208.
46. K. Murase, Y. Matsunaga, and Y. Nakade, "A backpropagation algorithm which automatically determines the number of association units," in Proc. Int. J. Conf. Neural Networks, Singapore, pp. 783-788.
47. T.-C. Lee, Structure Level Adaptation for Artificial Neural Networks. Boston, MA: Kluwer, 1991.
48. A. Sperduti and A. Starita, "Speed up learning and network optimization with extended backpropagation," Neural Networks, vol. 6, pp. 365-383, 1993.
49. L. Niles, H. Silverman, J. Tajchman, and M. Bush, "How limited training data can allow a neural network to outperform an optimal statistical classifier," in Proc. Int. Conf. Acoust., Speech, Signal Processing, Glasgow, Scotland, vol. 1, 1989, pp. 17-20.
50. P. Burrascano, "Learning vector quantization for the probabilistic neural network," IEEE Trans. Neural Networks, vol. 2, pp. 458-461, 1991.
51. M. J. J. Holt, "Comparison of generalization in multilayer perceptrons with the log-likelihood and least-squares cost functions," in Proc. 11th Int. Conf. Pattern Recognition, The Hague, The Netherlands, vol. 2, 1992, pp. 17-20.
52. W. T. Lee and M. F. Tenorio, "On an asymptotically optimal adaptive classifier design criterion," IEEE Trans. Pattern Anal. Machine Intell., vol. 15, pp. 312-318, 1993.
53. N. Morgan and H. Bourlard, "Generalization and parameter estimation in feedforward nets: Some experiments," in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 630-637.
54. A. S. Weigend, B. A. Huberman, and D. E. Rumelhart, "Predicting the future: A connectionist approach," Int. J. Neural Syst., vol. 1, no. 3, pp. 193-209, 1990.
55. W. Zinnoff, F. Hergert, and H. G. Zimmermann, "Improving model selection by nonconvergent methods," Neural Networks, vol. 6, pp. 771-783, 1993.