A new adaptive merging and growing algorithm for designing artificial neural networks

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 39, Issue 3, 2009, Pages 705-722

  Islam, Md Monirul ^a,b   Sattar, Md Abdus ^a   Amin, Md Faijul ^b,c   Yao, Xin ^d,e   Murase, Kazuyuki ^b  

^a BANGLADESH UNIVERSITY OF ENGINEERING AND TECHNOLOGY   (Bangladesh)

^b UNIVERSITY OF FUKUI   (Japan)

^c KHULNA UNIVERSITY OF ENGINEERING AND TECHNOLOGY   (Bangladesh)

^d UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^e UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

Author keywords

Adding neurons; Artificial neural network (ANN) design; Generalization ability; Merging neurons; Retraining

Indexed keywords

ADAPTIVE STRATEGIES; ADDING NEURONS; ARTIFICIAL NEURAL NETWORK (ANN) DESIGN; BENCH-MARK PROBLEMS; BREAST CANCERS; CAN DESIGNS; CREDIT CARDS; DESIGN PROCESS; FIXED STRATEGIES; GENERALIZATION ABILITY; HIDDEN NEURONS; LEARNING ABILITIES; MACHINE-LEARNING; MERGE OPERATIONS; MIXED-MODE OPERATIONS; OTHER ALGORITHMS; RETRAINING; TRAINING PROCESS;

ADAPTIVE ALGORITHMS; BACKPROPAGATION; DESIGN; EDUCATION; LEARNING SYSTEMS; MERGING; NEURAL NETWORKS; NEURONS;

ARCHITECTURAL DESIGN;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; GENE; GENERAL ASPECTS OF DISEASE; STATISTICS;

ALGORITHMS; DISEASE; GENES; NEURAL NETWORKS (COMPUTER); STATISTICS AS TOPIC;

EID: 67349203854     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2008.2008724     Document Type: Article

References (77)

1. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing, vol. I, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, pp. 318-362.
2. T. Y. Kwok and D. Y. Yeung, "Constructive algorithms for structure learning in feedforward neural networks for regression problems," IEEE Trans. Neural Netw., vol. 8, no. 3, pp. 630-645, May 1997.
3. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Netw. vol. 4, no. 5, pp. 740-747, Sep. 1993.
4. F. Girosi, M. Jones, and T. Poggio, "Regularization theory and neural networks architectures," Neural Comput., vol. 7, no. 2, pp. 219-269, Mar. 1995.
5. J. H. Holland, Adaptation in Natural and Artificial Systems. Ann Arbor, MI: Univ. Michigan Press, 1975.
6. L. J. Fogel, A. J. Owens, and M. J. Walsh, Artificial Intelligence Through Simulated Evolution. New York: Wiley, 1966.
7. D. B. Fogel, Evolutionary Computation: Toward a New Philosophy of Machine Intelligence. New York: IEEE Press, 1995.
8. H.-P. Schwefel, Numerical Optimization of Computer Models. Chichester, U.K.: Wiley, 1981.
9. A. S. Weigend, D. E. Rumelhart, and B. A. Huberman, "Generalization by weight-elimination with application to forecasting," in Proc. Advances Neural Inform. Process. Syst., R. Lippmann, J. Moody, and D. S. Touretzky, Eds., 1991, vol. 3, pp. 875-882.
10. C. Schittenkopf, G. Deco, and W. Brauer, "Two strategies to avoid overfitting in feedforward neural networks," Neural Netw., vol. 10, pp. 804-818, 1997.
11. D. J. C. MacKay, "Bayesian interpolation," Neural Comput., vol. 4, no. 3, pp. 415-447, May 1992.
12. P. Lauret, E. Fock, and T. A. Mara, "A node pruning algorithm based on a Fourier amplitude sensitivity test method," IEEE Trans. Neural Netw., vol. 17, no. 2, pp. 273-293, Mar. 2006.
13. D. J. C. MacKay, "A practical Bayesian framework for backpropagation networks," Neural Comput., vol. 4, no. 3, pp. 448-472, May 1992.
14. P. J. Angeline, G. M. Saunders, and J. B. Pollack, "An evolutionary algorithm that constructs recurrent neural networks," IEEE Trans. Neural Netw., vol. 5, no. 1, pp. 54-65, Jan. 1994.
15. S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Englewood Cliffs, NJ: Prentice-Hall, 1995.
16. J. P. Nadal, "Study of a growth algorithm for a feedforwaed network," Int. J. Neural Syst., vol. 1, no. 1, pp. 55-59, 1989.
17. N. Burgess. "A constructive algorithm that converges for real-valued input patterns," Int. J. Neural Syst., vol. 5, no. 1, pp. 59-66, Mar. 1994.
18. R. Setiono and L. C. K. Hui, "Use of a quasi-Newton method in a feedforward neural network construction algorithm," IEEE Trans. Neural Netw., vol. 6, no. 1, pp. 273-277, Jan. 1995.
19. T.Y. Kwok and D. Y. Yeung, "Objective functions for training new hidden units in constructive neural networks," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 1131-1148, Sep. 1997.
20. M. Lehtokangas, "Fast initialization for cascade-correlation learning," IEEE Trans. Neural Netw., vol. 10, no. 2, pp. 410-414, Mar. 1999.
21. M. Lehtokangas, "Modified cascade-correlation learning for classification," IEEE Trans. Neural Netw., vol. 11, no. 3, pp. 795-798, May 2000.
22. D. Liu, T.-S. Chang, and Y. Zhang, "A constructive algorithm for feedforward neural networks with incremental training," IEEE Trans. Circuits Syst. I, Fundam. Theory Appl., vol. 49, no. 12, pp. 1876-1879, Dec. 2002.
23. L. Ma and K. Khorasani, "Constructive feedforward neural networks using Hermite polynomial activation functions," IEEE Trans. Neural Netw. vol. 16, no. 4, pp. 821-833, Jul. 2005.
24. Y. Le Cun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in Proc. Advances Neural Inform. Process. Syst., D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 598-605.
25. B. Hassibi and D. G. Stork, "Second-order derivatives for network pruning: Optimal brain surgeon," in Proc. Advances Neural Inform. Process. Syst., C. Lee, S. Hanson, and J. Cowan, Eds. San Mateo, CA: Morgan Kaufmann, 1993, vol. 5, pp. 164-171.
26. A. P. Engelbrecht, "A new pruning heuristic based on variance analysis of sensitivity information," IEEE Trans. Neural Netw., vol. 12, no. 6, pp. 1386-1399, Nov. 2001.
27. Y. Hirose, K. Yamashita, and S. Hijiya, "Back-propagation algorithm which varies the number of hidden units," Neural Netw., vol. 4, no. 1, pp. 61-66, 1991.
28. Md. M. Islam and K. Murase, "An algorithm for automatic design of two hidden layered artificial neural networks," in Proc. IJCNN, 2000, pp. 467-472.
29. Md. M. Islam and K. Murase, "A new algorithm to design compact two-hidden-layer artificial neural networks," Neural Netw., vol. 14, no. 9, pp. 1265-1278, Nov. 2001.
30. I. Rivals and L. Personnaz, "Neural-network construction and selection in nonlinear modeling," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 804-819, Jul. 2003.
31. R. A. Jacobs, "Increased rates of convergence through learning rate adaptation," Neural Netw., vol. 1, no. 4, pp. 295-307, 1988.
32. Y. Chauvin, "Generalization performance of overtrained backpropagation networks," in Proc. EUROSZP Workshop, 1990, pp. 46-55.
33. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Netw., vol. 2, no. 5, pp. 359-366, 1989.
34. K. Hornik, "Approximation capabilities of multilayer feedforward networks," Neural Netw., vol. 4, no. 2, pp. 251-257, 1991.
35. J. Park and I. W. Sandberg, "Universal approximation using radial basis function networks," Neural Comput., vol. 3, no. 2, pp. 246-257, 1991.
36. K. Hornik, "Some new results on neural-network approximation," Neural Netw., vol. 6, pp. 1069-1072, 1993.
37. J. Park and I. W. Sandberg, "Approximation and radial-basis-function networks," Neural Comput., vol. 5, no. 2, pp. 305-316, Mar. 1993.
38. J. D. Schaffer, D. Whitley, and L. J. Eshelman, "Combinations of genetic algorithms and neural networks: A survey of the state of the art," in Proc. Int. Workshop COGANN, D. Whitley and J. D. Schaffer, Eds., 1992, pp. 1-37.
39. X. Yao, "A review of evolutionary artificial neural networks," Int. J. Intell. Syst., vol. 8, no. 4, pp. 539-567, 1993.
40. X. Yao, "Evolving artificial neural networks," Proc. IEEE, vol. 87, no. 9, pp. 1423-1447, Sep. 1999.
41. T. Ash, "Dynamic node creation in backpropagation networks," Connect. Sci., vol. 1, no. 4, pp. 365-375, 1989.
42. M. Wynne-Jones, "Node splitting: A constructive algorithm for feedforward neural networks," Neural Comput. Appl., vol. 1, no. 1, pp. 17-22, Mar. 1993.
43. T. M. Nabhan and A. Y. Zomaya, "Toward generating neural network structures for function approximation," Neural Netw., vol. 7, no. 1, pp. 89-99, 1994.
44. S. Young and T. Downs, "CARVE - A constructive algorithm for real-valued examples," IEEE Trans. Neural Netw., vol. 9, no. 6, pp. 1180-1190, Nov. 1998.
45. M. Iqbal Bin Shahaid, Md. M. Islam, M. A. H. Akhand, and K. Murase, "A new algorithm to design multiple hidden layered artificial neural networks," in Proc. Joint 3rd Int. Conf. SCIS, 7th ISIS, Tokyo, Japan, Sep. 20-24, 2006, pp. 463-468.
46. Md. A. Sattar, Md. M. Islam, and K. Murase, "A new constructive algorithm for designing and training artificial neural networks," in Proc. 14th ICONIP, Kitakyushu, Japan, Nov. 13-16, 2007, pp. 317-327.
47. Md. M. Islam, Md. F. Amin, S. Ahmmed, and K. Murase, "An adaptive merging and growing algorithm for designing artificial neural networks," in Proc. Int. Joint Conf. Neural Netw., Hong Kong, Jun. 1-8, 2008, pp. 2004-2009.
48. J. E. Moody, "Prediction risk and architecture selection for neural in networks," From Statistics to Neural Networks: Theory and Pattern Recognition Applications, V. Cherkassky, J. H. Friedman, and H. Wechsler, Eds. New York: Springer-Verlag, 1993, pp. 147-165.
49. J. Torreele, "Temporal processing with recurrent networks: An evolutionary approach,"in Proc. 4th Int. Conf. Genetic Algorithms R. K. Belew and L. B. Booker, Eds., 1991, pp. 555-561.
50. M. Mandischer, "Evolving recurrent neural networks with non-binary encoding," in Proc. IEEE Int. Conf. Evol. Comput., 1995, pp. 584-589. Part 2(of 2).
51. F. Heimes, G. Zalesski, W. Land, Jr., and M. Oshima, "Traditional and evolved dynamic neural networks for aircraft simulation," in Proc. IEEE Int. Conf. Syst., Men, Cybern., 1997, pp. 1995-2000. Part 3(of 5).
52. D. Whitley, T. Starkweather, and C. Bogart, "Genetic algorithms and neural networks: Optimizing connections and connectivity," Perallel Comput., vol. 14, no. 3, pp. 347-361, Aug. 1990.
53. D. Whitley and T. Starkweather, "Optimizing small neural networks using a distributed genetic algorithm," in Proc. Int. Joint Conf. Neural Netw. Hillsdale, NJ: Lawrence Erlbaum, 1990, vol. I, pp. 206-209.
54. X. Yao and Y. Shi, "A preliminary study on designing artificial neural networks using co-evolution," in Proc. IEEE Singapore Int. Conf. Intell. Control Instrum., Singapore, Jun. 1995, pp. 149-154.
55. D. J. Chalmers, "The evolution of learning: An experiment in genetic connectionism," in Proc. Connectionist Models Summer School, D. S. Touretzky, J. L. Elman, and G. E. Hinton, Eds., 1990, pp. 81-90.
56. Y. Bengio and S. Bengio, "Learning a synaptic learning role," in "Dét. Informatique et de Recherche Operationelle," Univ. Montreal, Montreal, QC, Canada, Tech. Rep. 751, Nov. 1990.
57. J. F. Fontanari and R. Meir, "Evolving a learning algorithm for the binary perceptron," Network, vol. 2, no. 4, pp. 353-359, Nov. 1991.
58. J. R. Koza and J. P. Rice, "Genetic generation of both the weights and architecture for a neural network," in Proc. IEEE IJCNN, Seattle, WA, 1991, vol. 2, pp. 397-404.
59. S. Bornholdt and D. Graudenz, "General asymmetric neural networks and structure design by genetic algorithms," Neural Netw., vol. 5, no. 2, pp. 327-334, 1992.
60. S. Oliker, M. Furst, and O. Maimon, "A distributed genetic algorithm for neural network design and training," Complex Syst., vol. 6, pp. 459-477, 1992.
61. X. Yao and Y. Liu, "A new evolutionary system for evolving artificial neural networks," IEEE Trans. Neural Netw., vol. 8, no. 3, pp. 694-713, May 1997.
62. A. Mahmood, S. Sharmin, D. Barua, and Md. M. Islam, "Graph matching recombination for evolving neural networks," in Proc. 4th Int. Symp. Neural Netw., Nanjing, China, Jun. 3-7, 2007, pp. 562-568.
63. P. Baldi and Y. Chauvin, "Temporal evolution of generalization during learning in linear networks," Neural Comput., vol. 3, no. 4, pp. 589-603, 1991.
64. Md. M. Islam and K. Murase, "A new crossover operator and its application to artificial neural networks evolution," IEICE Trans. Inf. Syst., vol. E84-D, no. 9, pp. 1144-1154, 2001.
65. S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in Proc. Advences Neural Inform. Process. Syst., D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 524-532.
66. S. Haykin, Neural Networks: A Comprehensive Foundation. Englewood Cliffs, NJ: Prentice-Hall, 1999.
67. A. I. Rasiah, R. Togneri, and Y. Attikiouzel, "Modelling 1-D signals using Hermite basis functions," Proc. Inst. Elect Eng. - Vis. Image Signal Process., vol. 144, no. 6, pp. 345-354, Dec. 1997.
68. L. Prechelt, "PROBEN1 - A set of neural network benchmark problems and benchmarking rules," Faculty Informatics, Univ. Karlsruhe, Karlsruhe, Germany, Tech. Rep. 21/94, 1994.
69. Md. M. Islam, X. Yao, and K. Murase, "A constructive algorithm for training cooperative neural network ensembles," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 820-834, Jul. 2003.
70. S. V. Odri, D. P. Petrovacki, and G. A. Krstonosic, "Evolutional development of a multilevel neural network," Neural Netw., vol. 6, no. 4, pp. 583-595, 1993.
71. L. Prechelt, "Automatic early stopping using cross validation: Quantifying the criteria," Neural Netw., vol. 11, no. 4, pp. 761-767, Jun. 1998.
72. L. Prechelt, "Some notes on neural learning algorithm benchmarking," Neurocomputing, vol. 9, no. 3, pp. 343-347, Dec. 1995.
73. L. Prechelt, "A quantitative study of experimental evaluations of neural network learning algorithms," Neural Netw., vol. 9, no. 3, pp. 457-462, Apr. 1996.
74. T. B. Ludermir, A. Yamazaki, and C. Zanchettin, "An optimization methodology for neural network weights and architectures," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1452-1459, Nov. 2006.
75. S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, no. 4598, pp. 671-680, May 1983.
76. F. Glover, "Future paths for integer programming and links to artificial intelligence," Comput. Oper. Res., vol. 13, no. 5, pp. 533-549, May 1986.
77. K. Jim, C. L. Giles, and B. G. Horne, "An analysis of noise in recurrent neural networks: Convergence and generalization," IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1424-1438, Nov. 1996.