A new constructive algorithm for architectural and functional adaptation of artificial neural networks

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 39, Issue 6, 2009, Pages 1590-1605

  Islam, Md Monirul ^a,b   Sattar, Abdus ^a   Amin, Md Faijul ^b   Yao, Xin ^c   Murase, Kazuyuki ^b  

^a BANGLADESH UNIVERSITY OF ENGINEERING AND TECHNOLOGY   (Bangladesh)

^b UNIVERSITY OF FUKUI   (Japan)

^c UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

Architectural adaptation; Artificial neural networks (ANNs); Constructive approach; Functional adaptation; Generalization ability

Indexed keywords

ARCHITECTURAL ADAPTATION; ARTIFICIAL NEURAL NETWORKS (ANNS); CONSTRUCTIVE APPROACH; FUNCTIONAL ADAPTATION; GENERALIZATION ABILITY;

BACKPROPAGATION ALGORITHMS; COMPUTATIONAL COMPLEXITY; CONSTRUCTION; LEARNING ALGORITHMS; LOCOMOTIVES; NEURAL NETWORKS; NEURONS; RAILROAD CARS; SET THEORY;

BACKPROPAGATION;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK;

ALGORITHMS; NEURAL NETWORKS (COMPUTER);

EID: 70349614379     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2009.2021849     Document Type: Article

References (48)

1. T. Y. Kwok and D. Y. Yeung, "Constructive algorithms for structure learning in feedforward neural networks for regression problems," IEEE Trans. Neural Netw., vol. 8, no. 3, pp. 630-645, May 1997.
2. A. P. Engelbrecht, "A new pruning heuristic based on variance analysis of sensitivity information," IEEE Trans. Neural Netw., vol. 12, no. 6, pp. 1386-1399, Nov. 2001.
3. B. Hassibi and D. G. Stork, "Second-order derivatives for network pruning: Optimal brain surgeon," in Advances in Neural Information Processing Systems, vol. 5, C. Lee, S. Hanson, and J. Cowan, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 164-171.
4. Md. M. Islam,Md. A. Sattar,Md. F. Amin, X. Yao, and K. Murase, "A new adaptive merging and growing algorithm for designing artificial neural networks," IEEE Trans. Syst., Man, Cybern. B: Cybern., vol. 39, no. 3, pp. 705-722, Jun. 2009.
5. I. Rivals and L. Personnaz, "Neural-network construction and selection in nonlinear modeling," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 804-819, Jul. 2003.
6. S.-J. Han and S.-B. Cho, "Evolutionary neural networks for anomaly detection based on the behavior of a program," IEEE Trans. Syst., Man, Cybern. B: Cybern., vol. 36, no. 3, pp. 559-570, Jun. 2006.
7. X. Yao and Y. Liu, "A new evolutionary system for evolving artificial neural networks," IEEE Trans. Neural Netw., vol. 8, no. 3, pp. 694-713, May 1997.
8. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Netw., vol. 2, no. 5, pp. 359-366, 1989.
9. Md. A. Sattar, Md. M. Islam, and K. Murase, "A new constructive algorithm for designing and training artificial neural networks," in Proc. 14th Int. Conf. Neural Inf. Process., Kitakyushu, Japan, Nov. 13-16, 2007, pp. 317-327.
10. An empirical evaluation of deep architectures on problems with many factors of variation," in Proc. 24th Int. Conf. Mach. Learn., Corvallis, OR, Jun. 20-24, 2007, pp. 473-480.
11. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, Jul. 2006.
12. J.-D. Ren, H.-Y. Huang, and J. Bao, "The research on an algorithm for special two-hidden-layer artificial neural network," in Proc. 2nd Int. Conf. Mach. Learn. Cybern., Xi'an, China, Nov. 2-5, 2003, pp. 1127-1131.
13. Md. M. Islam and K. Murase, "A new algorithm to design compact two-hidden-layer artificial neural networks," Neural Netw., vol. 14, no. 9, pp. 1265-1278, Nov. 2001.
14. S. E. Fahlman and C. Lebiere, "The cascade-correlation learning architecture," in Proc. Advances Neural Inf. Process. Syst., D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 524-532.
15. J. de Villiers and E. Barnard, "Backpropagation neural nets with one and two hidden layers," IEEE Trans. Neural Netw., vol. 4, no. 1, pp. 136-141, Jan. 1992.
16. S. Tamura and M. Tateishi, "Capabilities of a four-layered feedforward neural network: Four layers versus three," IEEE Trans. Neural Netw. vol. 8, no. 2, pp. 251-255, Mar. 1997.
17. L. Ma and K. Khorasani, "Constructive feedforward neural networks using Hermite polynomial activation functions," IEEE Trans. Neural Netw. vol. 16, no. 4, pp. 821-833, Jul. 2005.
18. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Netw. vol. 4, no. 5, pp. 740-747, Sep. 1993.
19. P. Lauret, E. Fock, and T. A. Mara, "A node pruning algorithm based on a Fourier amplitude sensitivity test method," IEEE Trans. Neural Netw., vol. 17, no. 2, pp. 273-293, Mar. 2006.
20. T. B. Ludermir, A. Yamazaki, and C. Zanchettin, "An optimization methodology for neural network weights and architectures," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1452-1459, Nov. 2006.
21. P. Guo, M. R. Lyu, and C. L. P. Chen, "Regularization parameter estimation for feedforward neural networks," IEEE Trans. Syst., Man, Cybern. B: Cybern., vol. 33, no. 1, pp. 35-44, Feb. 2003.
22. C.-S. Leung, K.-W.Wong, P.-F. Sum, and L.-W. Chan, "A pruning method for the recursive least squares algorithm," Neural Netw., vol. 14, no. 2, pp. 147-174, Mar. 2001.
23. M. Mezard and J.-P. Nadal, "Learning in feedforward layered networks: The tiling algorithm," J. Phys. A: Math. Gen., vol. 22, no. 12, pp. 2191-2203, Jun. 1989.
24. R. E. Schapire, "The strength of weak learnability," Mach. Learn. vol. 5, no. 2, pp. 197-227, Jun. 1990.
25. Md. M. Islam, X. Yao, S. M. S. Nirjon,M.A. Islam, and K. Murase, "Bagging and boosting negatively correlated neural networks," IEEE Trans. Syst., Man, Cybern. B: Cybern., vol. 38, no. 3, pp. 771-784, Jun. 2008.
26. D. Parikh and R. Polikar, "An ensemble-based incremental learning approach to data fusion," IEEE Trans. Syst., Man, Cybern. B: Cybern. vol. 37, no. 2, pp. 437-450, Apr. 2007.
27. R. Parekh, J. Yang, and V. Honavar, "Constructive neural-network learning algorithms for pattern classification," IEEE Trans. Neural Netw., vol. 11, no. 2, pp. 436-451, Mar. 2000.
28. D. S. Phatak and I. Koren, "Connectivity and performance tradeoffs in the cascade correlation learning architecture," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 930-935, Nov. 1994.
29. M. Lehtokangas, "Modified cascade-correlation learning for classification," IEEE Trans. Neural Netw., vol. 11, no. 3, pp. 795-798, May 2000.
30. Y. Freund and R. E. Schapire, "Experiments with a new boosting algorithm," in Proc. 13th Int. Conf. Mach. Learn., 1996, pp. 148-156.
31. A. I. Rasiah, R. Togneri, and Y. Attikiouzel, "Modeling 1-D signals using Hermite basis functions," Proc. Inst. Elect. Eng. - Vis. Image Signal Process., vol. 144, no. 6, pp. 345-354, Dec. 1997.
32. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing, vol. I, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge,MA: MIT Press, 1986, pp. 318-362.
33. L. Prechelt, "PROBEN1 - A set of neural network benchmark problems and benchmarking rules," Faculty Informat., Univ. Karlsruhe, Karlsruhe, Germany, Tech. Rep. 21/94, 1994.
34. A. F. Murray and P. J. Edwards, "Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training," IEEE Trans. Neural Netw., vol. 5, no. 5, pp. 792-802, Sep. 1994.
35. S. Russel and P. Norvig, Artificial Intelligence: A Modern Approach 2nd ed. Upper Saddle River, NJ: Pearson Edu., 2004.
36. L. Prechelt, "Automatic early stopping using cross validation: Quantifying the criteria," Neural Netw., vol. 11, no. 4, pp. 761-767, Jun. 1998.
37. T. Y. Kwok and D. Y. Yeung, "Objective functions for training new hidden units in constructive neural networks," IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 1131-1148, Sep. 1997.
38. K. Hornik, "Approximation capabilities of multilayer feedforward networks," Neural Netw., vol. 4, no. 2, pp. 251-257, 1991.
39. E. Kreyszig, Introductory Functional Analysis With Applications. New York: Wiley, 1989.
40. A. I. Galushkin, Neural Networks Theory. Berlin, Germany: Springer-Verlag, 2007, pp. 53-63.
41. L. Prechelt, "A quantitative study of experimental evaluation of neural network learning algorithms," Neural Netw., vol. 9, no. 3, pp. 457-462, Apr. 1996.
42. T. Fawcett, "An introduction to ROC analysis," Pattern Recognit. Lett., vol. 27, no. 8, pp. 861-874, Jun. 2006.
43. R. Setiono and L. C. K. Hui, "Use of quasi-Newton method in a feedforward neural network construction algorithm," IEEE Trans. Neural Netw., vol. 6, no. 1, pp. 273-277, Jan. 1995.
44. S. Kirkpatrick, C. D. Gelatt, Jr., and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, no. 4598, pp. 671-680, May 1983.
45. F. Glover, "Future paths for integer programming and links to artificial intelligence," Comput. Oper. Res., vol. 13, no. 5, pp. 533-549, May 1986.
46. S. E. Fahlman, "An empirical study of learning speed in backpropagation networks," Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-88-162, 1988.
47. Y. Le Cun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in Proc. Advances Neural Inf. Process. Syst., D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, vol. 2, pp. 598-605.
48. P. L. Bartlett, "The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network," IEEE Trans. Inf. Theory, vol. 44, no. 2, pp. 525-536, Mar. 1998.