A lempel-ziv complexity-based neural network pruning algorithm

International Journal of Neural Systems

Volumn 21, Issue 5, 2011, Pages 427-441

  Ahmed, Sultan Uddin ^a   Shahjahan, Md ^a   Murase, Kazuyuki ^b  

^a KHULNA UNIVERSITY OF ENGINEERING AND TECHNOLOGY   (Bangladesh)

^b UNIVERSITY OF FUKUI   (Japan)

Author keywords

ANN construction; Artificial Neural Networks (ANNs); complexity of ANN; Lempel Ziv complexity; pruning

Indexed keywords

ARTIFICIAL NEURAL NETWORKS (ANNS); BENCH-MARK PROBLEMS; COMPLEXITY OF ANN; GENERALIZATION ABILITY; HIDDEN UNITS; LEMPEL ZIV COMPLEXITY; NETWORK PRUNING; NETWORK TRAINING; PRUNING; PRUNING ALGORITHMS; PRUNING METHODS; TIME SEQUENCES; TRAINING PROCESS;

ALGORITHMS; RADIAL BASIS FUNCTION NETWORKS;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; DIABETES MELLITUS; HEART DISEASE; HUMAN; NEOPLASM;

ALGORITHMS; COMPUTER SIMULATION; DIABETES MELLITUS; HEART DISEASES; HUMANS; NEOPLASMS; NEURAL NETWORKS (COMPUTER);

EID: 80053344342     PISSN: 01290657     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0129065711002936     Document Type: Article

References (64)

1. S. Buchholz and N. L. Bihan, Polarized signal classification by complex and quaternionic multilayer perceptrons, International Journal of Neural Systems 18(2) (2008) 75-85. (Pubitemid 351730107)
2. W. Zou, Z. Chi and K. C. Lo, Improvement of image classification using wavelet coefficients with structured-based neural network, International Journal of Neural Systems 18(3) (2008) 195-205.
3. X. H. Yu and G. A. Chen, Efficient backpropagation learning using optimal learning rate and momentum, Neural Networks 10(3) (1997) 517-527.
4. N. Katada and H. Nishimura, Dynamic memorization characteristics in neural networks with different neuronal dynamics, International Journal of Neural Systems 17(3) (2007) 161-170. (Pubitemid 47117607)
5. G. Yang, Z. Tang, Z. Zhang and Y. Zhu, A flexible annealing chaotic neural network to maximum clique problem, International Journal of Neural Systems 17(3) (2007) 171-181.
6. M. Assaduzzaman, M. Shahjahan and K. Murase, Faster training using fusion of activation functions for feed forward neural networks, International Journal of Neural System 19(6) (2009) 437-448.
7. D. Bolle and R. Heylen, Adaptive thresholds for neural networks with synaptic noise, International Journal of Neural Systems 17(4) (2007) 241-252. (Pubitemid 47258346)
8. X. Yao and Y. Liu, A new evolutionary system for evolving artificial neural networks, IEEE Trans. Neural Networks 8(3) (1997) 694-713. (Pubitemid 127767812)
9. G. S. Ng, A. Wahab and D. Shi, Entropy learning and relevance criteria for neural network pruning, International Journal of Neural Systems 13(5) (2003) 291-305.
10. T. Schreiner, Pruning methods for neural networks, Diploma Thesis 1140, University of Stuttgart (1994).
11. R. Jacobs, M. Jordan, S. Nowlan and G. Hinton, Adaptive mixtures of local experts, Neural Computation 3(1) (1991) 79-87.
12. O. Sporns, Small world connectivity, motif composition, and complexity of fractal neuronal connections, Biosystems 85 (2006) 55-64. (Pubitemid 43850657)
13. O. Sporns and M. Lungarella, Evolving coordinated behaviors by maximizing informational structure, Proceedings of the Tenth International Conference on the Simulation and Synthesis of Living Systems (2006) 323-329.
14. L. S. Yaeger and O. Sporns, Evolution of neural structure and complexity in a computational ecology, Proceedings of the Tenth International Conference on the Simulation and Synthesis of Living Systems (2006) 330-336.
15. T. Jorgensen and B. Haynes, Evolution of artificial neural networks for robot control using speciation and complexity measure, Proceedings of the IADIS International Conference on Intelligent Systems and Agents (2008).
16. T. Jorgensen and B. Haynes, Complexifying artificial networks through topological reorganization, WCE Int. Conf. Comp. Intel. and Intel. Syst., London, U. K. (2008).
17. T. Jorgensen and B. Haynes, Evolving cooperative behavior, SAB06 Workshop on Bio Inspired Cooperation and Adaptive Behaviours in Robots, Rome (2006).
18. K. O. Stanley and R. Miikkulainen, Continual coevolution through complexification, The Genet. And Evol. Conf. (2002).
19. Y. L. Cun, J. S. Denker and S. A. Solla, Optimal brain damage, Adv. Neural Inform. Process. Syst. 2 (1990).
20. B. Hassibi and D. G. Stork, Second order derivatives for network pruning: Optimal brain surgeon, Adv. Neural Inform. Process Syst. 4 (1993).
21. T. Ash, Dynamic node creation in back propagation networks, Connect. Sci. 1(4) (1989) 365-375.
22. J. P. Nadal, Study of a growth algorithm for a feed forward network, International Journalof Neural System 1(1) (1989) 55-59.
23. N. Burgess, A constructive algorithm that converges for real-valued input patterns, International Journal of Neural System 5(1) (1994) 59-66.
24. R. Setiono and L. C. K. Hui, Use of a quasi- Newton method in a feed forward neural network construction algorithm, IEEE Trans. Neural Netw. 6(1) (1995) 273-277.
25. M. W. Jones, Node splitting: A constructive algorithm for feed forward neural networks, Neural Comput. Appl. 1(1) (1993) 17-22.
26. S. Young and T. Downs, CARVE - A constructive algorithm for real-valued examples, IEEE Trans. Neural Netw. 9(6) (1998) 1180-1190. (Pubitemid 128742400)
27. M. I. B. Shahid, M. M. Islam, M. A. H. Akhand and K. Murase, A new algorithm to design multiple hidden layered artificial neural networks, Joint 3rd Int. Conf. SCIS, 7th ISIS, Tokyo, Japan (2006) 463-468.
28. R. Reed, Pruning algorithms - A survey, IEEE Trans. Neural Netw. 4(5) (1993) 740-747.
29. P. Lauret, E. Fock and T. A. Mara, A node pruning algorithm based on a Fourier amplitude sensitivity test method, IEEE Trans. Neural Netw. 17(2) (2006) 273-293. (Pubitemid 43380055)
30. P. Engelbrecht, A new pruning heuristic based on variance analysis of sensitivity information, IEEE Trans. Neural Netw. 12(6) (2001) 1386-1399.
31. Y. Hirose, K. Yamashita and S. Hijiya, Backpropagation algorithm which varies the number of hidden units, Neural Netw. 4(1) (1991) 61-66.
32. M. M. Islam and K. Murase, An algorithm for automatic design of two hidden layered artificial neural networks, International Joint Conference on Neural Network (IJCNN 2000) (2000) 467-472.
33. M. M. Islam and K. Murase, A new algorithm to design compact two-hidden-layer artificial neural networks, Neural Netw. 14(9) (2001) 1265-1278. (Pubitemid 32910636)
34. I. Rivals and L. Personnaz, Neural-network construction and selection in nonlinear modeling, IEEE Trans. Neural Netw. 14(4) (2003) 804-819.
35. M. M. Islam, M. A. Sattar, M. F. Amin, X. Yao and K. Murase, A new adaptive merging and growing algorithm for designing artificial neural networks, IEEE Transactions on Systems, Man, and Cybernetics - PART B: Cybernetics 39(3) (2009) 705-722.
36. J. Torreele, Temporal processing with recurrent networks: An evolutionary approach, 4th Int. Conf. Genetic Algorithms, R. K. Belew and L. B. Booker, (eds.), pp. 555-561, 1991.
37. M. Mandischer, Evolving recurrent neural networks with non-binary encoding, IEEE Int. Conf. Evol. Comput. Part 2 (of 2) (1995) 584-589.
38. H. Duan, C. F. Xu and Z. H. Xing, A hybrid artificial bee colony optimization and quantum evolutionary algorithm for continuous optimization problems, International Journal of Neural Systems 20(1) (2010) 39-50.
39. D. Whitley, T. Starkweather and C. Bogart, Genetic algorithms and neural networks: Optimizing connections and connectivity, Parallel Comput. 14(3) (1990) 347-361.
40. D. Whitley and T. Starkweather, Optimizing small neural networks using a distributed genetic algorithm, Int. Joint Conf. Neural Netw. Hillsdale, NJ: Lawrence Erlbaum, Vol. 1, pp. 206-209 1990.
41. X. Yao and Y. Shi, A preliminary study on designing artificial neural networks using co-evolution, IEEE Singapore Int. Conf. Intell.Control Instrum., Singapore, 149-154 (1995).
42. Lempel A and Ziv J, On the complexity of finite sequences, IEEE Trans Inform Theory 22(1) (1976) 75-81.
43. B. Hassibi, D. G. Stork and G. Wolff, Optimal brain surgeon: Extensions and performance comparisons, IEEE Int. Conf. Neural Network, San Francisco (1993).
44. J. Xu and D. W. C. Ho, A node pruning algorithm based on optimal brain surgeon for feed forward neural networks, 3rd Int. Symposium on Neural Network, China (2006).
45. M. W. Pedersen, L. K. Hansen and J. Larsen, Pruning with generalization based weight saliencies: γOBD, γOBS, Adv. Neural Inform. Process. Syst. 9 (1996).
46. M. W. Pedersen and L. K. Hansen, Recurrent networks: Second order properties and pruning, Adv. Neural Inform. Process. Syst. 8 (1995).
47. A. Stahlberger and M. Riedmiller, Fast network pruning and feature extraction using the unit-OBS algorithm, Adv. Neural Inform. Process. Syst. 10 (1997).
48. M. C. Mozer and P. Smolensky, Skeletonization: A technique for trimming the fat from a network via relevance assessment, Adv. Neural Inform. Process. Syst. 1 (1989).
49. E. D. Karnin, A simple procedure for pruning backpropagation trained neural networks, IEEE Trans. Neural Networks 1(2) (1990).
50. D. James and P. Tucker, A comparative analysis of simplification and complexification in the evolution of neural network topologies, Proc. of the Genet. And Evol. Conf. (2004).
51. R. Setiono, A penalty function approach for pruning feedforward neural networks, Neural Comp. 9(1) (1997).
52. T. Q. Huynh and R. Setiono, Effective neural network pruning using cross validation, Int. joint Conf. Neural Networks, Montreal, Canada (2005).
53. M. Marchesi, G. Orlandi, F. Piazza, G. Pignotti and A. Uncini, Dynamic topology neural network, 3rd Italian Workshop on Parallel Arch. and Neural Networks (1990).
54. T. D. Jorgensen and B. P. Haynes, Pruning artificial neural networks using neural complexity measures, International Journal of Neural Systems 18(5) (2008) 389-403.
55. G. Tononi, O. Sporns and G. M. Edelman, A measure for brain complexity: relating functional segregation and integration in the nervous system, Proc. Nat. Acad. Sci., USA 91 (1994) 5033-5037. (Pubitemid 24177786)
56. M. A. Kon and L. Plaskota, Information complexity of neural networks, Neural Networks 13 (2000) 365-375. (Pubitemid 30252012)
57. A. S. Weigend, D. E. Rumelhart and B. A. Huberman, Back-propagation, weight-elimination and time series prediction, in Proc. of the 1990 Summer School on Connectionist Models (1990).
58. Y. Chauvin, A back-propagation algorithm with optimal use of hidden units, Adv. Neural Inform. Process. Syst. 1 (1989).
59. S. J. Nowlan and G. E. Hinton, Simplifying neural networks by soft weights-Sharing, Neural Compute. 4 (1992).
60. E. C. Paz, Pruning neural networks with distribution estimation algorithms, The Genet. and Evol. Conf. (2003).
61. D. Whitley and C. Bogart, The evolution of connectivity: Pruning neural networks using genetic algorithms, Int. Joint Conf. Neural Networks (1990) 134-136.
62. P. Hancock, Pruning neural nets by genetic algorithm, International Conference on Artificial Neural Networks (1992) 991-994.
63. M. Schmidt and T. Stidsen, Using GA to train NN using weight sharing, weight pruning and unit pruning, Technical Report, Aarhus University, Computer Science Department, Aarhus, Denmark (1995).
64. Willi-Hans Steeb, The Nonlinear Workbook: Chaos, Fractals, Cellular Automata, Neural Networks, Genetic Algorithms, Gene Expression Programming, Support Vector Machine, Wavelets, Hidden Markov Model, pp. 130, World Scientific Publishing Co. 3rd edition (2005).