Learning in the feed-forward random neural network: A critical review

Performance Evaluation

Volumn 68, Issue 4, 2011, Pages 361-384

  Georgiopoulos, Michael ^a   Li, Cong ^a   Kocak, Taskin ^b  

^a UNIVERSITY OF CENTRAL FLORIDA   (United States)

^b BAHCESEHIR UNIVERSITY   (Turkey)

Author keywords

ART; CART; Error functions; Evolutionary neural networks; Gradient descent; Learning; Multi layer perceptron; Multi objective optimization; Random neural network; SVM

Indexed keywords

ART; CART; ERROR FUNCTIONS; EVOLUTIONARY NEURAL NETWORK; GRADIENT DESCENT; LEARNING; MULTI LAYER PERCEPTRON; MULTI OBJECTIVE; RANDOM NEURAL NETWORK; SVM;

ARTS COMPUTING; EVOLUTIONARY ALGORITHMS; LEARNING ALGORITHMS; MULTIOBJECTIVE OPTIMIZATION;

NEURAL NETWORKS;

EID: 79952589387     PISSN: 01665316     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.peva.2010.07.006     Document Type: Article

References (129)

1. E. Gelenbe Random neural networks with negative and positive signals and product form solution Neural Computation 1 1989 502 510
2. D.E. Rumelhart, G.E. Hinton, and R.J. Williams Learning internal representations by error propagation D.E. Rumelhart, J.L. McClelland, PDP Research Group Parallel Distributed Processing; Explorations in the Microstructure of Cognition 1986 MIT Press Cambridge, MA
3. E. Gelenbe Stability of the random neural network model Neural Computation 2 1990 239 247
4. E. Gelenbe Learning in the recurrent neural network Neural Computation 5 1993 154 164
5. U. Halici, E. Karaoz, A linear approximation for the Gelenbe's learning algorithm for recurrent random neural networks, in: TAINN VI Proceedings of the 6th Turkish Symposium on Artificial Intelligence and Neural Networks, Istanbul, Turkey, 1996, pp. 8694.
6. V. Atalay, Learning by optimization in random neural networks, in: Proceedings of the Thirteenth International Symposium on Computer and Information Sciences, Antalya, Turkey, 1998, pp. 143148.
7. H. Bakircioglu, and T. Kocak Survey of random neural network applications European Journal of Operational Research 126 2000 319 330
8. S. Timotheou The random neural network: a survey Computer Journal 53 3 2010 251 267
9. E. Gelenbe, R. Lent, and Z. Xu Design and performance of cognitive packet networks Performance Evaluation 46 2001 155 176 (Pubitemid 32768295)
10. T. Kocak, J. Seeber, and H. Terzioglu Design and implementation of a random neural network routing engine IEEE Transactions on Neural Networks 14 2003 1128 1143
11. V. Atalay, E. Gelenbe, and N. Yalabik The random neural network model for texture generation International Journal of Pattern Recognition and Artificial Intelligence 6 1992 131 141
12. E. Gelenbe, T. Feng, and K.R.R. Krishnan Neural network methods for volumetric magnetic resonance imaging of the human brain Proceedings of IEEE 84 1996 1488 1496 (Pubitemid 126752600)
13. H. Bakircioglu, E. Gelenbe, and T. Kocak Image enhancement and fusion with the random neural network Turkish Journal of Electrical Engineering and Computer Science 5 1997 65 77
14. C. Cramer, E. Gelenbe, and P. Gelenbe Image and video compression IEEE Potentials 17 1998 29 33
15. H. Bakircioglu, E. Gelenbe, Random neural network recognition of shaped objects in strong clutter, in: Proceedings of 3rd Conference on Applications of Artificial Neural Networks in Image Processing, San Jose, CA, USA, 1998, pp. 2228. (Pubitemid 35038461)
16. H. Abdelbaki, E. Gelenbe, T. Kocak, Matched neural filters for EMI based mine detection, in: Proceedings of Internatinal Joint Conference on Neural Networks, Washington, DC, 1999, pp. 32363240. (Pubitemid 30593517)
17. K. Hussain, G.S. Moussa, Laser intensity vehicle classification system based on random neural network, in: Proceedings of 43rd Annual Southeast Regional Conference, Kennesaw, Georgia, USA, 2005, pp. 3135.
18. E. Gelenbe, and T. Kocak Wafer surface reconstruction from top-down scanning electron microscope images Microelectronic Engineering 75 2004 216 233
19. E. Gelenbe, and J.M. Fourneau Random neural networks with multiple classes of signals Neural Computation 11 1999 953 963
20. E. Gelenbe, and K. Hussain Learning in the multiple class random neural network IEEE Transactions on Neural Networks 13 2002 1257 1267 (Pubitemid 35428517)
21. E. Gelenbe, Z.-H. Mao, and Y.-D. Li Function approximation with spiked random networks IEEE Transactions on Neural Networks 10 1 1999 3 9
22. E. Gelenbe, Z.-H. Mao, and Y.-D. Li Function approximation by random neural networks with bounded number of layers Journal of Differential Equations and Dynamical Systems 12 1&2 2004 143 170
23. E. Gelenbe, and S. Timotheou Synchronized intercations in spiked neuronal models The Computer Journal 51 6 2008 723 730
24. E. Gelenbe, and S. Timotheou Random neural networks with synchronized interactions Neural Computation 20 2008 2308 2324
25. S. Timotheou, Nonegative least squares learning for the random neural network, in: 18th International Conference on Artificial Neural Networks, Prague, Czech Republic, 2008, pp. 195204.
26. R.A. Jacobs Increased rates of convergence through learning rate adaptation Neural Networks 1 1988 151 160
27. M. Riedmiller, H. Braun, A direct adaptive method for faster back-propagation learning: the RPROP algorithm, in: Proceedings of the International Conference on Neural Networks, 1993, pp. 586591.
28. M.T. Hagan, and M. Menhaj Training feed-forward neural networks with the marquardt algorithm IEEE Transactions on Neural Networks 5 6 1994 989 993
29. A. Likas, and A. Stafylopatis Training the random neural network using quasi-newton methods European Journal of Operational Research 126 2000 331 339
30. S. Basterrech, S. Mohammed, G. Rubino, and M. Soliman LevenbergMarquardt training algorithms for random neural networks The Computer Journal 2009
31. C. Hubert, Pattern completion with the random neural network using the RPROP learning algorithm, in: Proceedings of International Conference on Man and Cybernetics, LeTouquet, France, vol. 2, 1993, pp. 613617.
32. J. Aguilar, and A. Colmenares Resolution of pattern recognition problems using a hybrid genetic/random neural network learning algorithm Pattern Analysis and Applications 1 1998 52 61 (Pubitemid 128697163)
33. J. Kennedy, R. Eberhart, Particle swarm optimization, in: Proceedings of the IEEE International Conference on Neural Networks, Perth Australia, vol. 4, 1995, pp. 19421947.
34. R. Stork, and K. Price Differential evolution a simple and efficient heuristic for global optimization over continuous spaces Journal of Global Optimization 11 1997 341 359 (Pubitemid 127502202)
35. Y. Shi, R. Eberhart, A modified particle swarm optimizer, in: Proceedings of the IEEE Conference on Evolutionary Computation, Singapore, 1998, pp. 6973.
36. Y. Shi, R.C. Eberhart, Fuzzy adaptive particle swarm optimization, in: Proceedings of the 2001 Congress on Evolutionary Computation, Seoul, Korea, 2001, pp. 101106. (Pubitemid 32806325)
37. L. Zhang, H. Yu, and S. Hu A new approach to improve particle swarm optimization Genetic and Evolutionary Computation 2003 134 139
38. Q. Zheng, Y. Fan, Z. Shi, and Y. Wang Adaptive inertia weight particle swarm optimization Lecture Notes in Computer Science vol. 4029 2006 Springer Berlin, Heidelberg 450 459 (Pubitemid 44089833)
39. G. Zee-Lee Particle swarm optimization to solve the economic dispatch considering the generator constraints IEEE Transactions on Power Systems 18 2003 1187 1195
40. P. Jinho, C. Kiyong, and D. Allstot Parasitic-aware RF circuit design and optimization IEEE Transactions on Circuits and Systems 51 2004 1953 1965
41. M.A. Abido Optimal design of power-system stabilizers using particle swarm optimization IEEE Transactions on Energy Conversion 17 2002 406 413 (Pubitemid 35151678)
42. J. Salerno, Using the particle swarm optimization technique to train a recurrent neural model, in: Proceedings of the 9th International Conference on Tools with Artificial Intelligence, IEEE Press, pp. 4549.
43. C.F. Juang A hybrid genetic algorithm and particle swarm optimization for recurrent neural network design IEEE Transactions on Systems Man and Cybernetics 32 2004 997 1006
44. Y. Da, and X.R. Ge An improved PSO-based ANN with simulated annealing technique Neurocomputing Letters 63 2005 527 533 (Pubitemid 40103877)
45. M. Karvalho, T.B. Ludermir, Particle swarm optimization of neural network architectures and weights, in: Proceedings of the 7th International Conference on Hybrid Intelligent Systems, Kaiserslautern, Germany, 2007, pp. 336339.
46. M. Meissner, M. Schmuker, and G. Schneider Optimized particle swarm optimization (OPSO) and its application to artificial neural network training BMC Bioinformatics 7 125 2006
47. C. Zhang, H. Shao, An ANN's evolved by a new evolutionary system and its application, in: Proceedings of the 39th IEEE Conference on Decision and Control, Sydney, Australia, vol. 4, 2000, pp. 35623563. (Pubitemid 32528212)
48. M. Settles, B. Rodebaugh, and T. Soule Comparison of genetic algorithm and particle swarm optimizer when evolving a recurrent neural network Proceedings of the Genetic and Evolutionary Computation Conference GECCO, Chicago, IL Lecture Notes in Computer Science vol. 2723 2003 151 152
49. Y. Yu, L. Xi, and S. Wang An improved particle swarm optimization for evolving feed-forward artificial neural networks Neural Processing Letters 26 2007 217 231 (Pubitemid 350005976)
50. S. Kiranyaz, I. Turker, A. Yildirim, and M. Gabbou Evolutionary artificial neural networks by multi-dimensional particle swarm optimization Neural Networks 22 2009 1448 1462
51. H.A. Abass Speeding up back-propagation using multi-objective evolutionary algorithms Neural Computation 15 2003 2705 2726
52. C. Cramer, E. Gelenbe, and H. Bakircioglu Low bit rate video compression with neural networks and temporal sub-sampling Proceedings of the IEEE 84 10 1996 1529 1543
53. E. Gelenbe, C. Cramer, M. Sungur, and P. Gelenbe Traffic and video quality in adaptive neural compression Multimedia Systems 4 6 1996 357 369 (Pubitemid 126598437)
54. E. Gelenbe, M. Sungur, Random network learning and image compression, in: IEEE International Conference on IEEE World Congress on Computational Intelligence, 1994, pp. 39963999.
55. C.E. Cramer, and E. Gelenbe Video quality and traffic QOS in learning-based subsampled and receiver interpolated video sequences IEEE Journal on Selected Areas in Communications 18 2 2000 150 166
56. H. Bakircioglu, E. Gelenbe, and L. Carin Random neural network recognition of shaped objects in strong clutter Proceedings of the 7th International Conference on Neural Networks Lecture Notes in Computer Science vol. 1327 1997 961 966 (Pubitemid 127140357)
57. H. Bakircioglu, E. Gelenbe, Feature-based RNN target recognition, in: SPIE Proceedings, Orlando, FL, in: Aerosense 98, vol. 3370, (Algorithms for Synthetic Aperture Radar Imagery, V), 1998, pp. 508518.
58. S. Mohamed, and G. Rubino A study of real-time packet video quality using random neural networks IEEE Transactions on Circuits and Systems for Video Technology 12 12 2002 1071 1073
59. S. Mohamed, G. Rubino, H. Afifi, F. Cervantes, Real-time video quality assessment in packet networks: a neural network model, in: Proceedings of International Conference of Parallel and Distibuted Processing Techniques and Applications, PDPTA'01, Las Vegas, NV, 2001.
60. S. Mohamed, G. Rubino, and M. Varela Performance evaluation of real time speech through a packet network Performance Evaluation 57 2004 141 161
61. G. Rubino, M. Varela, and J. Bonnin Controlling multimedia QoS in the future home network using the PSQA metric Computer Journal 49 2006 137 155 (Pubitemid 43380654)
62. G. Rubino, P. Tirilly, M. Varela, Evaluating users satisfaction in packet networks using random neural networks, in: Proceedings 16th International Conference on Artificial Neural Networks, ICANN 2006, Athens, Greece, pp. 30312.
63. H. Abdelbaki, E. Gelenbe, T. Kocak, E. El-Khamy, Random neural network filter for land mine detection, in: Proceedings of the 16th National Radio Science Conference, NRSC, Am Shams University, Cairo Egypt, 1999.
64. G. Loukas, and G. Oke A biologically inspired denial of service detector using the random neural network Proceeding of the 4th IEEE International Conference on Mobile Adhoc and Sensor Systems MASS'07, Pisa, Italy 2007 IEEE Computer Society Washington, DC, USA 1 7
65. G. Oke, G. Loukas, E. Gelenbe, Detecting denial of service attacks with bayesian classifiers and the random neural network, in: Proceedings of Fuzz-IEEE, London, UK, 2007 pp. 19641969.
66. G. Loukas, G. Oke, Likelihood ratios and recurrent random neural networks in detection of denial of service attacks, in: Proceedings of International Symposium of Computer and Telecommunication Systems, SPECTS'07, San Diego, CA, USA, 2007.
67. Gülay ke, and Georgios Loukas A denial of service detector based on maximum likelihood detection and the random neural network Computer Journal 50 6 2007 717 727 (Pubitemid 350031419)
68. E. Gelenbe, E. Seref, and Z. Xu Simulation with learning agents Proceedings of the IEEE 89 2001 148 157 (Pubitemid 33766568)
69. U. Halici Reinforcement learning with internal expectation for the random neural network European Journal of Operations Research 126 2000 288 307
70. E. Gelenbe, M. Gellman, R. Lent, P. Liu, P. Su, Autonomous smart routing for network QoS, in: Proceedings of the First International Conference on Autonomic Computing, New York, NY, 2004, pp. 232239.
71. E. Gelenbe Steps toward self-aware networks Communications of the ACM 52 7 2009 66 75
72. G. Sakellari The cognitive packet network: a survey The Computer Journal 53 2010 268 279
73. E. Gelenbe Product form queuing networks with negative and positive customers Journal of Applied Probability 28 1991 656 663
74. J.R. Artalejo G-networks: a versatile approach for work removal in queuing networks European Journal of Operations Research 126 2000 233 249
75. UCI repository of machine learning databases. URL: http://www.ics.uci. edu/mlearn/MLRepository.html.
76. S.E Fahlman Faster learning variations on back-propagation: an empirical study D.E. Touretzky, G. Hinton, T. Senjnowski, Proceedings of the Connectionist Models Summer School San Mateo 1988 Morgan Kaufmann 38 51
77. Y. LeCun Efficient BackProp Genevieve B. Orr, Klauss-Robert Muller, Neural Networks: Tricks of the Trade Lecture Notes in Computer Science 1998 Springer 9 50
78. L. Beheva, S. Kumar, and A. Patnaik On adaptive learning rate that guarantees convergence in feedforward networks IEEE Transactions on Neural Networks 17 5 2006 1116 1125 (Pubitemid 46445171)
79. C-T. Kim, and J.-J. Lee Training two-layer feedforward networks with variable projection method IEEE Transactions on Neural Networks 19 2 2008 371 375 (Pubitemid 351359303)
80. S. Grossberg Adaptive pattern classification and universal recoding, II: feedback, expectation, olfaction, and illusions Biological Cybernetic 1976 187 202
81. G.A. Carpenter, S. Grossberg, N. Markuzon, J.H. Reynolds, and D.B. Rosen Fuzzy ARTMAP: a neural network architecture for incremental supervised learning of analog multidimensional maps IEEE Transactions on Neural Networks 1992 698 713 (Pubitemid 23555782)
82. G. Anagnostopoulos, Novel approaches in adaptive resonance theory for machine learning, Ph.D. Dissertation, Department of Electrical Engineering and Computer Science, University of Central Florida, May 2001.
83. G.C. Anagnostopoulos, M. Bharadwaj, M. Georgiopoulos, S.J. Verzi, G.L. Heileman, Exemplar-based pattern recognition via semi-supervised learning, in: Proceedings of the 2003 International Joint Conference on Neural Networks, Portland, Oregon, USA, vol. 4, 2003, pp. 27822787.
84. J.R. Williamson Gaussian ARTMAP: a neural network for fast incremental learning of noisy multidimensional maps Neural Networks 9 5 1996 881 897 (Pubitemid 26264498)
85. A. Kaylani, M. Georgiopoulos, M. Mollaghasemi, G.C. Anagnostopoulos, C. Sentelle, and M. Zhong An adaptive multi-objective approach to evolving ART architectures IEEE Transactions on Neural Networks 21 4 2010 529 550
86. D.E. Goldberg Genetic Algorithms in Search, Optimization, and Machine Learning 1989 Addison-Wesley
87. I. Das, and J. Dennis A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems Structural Optimization 14 1997 6 69
88. C.J.C. Burges A tutorial on support vector machines for pattern recognition Data Mining and Knowledge Discovery 2 2 1998 121 167 (Pubitemid 128695475)
89. L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone Classification and Regression Trees 1984 Wadsworth
90. C.-C. Chan, C.-J. Lin, LIBSVM: a library for support vector machines. URL: http://www.csie.ntu.edu.tw/cjlin/libsvm.
91. J.C. Platt Fast training of support vector machines using sequential minimal optimization Advances in Kernel Methods: Support Vector Learning 1999 185 208
92. T. Lim, W. Loh, and Y. Shih A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms Machine Learning 40 2000 203 229
93. J. Demsar Statistical comparisons of classifiers over multiple datasets Journal of Machine Learning Research 7 2006 1 30 (Pubitemid 43022939)
94. S. Garcia, and F. Herrera An extension on "statistical comparisons of classifiers over multiple data-sets" for all pairwise comparions Journal of Machine Learning Research 9 2008 2677 2694
95. N.B. Karayiannis, and A.N. Venetsanopoulos Fast learning algorithms for neural networks IEEE Transactions on Circuits and Systems-II: Analog and Digital Signal Processing 39 7 1992
96. B-H. Juang, and S. Katagiri Discriminative learning for minimum error classification IEEE Transactions on Signal Processing 40 12 1992 3043 3054 (Pubitemid 23603018)
97. A. van Ooyen, and B. Nienhuis Improving the convergence of the back-propagation algorithm Neural Networks 5 1992 465 471
98. V. Nedeljkovic A novel multi-layer neural networks training algorithm that minimizes the probability of classification error IEEE Transactions on Neural Networks 4 4 1993
99. J. Cid-Sueiro, J. Ignacio Arribas, S. Urban-Munoz, and A.R. Figueiras-Vidal Cost functions to estimate a posteriori probabilities in multi-class problems IEEE Transactions on Neural Networks 10 3 1999 645 656
100. C.M. Bishop Neural Networks for Pattern Recognition 2008 Oxford University Press
101. X. Yao Evolving artificial neural networks Proceedings of the IEEE 87 9 1999 1423 1447 (Pubitemid 30503424)
102. P.A. Castillo, J. Carpio, J.J. Merelo, A. Prieto, and V. Rivas Evolving multi-layer perceptrons Neural Processing Letters 12 2000 115 127
103. J.M. Yang, and C.Y. Kao A robust evolutionary algorithm for training neural networks Neural Computing and Applications 10 2001 214 230
104. E. Cantu-Paz, and C. Kamath An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems IEEE Transactions on Systems, Man and Cybernetics Part B 35 2005 915 927 (Pubitemid 41444577)
105. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery Numerical Recipes in C 1992 Cambridge University Press
106. L. Ingber Simulated annealing: practice versus theory Journal of Mathematical and Computer Modeling 18 11 1993 29 57
107. H. Muehlenbein Schlierkamp-Vosen, predictive models for the breeder genetic algorithm, I. Continuous parameter optimizations Evolutionary Computation 1 1 1993 25 49
108. H.-M. Voigt Soft Genetic Operators in Evolutionary Computation, Evolution and Biocomputation Lecture Notes in Computer Science vol. 899 1995 Springer Berlin 123 141
109. F. Aluffi-Pentini, V. Parisi, and F. Zirilli Global optimization and stochastic differential equations Journal of Optimization Theory and Applications 47 1 1985 1 16 (Pubitemid 15569316)
110. B.D. Ripley Neural Networks and related methods for classification Journal of Royal Statistical Society Series B 56 1994 409 456
111. L.F.A. Wessels, and E. Barnard Avoiding false local minima by proper initialization of connections IEEE Transactions on Neural Networks 3 1992 899 905 (Pubitemid 23580842)
112. I. Jordanov, and A. Georgieva Neural network learning with global heuristic search IEEE Transactions on Neural Networks 18 3 2007 937 942 (Pubitemid 46773760)
113. C.C. Coello Evolutionary multi-objective optimization: a historical view of the field IEEE Computational Intelligence Magazin 1 2006 2 36
114. H. Ishibuchi, T. Murata, and I.B. Turksen Single-objective and two-objective genetic algorithms for selecting limguistic rules for pattern classification problems Fuzzy Sets and Systems 89 2 1997 13 150
115. R.M. Everson, and J.E. Fieldsend Multi-class ROC analysis from a multi-objective optimization perspective Pattern Recognition Letters 27 8 2006 91 927
116. Y. Jin, R. Wen, and B. Sendhoff Evolutionary multi-objective optimizationg of Spiking neural networks Proceedings of the International Conference of Artificial Neural Networks LNCS vol. 4668 2007 Springer Verlag Berlin, Heidelberg 37 379
117. L. Graning, Y. Jin, B. Sendhoff, Generalization improvement in multi-objective learning, in: Proceedings of the 2006 International Joint Conference on Neural Networks, 2006, pp. 99839900.
118. Y. Jin, and B. Sendhoff Pareto-based multi-objective Machine Learning: an overview and case studies IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 38 3 2008 397 415 (Pubitemid 351663123)
119. J. Park, and J.W. Sandber Universal approximation using radial basis function neural networks Neural Computation 3 2 1991 246 257
120. G.A. Carpenter, and S. Grossberg The ART of adaptive pattern recognition by a self-organizing neural network Computer 21 3 1988 77 88 (Pubitemid 18612300)
121. M. Georgiopoulos, G.L. Heileman, and J. Huang Convergence properties of learning in ART1 Neural Computation 2 4 1990 502 510
122. M. Georgiopoulos, G.L. Heileman, and J. Huang Properties of learning related to pattern diversity in ART1 Neural Networks 4 6 1991 751 758
123. M. Georgiopoulos, J. Huang, and G.L. Heileman Properties of learning in ARTMAP Neural Networks 7 3 1994 495 506
124. J. Huang, M. Georgiopoulos, and G.L. Heileman Fuzzy ART properties Neural Networks 8 2 1995 203 213
125. M. Georgiopoulos, H. Fernlund, G. Bebis, and G.L. Heileman Order of search in Fuzzy ART and Fuzzy ARTMAP: effect of the choice parameter Neural Networks 9 9 1996 1541 1559 (Pubitemid 27055273)
126. G.C. Anagnostopoulos, and M. Georgiopoulos Category regions as new geometrical concepts in fuzzy ART and fuzzy ARTMAP Neural Networks 15 10 2002 1205 1221
127. G.J. Gibson, and C.F.N. Cowan On the decision regions of multi-layer perceptrons Proceedings of the IEEE 78 10 1990
128. D. Hush, J.M. Salas, and B. Horne Error surfaces for multi-layer perceptrons IEEE Transactions for Systems, Man and Cybernetics 22 1992 1152 1161
129. M. Brady, R. Raghavan, J. Slawny, Gradient descent fails to separate, in: Proceedings IEEE International Conference on Neural Networks, 1988, vol. 1, pp. 649656.