Cooperative recurrent modular neural networks for constrained optimization: A survey of models and applications

Cognitive Neurodynamics

Volumn 3, Issue 1, 2009, Pages 47-81

  Kamel, Mohamed S ^a   Xia, Youshen ^b  

^a UNIVERSITY OF WATERLOO   (Canada)

^b FUZHOU UNIVERSITY   (China)

Author keywords

Constrained optimization problems; Cooperative modular neural networks; Engineering applications; Recurrent neural networks

Indexed keywords

ARTIFICIAL NEURAL NETWORK; BINOCULAR CONVERGENCE; BIOENGINEERING; COMPARATIVE STUDY; IMAGE RECONSTRUCTION; MATHEMATICAL MODEL; MOLECULAR MODEL; NERVE CELL NETWORK; REVIEW;

EID: 67349288507     PISSN: 18714080     EISSN: 18714099     Source Type: Journal    
DOI: 10.1007/s11571-008-9036-2     Document Type: Review

References (151)

1. Agganwal JK (1993) Multisensor fusion for computer vision. Springer-Verlag, Berlin
2. LA Alexandre AC Campilho M Kamel 2001 On combining classifiers using sum and product rules Pattern Recognit Lett 22 1283 1289 (Pubitemid 32784361)
3. 2 norms with application to bayesian-estimation IEEE Trans Signal Process 42 618 627
4. A Al-Smadi A Alshamali 2002 Fitting ARMA models to linear non-Gaussian processes using higher order statistics Signal Processing 82 1789 1793
5. R Anand K Mehrotra CK Mohan S Ranka 1995 Efficient classification for multiclass problems using modular neural networks IEEE Trans Neural Netw 6 117 124
6. Andrews HC, Hunt BR (1977) Digital image restoration. Prentice-Hall, New York
7. D Anguita A Boni 2002 Improved neural network for SVM learning IEEE Trans Neural Netw 13 1243 1244
8. Anzai Y, Shimada T (1988) Modular neural networks for shape and/or location recognition. Int Neural Network Soc First Ann Meeting, Boston, USA, 158 pp
9. Arteage-Bravo FJ (1990) Multilayer backpropagation network for learning the forward and inverse kinematics equations. Proc Int Conf Neural Netw, vol II, Washington, DC, pp 319-322
10. G Auda MS Kamel 1997 CMNN: cooperative modular neural network Int Conf Neural Netw 1 226 231
11. G Auda MS Kamel 1997 CMNN: cooperative modular neural networks for pattern recognition Pattern Recognit Lett 18 1391 1398
12. G Auda MS Kamel 1998 Modular neural network classifiers: a comparative study J Intell Robotic Syst 21 117 129
13. G Auda MS Kamel 1998 CMNN: cooperative modular neural network Neurocomputing 20 189 207 (Pubitemid 28423536)
14. G Auda MS Kamel 1999 Modular neural networks: a survey Int J Neural Syst 9 129 151
15. D Avitzour 1992 A maximum likehood approach to data association IEEE Trans AES 28 2 560 566
16. DR Baker CW Wampler II 1988 On the inverse kinematics of redundant manipulators Int J Robot Res 7 3 21 (Pubitemid 18595213)
17. Bazaraa MS, Sherali HD, Shetty CM (1993) Nonlinear programming - theory and algorithms, 2nd edn. Wiley, New York
18. Bekey GA (1992) Robotics and neural networks. In: Kosko B (ed) Neural networks for signal processing. Prentice-Hall, Englewood, pp 161-185
19. DS Broomhead D Lowe 1988 Multivariable functional interpolation and adaptive networks Complex Syst 2 321 355
20. CJ Burges 1998 A tutorial on support vector machines for patter recognition Data Mining Knowl Discov 2 2 1 47 (Pubitemid 128126769)
21. T Caelli L Guan W Wen 1999 Modularity in neural computing Proc IEEE 87 1497 1518 (Pubitemid 30503427)
22. PR Chang WH Yang KK Chan 1992 A neural network approach to MVDR beamforming problem IEEE Trans Antennas Propagation 40 313 322
23. S Chen SA Billings 1992 Neural networks for nonlinear dynamic system modelling and identification Int J Control 56 319 346
24. LJ Chen KS Narendra 2004 Identification and control of a nonlinear discrete-time system based on its linearization: a unified framework IEEE Trans Neural Netw 15 663 673
25. F-T Cheng T-H Chen Y-Y Sun 1994 Resolving manipulator redundancy under inequality constraints IEEE J Robot Automat 10 1 65 71
26. F-T Cheng R-J Sheu T-H Chen 1995 The improved compact QP method for resolving manipulator redundancy IEEE Trans Syst Man Cybern 25 11 1521 1530
27. Cichocki A, Unbehauen R (1993) Neural networks for optimization and signal processing. Wiley, England
28. Copeland M (1995) Modular network control for robot manipulator. Proc IEEE Visualize Future 26-29:183-187
29. C Cortes V Vapnik 1995 Support-vector networks Mach Learn 20 273 297
30. H Ding SK Tso 1999 A fully neural-network-based planning scheme for torque minimization of redundant manipulators IEEE Trans Industr Electr 46 199 206
31. Drosopoulos A (1994) Description of the OHGR database. Defence Research Establishment, Ottawa, Ont, Tech. Note 94-14
32. KL Du AKY Lai KKM Cheng MNS Swamy 2002 Neural methods for antenna array signal processing: a review Signal Processing 82 547 561
33. Durrant-Whyte HF (1985) Integrating distributed sensor information: an application to robot system coordinate. Proc IEEE Int Conf Syst Man Cyber, pp 415- 419
34. Fiori S (2003) Neural minor component analysis approach to robust constrained beamforming. IEEE Proc Vision Image Signal Process 150:205-218
35. Fogelman-Soulie F (1993) Multi-modular neural network-hybrid architectures: a review. Proc Int Joint Conf Neural Netw 3:25-29
36. OL Frost 1972 An algorithm for linear constrained adaptive array processing Proc IEEE 60 926 935
37. M Fukushima ZQ Luo P Tseng 2003 A sequential quadratically constrained quadratic programming method for differentiable convex minimization SIAM J Optim 13 1098 1119 (Pubitemid 37395688)
38. NP Galatsanos AK Katsaggelos 1992 Methods for choosing the regularization parameter and estimate the noise variance in image restoration and their relation IEEE Trans Image Process 1 322 336
39. XB Gao LZ Liao LQ Qi 2005 A novel neural network for variational inequalities with linear and nonlinear constraints IEEE Trans Neural Netw 16 1305 1317
40. GB Giannakis JM Mendel 1990 Cumulant-based order determination of non-Gaussian ARMA models IEEE Trans Acoust 38 1411 1423
41. MP Glazos S Hui SH Zak 1998 Sliding modes in solving convex programming problems SIAM J Control Optim 36 680 697
42. L Guan JA Anderson JP Sutton 1997 A network of networks processing model for image regularization IEEE Trans Neural Netw 8 169 174
43. Guez A, Ahmad Z (1989) Accelerated convergence in the inverse kinematics via multilayer feedforward networks. Proc Int Conf Neural Netw, vol II, Washington, DC, pp 341-344
44. LK Hansen P Salamon 1990 Neural network ensembles IEEE Trans Pattern Anal Mach Intell 12 993 1002
45. S Hashem 1997 Optimal linear combination of neural networks Neural Netw 10 599 614
46. Haykin S (1994) Neural networks: a comprehensive foundation. Macmillan Publishing, New York
47. 1 norm optimization approach for fault diagnosis of nonlinear circuits with tolerance. IEEE Proc Circuit Devices 148:223-229
48. L Hodge G Auda MS Kamel 1999 Learning decision fusion in cooperative modular neural networks Int Joint Conf Neural Netw 4 2777 2781
49. MC Hong T Stathaki AK Katstaggelos 2002 Iterative regularized least-mean mixed-norm image restoration Opt Eng 41 2515 2524
50. Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational ability. Proc Natl Acad Sci USA Biophy 79:2554-2558
51. K Hornik M Stinchcombe H White 1989 Multilayer feedforward networks are universal approximators Neural Netw 2 359 366
52. MM Islam X Yao K Murase 2003 A constructive algorithm for training cooperative neural network ensembles IEEE Trans Neural Netw 14 820 834
53. RA Jacobs MI Jordan AG Barto 1991 Task decomposition through competition in modular connectionist architecture: the what and where vision tasks Conit Sci 15 219 250
54. Kadirkamanathan V, Fabri SG (1998) Recursive structure estimation for nonlinear identification with modular network. Proc IEEE, pp 343-348
55. V Kadirkamanathan M Niranjan 1993 A function estimation approach to sequential learning with neural networks Neural Comput 5 954 975
56. Kamel MS (1999) Neural networks: the state of the art. Proceeding of the eleventh international conference on microelectronics, pp 22-24, Nov
57. AK Katsaggelos J Biemond RW Schafer RM Mersereau 1991 A regularized iterative image-restoration algorithm IEEE Trans Signal Process 39 914 929
58. Kay SM (1993) Fundamentals of statistical signal processing: estimation theory. Prentice-Hall, Englewood Cliffs
59. Kecman V (1996) System identification using modular neural network with improved learning. Proceedings of International Workshop on Neural Networks for Identification, Control, Robotics, and Signal/Image Processing 21-23:40-48
60. A Kehagias V Petridis 1997 Predictive modular neural networks for time series classification Neural Netw 10 31 49
61. MP Kennedy LO Chua 1988 Neural networks for nonlinear programming IEEE Trans Circuits Syst 35 5 554 562
62. Kinderlehrer D, Stampcchia G (1980) An introduction to variational inequalities and their applications. Academic Press, New York
63. CA Klein CH Huang 1983 Review of pseudoinverse control for use with kinematically redundant manipulators IEEE Trans Syst Man Cybern 13 245 250
64. Kohonen T (1989) Self-organization and associative memory, 3rd edn. Springer-verlag, Berlin
65. P Koukoulas N Kalouptsidis 2000 Second-order Volterra system identification IEEE Trans Signal Process 48 3574 3577
66. 1 norm estimation IEEE Trans Signal Process 40 159 168
67. H Leung G Hennessey A Drosopoulos 2000 Signal detection using the radial basis function coupled map lattice IEEE Trans Neural Netw 11 5 1133 1151
68. Y Leung KZ Chen YC Jiao 2001 A new gradient-based neural network for solving linear and quadratic programming problems IEEE Trans Neural Netw 12 1074 1083
69. 1 estimator and the Huber m-estimator SIAM J Optim 8 457 475
70. W Li X Huang H Leung 2004 Performance evaluation of digital beamforming strategies for satellite communications IEEE Trans Aerosp Electron Syst 40 12 26
71. Ljung L (1999) System identification - theory for the user, 2nd edn. Prentice-Hall, Upper Saddle River
72. BL Lu M Ito 1999 Task decomposition and module combination based on class relations: a modular neural network for pattern classification IEEE Trans Neural Netw 10 1244 1250
73. CY Maa MA Shanblatt 1992 Linear and quadratic programming neural network analysis IEEE Trans Neural Netw 3 580 594
74. AA Maciejemski CA Klein 1985 Obstacle avoidance for kinematically redundant manipulators in dynamically varying environments Int J Robot Res 4 3 109 117
75. RJ Mammone RJ Rothacker 1987 General iterative method of restoring linearly degraded images J Opt Soc Am A Opt Image Sci Vis 4 208 215
76. OL Mangasarian DR Musicant 1999 Successive overrelaxation for support vector machines IEEE Trans Neural Netw 10 1032 1037
77. W [tmp] McCulloch W Pitts 1943 A logical calculus of the ideas imminent in nervous activity Bull Math Biophys 5 115 133
78. P Melin C Felix O Castillo 2005 Face recognition using modular neural networks and the fuzzy Sugeno integral for response integration Int J Intell Syst 20 275 291
79. Muller KR, Smola AJ, Ratsch G, Schokopf BS, Kohlmorgen J, Vapnik V (1997) Predicting time series with support vector machines. Proceedings of ICANN, Springer, LNCS 1327:999-1004
80. A Nehorai P Stoica 1988 Adaptive algorithms for constrained ARMA signals in the presence of noise IEEE Trans Acoust 36 1282 1291
81. Nesterov Y, Nemirovsky A (1994) Interior-point polynomial methods in convex programming. Studies in Applid Mathematics, SIMA, 13
82. IS Oh CY Suen 2002 A class-modular feedforward neural network for handwriting recognition Pattern Recognit 35 229 244
83. E Oja 1992 Principal components, minor components and linear neural networks Neural Netw 5 927 935
84. S Ozawa 1998 An associative memory model derived from cross-coupled Hopfield nets and its role in noise-space dynamics Electr Eng 117 1253 1258
85. JK Paik AK Katsaggelos 1992 Image restoration using a modified Hopfield network IEEE Trans Image Process 1 49 63
86. SW Perry L Guan 1996 A partitioned modified Hopfield neural network algorithm for real-time image restoration Real-Time Imaging 2 215 224
87. IS Reed JD Mallett LE Brennan 1974 Rapid convergence rate in adaptive arrays IEEE Trans Aerosp Electron Syst AES-10 853 863
88. A Rodríguez-Vázquez R Domínguez-Castro A Rueda JL Huertas E Sánchez-Sinencio 1990 Nonlinear switched-capacitor 'neural networks' for optimization problems IEEE Trans Circuits Syst 37 384 397
89. JL Rojo-Alvarez M Martinez-Ramon M de Prado-Cumplido A Artes-Rodriguez AR Figueiras-Vidal 2004 Support vector method for robust ARMA system identification IEEE Trans Signal Process 52 155 164
90. J Roll AA Nazin L Ljung 2005 Nonlinear system identification via direct weight optimization Automatica 41 475 490
91. DE Rumelhart GE Hinton RJ Williams 1986 Learning internal representations by backpropagating errors Nature 323 553 536
92. ∞ minimization via a variant of Karmarkar's algorithm. IEEE Trans Signal Process 37:245-253
93. RE Schapire 1990 The strength of weak learnability Mach Learn 5 197 227
94. Sciavicco L, Siciliano B (2000) Modelling and control of robot manipulators. Springer-Verlag London, Great Britain
95. H Seraji MK Long TS Lee 1993 Motion control of 7-DOF arms: the configuration control approach IEEE Trans Rob Autom 4 125 139
96. Sharkey AJC (1998) Combing artfical neural nets: ensemble and modular multi-nets systems. Springer-Verlag, New York
97. A Sierra CS Cruz 1998 Global and local neural network ensembles Pattern Recogn Lettrs 19 651 655
98. KA Smith 1999 Neural networks for combinatorial optimization: a review of more than a decade of research Informs J Comput 11 15 28
99. RC Smith P Cheesman 1986 On the representation of estimation and spatial uncertainty Int J Robot Res 5 1 56 68
100. T Soderstrom P Stoica 1981 Comparison of some instrumental variable methods-consistency and accuracy aspects Automatica 17 101 115
101. MV Solodov P Tseng 1996 Modified projection-type methods for monotone variational inequalities SIAM J Control Optim 2 1814 1830
102. Y Sun 2000 Hopfield neural network based algorithms for image restoration and reconstruction - Part I: algorithms and simulations IEEE Trans Signal Process 48 2105 2118
103. Suykens JAK, Gestel TV, Brabanter JD, Moor BD, Vandewalle J (2002) Least squares support vector machines. World Scientific, Singapore
104. Tan Y, Xia Y, Wang J (2000) Neural network realization of support vector methods for pattern classification. Proc IEEE Int Joint Conf Neural Netw, pp 411-416
105. DW Tank JJ Hopfield 1986 Simple neural optimization networks: an A/D converter, signal decision circuit, and a linear programming circuit IEEE Trans Circuits Syst 33 5 533 541
106. Q Tao TJ Fang 2000 The neural network model for solving minimax problems with constraints Control Theory Control Appl 17 1 82 84 Chinese Edition
107. Q Tao JD Cao MS Xue H Qiao 2001 A high performance neural network for solving nonlinear programming problems with hybrid constraints Phys Lett A 288 2 88 94
108. AM Tekalp H Kaufman JW Woods 1985 Fast recursive estimation of the parameters of a space-varying autoregressive image model IEEE Trans Acoust 33 469 472
109. AM Thompson JC Brown JW Kay DM Titterington 1991 A study of methods of choosing the smoothing parameter in image restoration by regularization IEEE Trans Pattern Anal Mach Intell 13 326 339
110. Z Tian KL Bell HL Van Trees 2001 A recursive least squares implementation for LCMP beamforming under quadratic constraint IEEE Trans Signal Process 49 1138 1145
111. Van Trees HL (1998) Optimum array processing: detection and estimation theory. Class Notes: INFT 836
112. PK Varshney 1997 Multisensor data fusion J Electron Commun Eng 9 245 253
113. J Wang Q Hu D Jiang 1999 A Lagrangian neural network for kinematic control of redundant robot manipulators IEEE Trans Neural Netw 10 1123 1132
114. ZS Wang JY Cheung YS Xia JD Chen 2000 Minimum fuel neural networks and their applications to overcomplete signal representations IEEE Trans Circuits Syst I 47 8 1146 1159
115. YS Xia 1996 A new neural network for solving linear programming problems and its applications IEEE Trans Neural Netw 7 525 529
116. YS Xia 1996 A new neural network for solving linear and quadratic programming problems IEEE Trans Neural Netw 7 1544 1547
117. YS Xia 1997 Neural network for solving extended linear programming IEEE Trans Neural Netw 8 803 806
118. Xia YS (2003) Global convergence analysis of Lagrangian networks. IEEE Trans Circuits Syst I vol 50
119. YS Xia 2004 An extended projection neural network for constrained optimization Neural Comput 16 863 883
120. YS Xia G Feng 2004 A modified neural network for quadratic programming with real-time applications Neural Inf Process 3 69 76
121. Xia YS, Feng G (2006) A neural network for robust LCMP beamforming. Signal Processing 86(10):2901-2912
122. YS Xia MS Kamel 2007 Cooperative recurrent neural networks for the constrained L1 norm estimator IEEE Trans Signal Process 55 3192 3205
123. YS Xia MS Kamel 2007 Novel cooperative neural fusion algorithms for image restoration IEEE Trans Image Process 16 367 381
124. YS Xia MS Kamel 2007 A measurement fusion method for nonlinear system identification and its cooperative learning algorithm Neural Comput 19 1589 1632
125. YS Xia MS Kamel 2007 Cooperative recurrent neural networks for the constrained L1 norm estimator IEEE Trans Signal Process 55 3192 3205
126. YS Xia MS Kamel 2008 A generalized least absolute deviation method for parameter estimation of autoregressive signals IEEE Trans Neural Netw 19 107 118
127. YS Xia JS Wang 1995 Neural network for solving linear programming problems with bounded variables IEEE Trans Neural Netw 6 515 519
128. YS Xia J Wang 1998 A general methodology for designing globally convergent optimization neural networks IEEE Trans Neural Netw 9 1331 1343
129. YS Xia J Wang 1998 Neural network for solving the least absolute deviation fitting problem Neurcomputing 19 13 21
130. YS Xia J Wang 2001 A dual neural network for kinematic control of redundant robot manipulators IEEE Trans Syst Man Cybern B 31 1 147 154
131. YS Xia J Wang 2004 A general projection neural network for solving monotone variational inequality and related optimization problems IEEE Trans Neural Netw 15 318 328
132. Xia Y, Wang J (2004b) A one-layer recurrent neural network for support vector machine learning. IEEE Trans Syst Man Cybern B 1261-1269
133. YS Xia J Wang 2005 A recurrent neural network for solving nonlinear convex programs subject to linear constraints IEEE Trans Neural Netw 16 379 386
134. YS Xia H Leung H Chan 2006 A prediction fusion method for reconstructing spatial temporal dynamics using support vector machines IEEE Trans Circuits Syst Ii-Express Briefs 53 62 66
135. YS Xia L Henry J Wang 2002 A projection neural network and its application to constrained optimization problems IEEE Trans Circuits Syst I 49 4 447 458
136. YS Xia L Henry E Bossé 2002 Neural data fusion algorithms based on a linearly constrained least square method IEEE Trans Neural Netw 13 2 320 329
137. Y Xia G Feng J Wang 2004 A Recurrent neural network with exponential convergence for solving convex quadratic program and related linear piecewise equations Neural Netw 17 1003 1015
138. YS Xia J Wang LM Fork 2004 Grasping force optimization for multifingered robotic hands using a recurrent neural network IEEE Trans Rob Autom 20 549 554
139. YS Xia G Feng J Wang 2005 A primal-dual neural network for on-line resolving constrained kinematic redundancy IEEE Trans Syst Man Cybern B 35 54 64
140. Xia YS, Feng G, Kamel MS (2007) Development and analysis of neural dynamical approaches to solving nonlinear programming problems with application to optimal control. IEEE Trans Automat Contr, December
141. S Yamaguchi H Itakura 1999 A modular neural network for control of mobile robots Int Conf Neural Inform Processing 2 661 666
142. S Yang A Browne 2001 Neural network ensembles: combing multiple models for enhanced performance using a multistage approach Expert Syst 21 279 288
143. PC Young 1970 An instrumental variable methods for real time identification of a noise process Automatica 6 2711 287
144. ME Zervakis AK Katsaggelos TM Kwon 1995 A class of robust entropic functionals for image restoration IEEE Trans Image Process 4 752 773
145. GP Zhang 2000 Neural networks for classification: a survey IEEE Trans Syst Man Cybern B 30 451 459
146. S Zhang AG Constantinides 1992 Lagrange programming neural networks IEEE Trans Circuits Syst 39 7 441 452
147. S Zhang X Zhu L-H Zou 1992 Second order neural networks for constrained optimization IEEE Trans Neural Netw 3 1021 1024
148. Y Zhang J Wang YS Xia 2003 A dual neural network for redundancy resolution of kinematically redundant manipulators subject to joint limits and joint velocity limits IEEE Trans Neural Netw 14 658 667
149. WX Zheng 1999 A least-squares based method for autoregressive signals in the presence of noise IEEE Trans Circuits Syst Ii-Analog Digital Signal Process 46 81 85
150. WX Zheng 2000 Estimation of the parameters of autoregressive signals from colored noise-corrupted measurements IEEE Signal Process Lett 7 201 204
151. AHEl Zooghby CG Christodoulou M Georgiopoulos 1998 Neural network-based adaptive beamforming for one- and two-dimensional antenna arrays IEEE Trans Antennas Propagation 46 1891 1893