Efficient training algorithms for a class of shunting inhibitory convolutional neural networks

IEEE Transactions on Neural Networks

Volumn 16, Issue 3, 2005, Pages 541-556

  Tivive, Fok Hing Chi ^a   Bouzerdoum, Abdesselam ^a  

^a UNIVERSITY OF WOLLONGONG   (Australia)

Author keywords

Convolutional neural network (CoNN); First and second order training methods; Shunting inhibitory neuron

Indexed keywords

ADAPTIVE ALGORITHMS; COMPUTATIONAL METHODS; CONVOLUTION; FACE RECOGNITION; FEATURE EXTRACTION; FUNCTIONS; LEAST SQUARES APPROXIMATIONS; OPTIMIZATION; VECTORS;

CONVOLUTIONAL NEURAL NETWORKS (CONN); FIRST ORDER TRAINING METHODS; LEVENBERG-MARQUARDT (LM) ALGORITHM; SECOND ORDER TRAINING METHODS; SHUNTING INHIBITORY NEURON;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; EVALUATION; MATHEMATICAL COMPUTING; METHODOLOGY; NERVE CELL INHIBITION; REGRESSION ANALYSIS; SIGNAL PROCESSING; STATISTICAL MODEL; STATISTICS;

ALGORITHMS; COMPUTER SIMULATION; MODELS, STATISTICAL; NEURAL INHIBITION; NEURAL NETWORKS (COMPUTER); NUMERICAL ANALYSIS, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED; REGRESSION ANALYSIS; SIGNAL PROCESSING, COMPUTER-ASSISTED; STOCHASTIC PROCESSES;

EID: 19344373705     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2005.845144     Document Type: Article

References (45)

1. K. Fukushima, "Cognitron: A self organizing multilayered neural network," Biol. Cybern., pp. 121-136, 1975.
2. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jacket, "Handwritten digit recognition with a backpropa-gation neural network," in Advances in Neural Information Processing Systems, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufman, 1990, vol. 2, pp. 396-404.
3. D. R. Lovell, "The neocognitron as a system for handwritten character recognition limitations and improvements," Ph.D. dissertation, Dept. Elect. Comput. Eng., Univ. Queensland, Australia, 1994.
4. S. Lawrence, C. L. Giles, A. C. Tsoi, and A. D. Back, "Face recognition: A convolutional neural network approach," IEEE Trans. Neural Netw., vol. 8, no. 1, pp. 98-113, Jan. 1997.
5. B. Fasel, "Robust face analysis using convolutional neural networks," in Proc. 16th Int. Conf. Pattern Recognition, vol. 2, 2002, pp. 11-15.
6. S. Behnke, "Face localization in the neural abstraction pyramid," in Proc. 7th Int. Conf. Knowledge-Based Intelligent Information and Engineering Systems (KES'03), vol. 2, pp. 139-145.
7. K. Fukushima, S. Miyake, and T. Ito, "Neocognitron: A neural network model for a mechanism of visual pattern recognition," IEEE Trans. Syst., Man, Cybern., vol. SMC-13, no. 5, pp. 826-834, 1983.
8. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
9. F. Tivive and A. Bouzerdoum, "A new class of convolutional neural network (siconnets) and their application to face detection," in Proc. Int. Joint Conf. Neural Networks, vol. 3, 2003, pp. 2157-2162.
10. S. Grossberg, Ed., Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press, 1988.
11. R. B. Pinter, "Adaptation of spatial modulation transfer function via nonlinear lateral inhibition," Biol. Cybern., 51, pp. 285-291, 1985.
12. R. B. Pinter, "Adaptation of receptive field organization by multiplicative lateral inhibition," J. Theor. Biol., vol. 110, pp. 435-444, 1984.
13. A. Bouzerdoum, "The elementary movement detection mechanism in insect vision," Phil. Tram. R. Soc. Lond, vol. B-339, pp. 375-384, 1993.
14. A. Bouzerdoum and R. B. Pinter, "Nonlinear lateral inhibition applied to motion detection in the fly visual system," in Nonlinear Vision, R. B. Pinter and B. Nabet, Eds. Boca Raton, FL: CRC, 1992, pp. 423-450.
15. A. Bouzerdoum and R. B. Pinter, "Shunting inhibitory cellular neural networks: Derivation and stability analysis," IEEE Trans. Circuits Syst. I, Fundam. Theory Appl., vol. 40, no. 3, pp. 215-221, Mar. 1993.
16. A. Bouzerdoum, "A new class of high-order neural networks with nonlinear decision boundaries," in Proc. 6th Int. Conf. Neural Information Processing, vol. 3, Perth, Australia, 1999, pp. 1004-1009.
17. A. Bouzerdoum, "Classification and function approximation using feed-forward shunting inhibitory artificial neural networks," in Proc. Int. Joint Conf. Neural Networks, 2000, pp. 613-618.
18. G. Arulampalam and A. Bouzerdoum, "A generalized feedforward neural network architecture for classification and regression," Neural Netw., vol. 16, pp. 561-568, 2003.
19. G. Arulampalam and A. Bouzerdoum, "Application of shunting inhibitory artificial neural networks to medical diagnosis," in Proc. 7th Australian and New Zealand Intelligent Information Systems Conf. (ANZIIS'01), Perth, Western Australia, 2001, pp. 89-94.
20. G. Arulampalam and A. Bouzerdoum, "Expanding the structure of shunting inhibitory artificial neural network classifiers," in Proc. Int. Joint Conf. Neural Networks, vol. 3, 2002, pp. 2855-2860.
21. C. Garcia and M. Delakis, "A neural architecture for fast and robust face detection," in Proc. 16th Int. Conf. Pattern Recognition (ICPR'02), vol. 2, Quebec, Canada, 2002, pp. 20044-20048.
22. C. Neubauer, "Evaluation of convolutional neural network for visual recognition," IEEE Trans. Neural Netw., vol. 9, no. 4, pp. 685-696, Jul. 1998.
23. D. E. Rumelhart, G. Hinton, and R. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing-Explorations in the Microstructure of Cognition, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, vol. 1, pp. 318-362. Foundations.
24. M. Riedmiller, "Advanced supervised learning in multilayer perceptions - From backpropagation to adaptive learning algorithms," Comput. Standards Interfaces, no. 5, pp. 265-278, 1994.
25. S. Fahlman, "An empirical study of learning speed in backpropagation networks," Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS 88-162, 1988.
26. T. Tollenaere, "Supersab: Fast adaptive bp with good scaling properties," Neural Netw., vol. 3, pp. 561-573, 1990.
27. R. Fletcher and C. M. Reeves, "Function minimization by conjugate gradients," Comput. J., vol. 7, pp. 149-154, 1964.
28. E. Polak and G. Ribiére, "Note sur la convergence de methodes de directions conjugees," Revue Francaise d'Informatique et de Recherche Operationnelle, vol. 3, pp. 35-43, 1969.
29. M. Hestenes and E. Stiefel, "Methods of conjugate gradients for solving linear systems," J. Res. Natl. Bur. Stand., vol. 49, pp. 409-436, 1952.
30. Y. H. Dai and Y. Yuan, "A nonlinear conjugate gradient method with a strong global convergence property," in SIAM J. Optim., vol. 10, 1999, pp. 177-182.
31. Y. F. Hu and C. Storey, "Global convergence result for conjugate gradients methods," J. Optim. Theory Appl., vol. 71, pp. 399-405, 1991.
32. J. C. Gilbert and J. Nocedal, "Global convergence properties of conjugate gradient methods for optimization," in SIAM J. Optim., vol. 2, 1992, pp. 21-42.
33. Y. H. Dai and Y. Yuan, "An efficient hybrid conjugate gradient method for unconstrained optimization," Ann. Oper. Res., no. 103, pp. 33-47, 2001.
34. M. J. D. Powell, "Nonconvex minimization calculations and the conjugate gradient method," in Lecture Notes in Mathematics. Berlin, Germany: Springer-Verlag, 1984, pp. 122-141.
35. A. S. Householder, The Theory of Matrices in Numerical Analysis. New York: Dover, 1975.
36. E. K. P. Chong and S. H. Zak, An Introduction to Optimization. New York: Wiley, 1996.
37. D. H. Li and M. Fukushima, "A modified BFGS method and its global convergence in nonconvex minimization," Dept Appl. Math. Phys., Kyoto Univ., Tech. Rep. 98003, 1998.
38. A. Liao, "Modifying BFGS method," Advance Comput. Res. Inst., Cornell Theory Center, Cornell Univ., Tech. Rep., Jan. 31, 1994.
39. M. T. Hagan and M. Menhaj, "Training feedforward networks with the marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989-993, Nov. 1994.
40. N. Ampazis and S. J. Perantonis, "Two highly efficient second-order algorithms for training feedforward networks," IEEE Trans. Neural Netw., vol. 13, no. 5, pp. 1064-1074, Sep. 2002.
41. F. Bärmann and F. Biegler-König, "A learning algorithm for multilayered neural networks based on linear least squares problems," Neural Netw., vol. 6, pp. 127-131, 1993.
42. B. Verma, "Fast training of multilayer perceptions," IEEE Trans. Neural Netw., vol. 8, no. 6, pp. 1314-1321, Nov. 1997.
43. M. D. Martino, S. Fanelli, and M. Protasi, "Exploring and comparing the best "direct methods" for the efficient training of MLP-networks," IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1497-1502, Nov. 1996.
44. S. L. Phung, A. Bouzerdoum, and D. Chai, "Skin segmentation using color pixel classification: Analysis and comparison," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 1, pp. 148-154, Jan. 2005.
45. C. Charalambous, "Conjugate gradient algorithm for efficient training of artificial neural networks," in Proc. Inst. Elect. Eng. G, vol. 139, 1992, pp. 301-310.