A new formulation for feedforward neural networks

IEEE Transactions on Neural Networks

Volumn 22, Issue 10, 2011, Pages 1588-1598

  Razavi, Saman ^a   Tolson, Bryan A ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

Author keywords

Feedforward neural networks; generalization; geometrical interpretation; internal behavior; measure of regularization; reformulated neural network; training

Indexed keywords

BLACK-BOX MODEL; DERIVATIVE-FREE OPTIMIZATION; ERROR RESPONSE; FUNCTION APPROXIMATION TECHNIQUES; GENERALIZATION; GENERALIZATION ABILITY; GEOMETRICAL INTERPRETATION; INTERNAL BEHAVIOR; LEARNING ABILITIES; MEASURE OF REGULARIZATION; MULTIPLE TEST; NETWORK WEIGHTS; TRAINING METHODS;

BACKPROPAGATION ALGORITHMS; PERSONNEL TRAINING;

FEEDFORWARD NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER PROGRAM; FEEDBACK SYSTEM;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; FEEDBACK; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); SOFTWARE DESIGN;

EID: 80053625918     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2011.2163169     Document Type: Review

References (35)

1. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Netw., vol. 2, no. 5, pp. 359-366, 1989. (Pubitemid 20609008)
2. T. P. Chen and H. Chen, "Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems," IEEE Trans. Neural Netw., vol. 6, no. 4, pp. 911-917, Jul. 1995.
3. M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, "Multilayer feedforward networks with a nonpolynomial activation function can approximate any function," Neural Netw., vol. 6, no. 6, pp. 861-867, 1993. (Pubitemid 23713273)
4. T. P. Chen and H. Chen, "Approximations of continuous functionals by neural networks with application to dynamic systems," IEEE Trans. Neural Netw., vol. 4, no. 6, pp. 910-918, Nov. 1993.
5. T. P. Chen, H. Chen, and R. W. Liu, "Approximation capability in C(.Rn ) by multilayer feedforward networks and related problems," IEEE Trans. Neural Netw., vol. 6, no. 1, pp. 25-30, Jan. 1995.
6. S. Tamura and M. Tateishi, "Capabilities of a four-layered feedforward neural network: Four layers versus three," IEEE Trans. Neural Netw., vol. 8, no. 2, pp. 251-255, Mar. 1997. (Pubitemid 127765120)
7. J. de Villiers and E. Barnard, "Backpropagation neural nets with one and two hidden layers," IEEE Trans. Neural Netw., vol. 4, no. 1, pp. 136-141, Jan. 1993.
8. S. Razavi, B. A. Tolosn, and D. H. Burn, "Numerical assessment of metamodelling strategies in computationally intensive optimization," Environ. Modell. Softw., 2011, to be published.
9. N. Y. Liang, G. B. Huang, P. Saratchandran, and N. Sundararajan, "A fast and accurate online sequential learning algorithm for feedforward networks," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1411-1423, Nov. 2006. (Pubitemid 44824256)
10. T. Kathirvalavakumar and S. J. Subavathi, "Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks," Neurocomputing, vol. 72, nos. 16-18, pp. 3915-3921, Oct. 2009.
11. H. T. Huynh and Y. Won, "Evolutionary algorithm for training compact single hidden layer feedforward neural networks," in Proc. IEEE Int. Joint Conf. Neural Netw., vols. 1-8. Hong Kong, Jun. 2008, pp. 3028-3033.
12. E. J. Teoh, K. C. Tan, and C. Xiang, "Estimating the number of hidden neurons in a feedforward network using the singular value decomposition," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1623-1629, Nov. 2006. (Pubitemid 44824273)
13. J. M. Benitez, J. L. Castro, and I. Requena, "Are artificial neural networks black boxes?" IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 1156-1164, Sep. 1997. (Pubitemid 127763346)
14. J. L. Castro, C. J. Mantas, and J. M. Benitez, "Interpretation of artificial neural networks by means of fuzzy rules," IEEE Trans. Neural Netw., vol. 13, no. 1, pp. 101-116, Jan. 2002. (Pubitemid 34236841)
15. A. B. Tickle, R. Andrews, M. Golea, and J. Diederich, "The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks," IEEE Trans. Neural Netw., vol. 9, no. 6, pp. 1057-1068, Nov. 1998. (Pubitemid 128742391)
16. D. Nguyen and B. Widrow, "Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights," in Proc. Int. Joint Conf. Neural Netw., vol. 3. San Diego, CA, Jun. 1990, pp. 21-26.
17. J. Y. F. Yam and T. W. S. Chow, "Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 430-434, Mar. 2001. (Pubitemid 32371498)
18. C. Xiang, S. Q. Ding, and T. H. Lee, "Geometrical interpretation and architecture selection of MLP," IEEE Trans. Neural Netw., vol. 16, no. 1, pp. 84-96, Jan. 2005. (Pubitemid 40241912)
19. L. Hamm, B. W. Brorsen, and M. T. Hagan, "Comparison of stochastic global optimization methods to estimate neural network weights," Neural Process. Lett., vol. 26, no. 3, pp. 145-158, 2007. (Pubitemid 350005972)
20. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, pp. 533-536, Oct. 1986. (Pubitemid 16025374)
21. M. T. Hagan, H. B. Demuth, and M. H. Beale, Neural Network Design. Boston, MA: PWS, 1996.
22. M. T. Hagan and M. B. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989-993, Nov. 1994.
23. M. H. Beale, M. T. Hagan, and, H. B. Demuth. (2010). Neural Network Toolbox 7 - User's Guide [Online]. Available: http://www2.cs.siu.edu/~rahimi/ cs437/slides/nnet.pdf
24. G. Lera and M. Pinzolas, "Neighborhood based Levenberg-Marquardt algorithm for neural network training," IEEE Trans. Neural Netw., vol. 13, no. 5, pp. 1200-1203, Sep. 2002.
25. D. Panagiotopoulos, C. Orovas, and D. Syndoukas, "A heuristically enhanced gradient approximation (HEGA) algorithm for training neural networks," Neurocomputing, vol. 73, nos. 7-9, pp. 1303-1323, Mar. 2010.
26. P. K. H. Phua and D. H. Ming, "Parallel nonlinear optimization techniques for training neural networks," IEEE Trans. Neural Netw., vol. 14, no. 6, pp. 1460-1468, Nov. 2003.
27. R. S. Sexton, R. E. Dorsey, and J. D. Johnson, "Beyond backpropagation: Using simulated annealing for global optimization for neural networks," in Proc. Annu. Meet. Decis. Sci. Inst., vols. 1-3. 1997, pp. 346-348.
28. M. Meissner, M. Schmuker, and G. Schneider, "Optimized particle swarm optimization (OPSO) and its application to artificial neural network training," BMC Bioinf., vol. 7, no. 1, p. 125, 2006.
29. R. Mendes, P. Cortez, M. Rocha, and J. Neves, "Particle swarms for feedforward neural network training," in Proc. Int. Joint Conf. Neural Netw., vols. 1-3. Honolulu, HI, May 2002, pp. 1895-1899. (Pubitemid 34647919)
30. F. H. F. Leung, H. K. Lam, S. H. Ling, and P. K. S. Tam, "Tuning of the structure and parameters of a neural network using an improved genetic algorithm," IEEE Trans. Neural Netw., vol. 14, no. 1, pp. 79-88, Jan. 2003.
31. R. S. Sexton, R. E. Dorsey, and J. D. Johnson, "Toward global optimization of neural networks: A comparison of the genetic algorithm and backpropagation," Decis. Support Syst., vol. 22, no. 2, pp. 171-185, Feb. 1998. (Pubitemid 128430975)
32. J.-T. Tsai, J.-H. Chou, and T.-K. Liu, "Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm," IEEE Trans. Neural Netw., vol. 17, no. 1, pp. 69-80, Jan. 2006. (Pubitemid 43263941)
33. B. A. Tolson and C. A. Shoemaker, "Dynamically dimensioned search algorithm for computationally efficient watershed model calibration," Water Resour. Res., vol. 43, no. 1, pp. W01413-1-W01413-16, Jan. 2007.
34. D. J. C. Mackay, "Bayesian interpolation," Neural Comput., vol. 4, no. 3, pp. 415-447, May 1992.
35. F. D. Foresee and M. T. Hagan, "Gauss-Newton approximation to Bayesian learning," in Proc. Int. Joint Conf. Neural Netw., vol. 3. Houston, TX, Jun. 1997, pp. 1930-1935.