A very fast learning method for neural networks based on sensitivity analysis

Journal of Machine Learning Research

Volumn 7, Issue , 2006, Pages 1159-1182

  Castillo, Enrique ^a   Guijarro Berdiñas, Bertha ^b   Fontenla Romero, Oscar ^b   Alonso Betanzos, Amparo ^b  

^a UNIVERSITY OF CANTABRIA   (Spain)

^b UNIVERSITY OF A CORUÑA   (Spain)

Author keywords

Initialization method; Least squares; Linear optimization; Neural networks; Sensitivity analysis; Supervised learning

Indexed keywords

KNOWLEDGE ENGINEERING; LEARNING ALGORITHMS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS; LINEAR EQUATIONS; NEURAL NETWORKS;

INITIALIZATION METHOD; LEAST-SQUARES; LINEAR OPTIMIZATION; SUPERVISED LEARNING;

SENSITIVITY ANALYSIS;

EID: 33745751396     PISSN: 15337928     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (49)

1. L. B. Almeida, T. Langlois, J. D. Amaral, and A. Plakhov. Parameter adaptation in stochastic optimization. In D. Saad, editor, On-line Learning in Neural Networks, chapter 6, pages 111-134. Cambridge University Press, 1999.
2. R. Battiti. First and second order methods for learning: Between steepest descent and Newton's method. Neural Computation, 4(2): 141-166, 1992.
3. E. M. L. Beale. A derivation of conjugate gradients. In F. A. Lootsma, editor, Numerical methods for nonlinear optimization, pages 39-43. Academic Press, London, 1972.
4. F. Biegler-König and F. Bärmann. A learning algorithm for multilayered neural networks based on linear least-squares problems. Neural Networks, 6:127-131, 1993.
5. W. L. Buntine and A. S. Weigend. Computing second derivatives in feed-forward networks: A review. IEEE Transactions on Neural Networks, 5(3):480-488, 1993.
6. E. Castillo, J. M. Gutiérrez, and A. Hadi. Sensitivity analysis in discrete bayesian networks. IEEE Transactions on Systems, Man and Cybernetics, 26(7):412-423, 1997.
7. E. Castillo, A. Cobo, J. M. Gutiérrez, and R. E. Pruneda. Working with differential, functional and difference equations using functional networks. Applied Mathematical Modelling, 23(2):89-107, 1999.
8. E. Castillo, A. Cobo, J. M. Gutiérrez, and R. E. Pruneda. Functional networks, a new neural network based methodology. Computer-Aided Civil and Infrastructure Engineering, 15(2):90-106, 2000.
9. E. Castillo, A. Conejo, P. Pedregal, R. Garcia, and N. Alguacil. Building and Solving Mathematical Programming Models in Engineering and Science. John Wiley & Sons Inc., New York., 2001.
10. E. Castillo, O. Fontenla-Romero, A. Alonso Betanzos, and B. Guijarro-Berdiñas. A global optimum approach for one-layer neural networks. Neural Computation, 14(6): 1429-1449, 2002.
11. E. Castillo, A. S. Hadi, A. Conejo, and A. Fernández-Canteli. A general method for local sensitivity analysis with application to regression models and other optimization problems. Technometrics, 46(4):430-445, 2004.
12. E. Castillo, C. Castillo A. Conejo and, R. Minguez, and D. Ortigosa. A perturbation approach to sensitivity analysis in nonlinear programming. Journal of Optimization Theory and Applications, 128(1):49-74, 2006.
13. A. Chella, A. Gentile, F. Sorbello, and A. Tarantino. Supervised learning for feed-forward neural networks: a new minimax approach for fast convergence. Proceedings of the IEEE International Conference on Neural Networks, 1:605-609, 1993.
14. V. Cherkassky and F. Mulier. Learning from Data: Concepts, Theory, and Methods. Wiley, New York, 1998.
15. R. Collobert, Y. Bengio, and S. Bengio. Scaling large learning problems with hard parallel mixtures. International Journal of Pattern Recognition and Artificial Intelligence, 17(3):349-365, 2003.
16. J. E. Dennis and R. B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Englewood Cliffs, NJ, 1983.
17. G. P. Drago and S. Ridella. Statistically controlled activation weight initialization (SCAWI). IEEE Transactions on Neural Networks, 3:899-905, 1992.
18. R. Fletcher and C. M. Reeves. Function minimization by conjugate gradients. Computer Journal, 7 (149-154), 1964.
19. O. Fontenla-Romero, D. Erdogmus, J.C. Principe, A. Alonso-Betanzos, and E. Castillo. Linear least-squares based methods for neural networks learning. Lecture Notes in Computer Science, 2714(84-91), 2003.
20. M. T. Hagan and M. Menhaj. Training feedforward networks with the marquardt algorithm. IEEE Transactions on Neural Networks, 5(6):989-993, 1994.
21. M. T. Hagan, H. B. Demuth, and M. H. Beale. Neural Network Design. PWS Publishing, Boston, MA, 1996.
22. M. Hollander and D. A. Wolfe. Nonparametric Statistical Methods. John Wiley & Sons, 1973.
23. J. C. Hsu. Multiple Comparisons. Theory and Methods. Chapman&Hall/ CRC, Boca Raton, FL, 1996.
24. D. R. Hush and J. M. Salas. Improving the learning rate of back-propagation with the gradient reuse algorithm. Proceedings of the IEEE Conference of Neural Networks, 1:441-447, 1988.
25. B. C. Ihm and D. J. Park. Acceleration of learning speed in neural networks by reducing weight oscillations. Proceedings of the International Joint Conference on Neural Networks, 3:1729-1732, 1999.
26. R. A. Jacobs. Increased rates of convergence through learning rate adaptation. Neural Networks, 1 (4):295-308, 1988.
27. Y. LeCun, I. Kanter, and S.A. Solla. Second order properties of error surfaces: Learning time and generalization. In R.P. Lippmann, J.E. Moody, and D.S. Touretzky, editors, Neural Information Processing Systems, volume 3, pages 918-924, San Mateo, CA, 1991. Morgan Kaufmann.
28. Y. LeCun, L. Bottou, G.B. Orr, and K.-R Müller. Efficient backprop. In G. B. Orr and K.-R. Müller, editors, Neural Networks: Tricks of the trade, number 1524 in LNCS. Springer-Verlag, 1998.
29. K. Levenberg. A method for the solution of certain non-linear problems in least squares. Quaterly Journal of Applied Mathematics, 2(2): 164-168, 1944.
30. E. Ley. On the peculiar distribution of the U.S. stock indeces' first digits. The American Statistician, 50(4):311-314, 1996.
31. E. N. Lorenz. Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 20:130-141, 1963.
32. D. W. Marquardt. An algorithm for least-squares estimation of non-linear parameters. Journal of the Society of Industrial and Applied Mathematics, 11(2):431-441, 1963.
33. M. F. Moller. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks, 6:525-533, 1993.
34. D. Nguyen and B. Widrow. Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. Proceedings of the International Joint Conference on Neural Networks, 3:21-26, 1990.
35. G. B. Orr and T. K. Leen. Using curvature information for fast stochastic search. In M.I. Jordan, M.C. Mozer, and T. Petsche, editors, Neural Information Processing Systems, volume 9, pages 606-612, Cambridge, 1996. MIT Press.
36. D. B. Parker. Optimal algorithms for adaptive networks: second order back propagation, second order direct propagation, and second order hebbian learning. Proceedings of the IEEE Conference on Neural Networks, 2:593-600, 1987.
37. S. Pethel, C. Bowden, and M. Scalora. Characterization of optical instabilities and chaos using MLP training algorithms. SPIE Chaos Opt, 2039:129-140, 1993.
38. M. J. D. Powell. Restart procedures for the conjugate gradient method. Mathematical Programming, 12:241-254, 1977.
39. S. Ridella, S. Rovetta, and R. Zunino. Circular backpropagation networks for classification. IEEE Transactions on Neural Networks, 8(1):84-97, January 1997.
40. A. K. Rigler, J. M. Irvine, and T. P. Vogl. Rescaling of variables in back propagation learning. Neural Networks, 4:225-229, 1991.
41. D. E. Rumelhart, G. E. Hinten, and R. J. Willian. Learning representations of back-propagation errors. Nature, 323:533-536, 1986.
42. N. N. Schraudolph. Fast curvature matrix-vector products for second order gradient descent. Neural Computation, 14(7): 1723-1738, 2002.
43. A. Sperduti and S. Antonina. Speed up learning and network optimization with extended back propagation. Neural Networks, 6:365-383, 1993.
44. J. A. K. Suykens and J. Vandewalle, editors. Nonlinear Modeling: advanced black-box techniques. Kluwer Academic Publishers Boston, 1998.
45. T. Tollenaere. Supersab: Fast adaptive back propagation with good scaling properties. Neural Networks, 3(561-573), 1990.
46. T. P. Vogl, J. K. Mangis, A. K. Rigler, W. T. Zink, and D. L. Alkon. Accelerating the convergence of back-propagation method. Biological Cybernetics, 59:257-263, 1988.
47. M. K. Weir. A method for self-determination of adaptive learning rates in back propagation. Neural Networks, 4:371-379, 1991.
48. B. M. Wilamowski, S. Iplikci, O. Kaynak, and M. O. Efe. An algorithm for fast convergence in training neural networks. Proceedings of the International Joint Conference on Neural Networks, 2:1778-1782, 2001.
49. J. Y. F. Yam, T. W. S Chow, and C. T Leung. A new method in determining the initial weights of feedforward neural networks. Neurocomputing, 16(1):23-32, 1997.