A modified back-propagation method to avoid false local minima

Neural Networks

Volumn 11, Issue 6, 1998, Pages 1059-1072

  Fukuoka, Yutaka ^a   Matsuki, Hideo ^b   Minamitani, Haruyuki ^b   Akimasa Ishida, ^a  

^a TOKYO MEDICAL AND DENTAL UNIVERSITY   (Japan)

^b KEIO UNIVERSITY   (Japan)

Author keywords

Annealing; Back propagation; False local minima; Premature saturation; Sigmoid derivative; Weight readjusting

Indexed keywords

BACKPROPAGATION; CONVERGENCE OF NUMERICAL METHODS; DIFFERENTIAL EQUATIONS; ERROR ANALYSIS; LEARNING SYSTEMS; SIMULATED ANNEALING;

FALSE LOCAL MINIMA; PREMATURE SATURATION; SIGMOID DERIVATIVES;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; MATHEMATICAL COMPUTING; PRIORITY JOURNAL;

EID: 0032144731     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(98)00087-2     Document Type: Article

References (31)

1. Bilbro G. L., Snyder W. E., Gamier S. J., Gault J. W. Mean field annealing: A formalism for constructing GNC-like algorithms. IEEE Transactions on Neural Networks. 3:1992;131-138.
2. Cybenko G. Approximation by superpositions of a sigmoid function. Mathematics of Control, Signals, and Systems. 2:1989;303-314.
3. Funahashi K. On the approximate realization of continuous mapping by neural networks. Neural Networks. 2:1989;183-192.
4. Geman S., Geman D. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence. 6:1984;721-741.
5. Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators. Neural Networks. 2:1989;359-366.
6. Hush D. R., Horne B. G. Progress in supervised neural networks; What's new since Lippmann? IEEE Signal Processing Magazine. 10:1993;8-39.
7. Hush D. R., Horne B., Salas J. M. Error surfaces for multilayer. IEEE Transactions on Systems, Man, and Cybernetics. 22:1992;1152-1161.
8. Hush, D. R., & Salas, J. M. (1988). Improving the learning rate of back-propagation with the gradient reuse algorithm. In Proceedings of the IEEE Conference on Neural Networks (Vol. I, pp. 441-447). Piscataway, NJ: IEEE Press.
9. Irie, B., & Miyake, S. (1988). Capabilities of three-layered perceptions. In Proceedings of the IEEE Conference on Neural Networks (Vol. I, pp. 641-648). Piscataway, NJ: IEEE Press.
10. Jacobs R. A. Increased rate of convergence through learning rate adaptation. Neural Networks. 1:1988;295-307.
11. Kirkpatrick S., Gelatt C. D. Jr., Vecchi M. P. Optimization by simulated annealing. Science. 220:1983;671-680.
12. Kolen, J. F., & Pollack, J. B. (1991). Back propagation is sensitive to initial conditions. In R. P. Lippmann, J. E. Moody, & D. S. Tpuretzky (Eds.), Advances in Neural Information Processing Systems 3 (pp. 860-867). San Mateo, CA: Morgan Kaufmann.
13. Kramer, A. H., & Sangiovanni-Vincentelli, A. (1989). Efficient parallel learning algorithms for neural networks. In D. Touretzky (Ed.), Advances in Neural Information Processing Systems (pp. 40-48). San Mateo, CA: Morgan Kaufmann.
14. Krzyzak, A., Dai, W., & Suen, C. Y. (1990). Classification of large set of handwritten characters using modified back propagation model. Proceedings of the International Joint Conference on Neural Networks (Vol. III, pp. 225-232). Piscataway, NJ: IEEE Press.
15. Kung, S. Y., & Hwang, J. N. (1988). An algebraic projection analysis for optimal hidden units size and learning rates in back-propagation learning. In Proceedings of the IEEE Conference on Neural Networks (Vol. I, pp. 363-370). Piscataway, NJ: IEEE Press.
16. Lee B. W., Sheu B. J. Paralleled hardware annealing for optimal solutions on electronic neural networks. IEEE Transactions on Neural Networks. 4:1993;588-599.
17. Lee Y., Oh S. H., Kim M. W. An analysis of premature saturation in back propagation learning. Neural Networks. 6:1993;719-728.
18. McInerney, J. M., Haines, K. G., Biafore, S., & Hecht-Nielsen, R. (1989). Back propagation error surfaces can have local minima. In Proceedings of the International Joint Conference on Neural Networks (Vol. II, p. 627). Piscataway, NJ: IEEE Press.
19. Nguyen, D., & Widrow, B. (1990). Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. In Proceedings of the International Joint Conference on Neural Networks (Vol. III, pp. 21-26). Piscataway, NJ: IEEE Press.
20. Parker, D. B. (1987). Optimal algorithms for adaptive networks: second order back propagation, second order direct propagation, and second order Hebbian learning. In Proceedings of the IEEE Conference on Neural Networks (Vol. II, pp. 593-600). Piscataway, NJ: IEEE Press.
21. Peterson C., Anderson J. R. A mean field theory learning algorithm for neural networks. Complex Systems. 1:1987;995-1019.
22. Rezgui, A., & Tepedelenlioglu, N. (1990). The effect of the slope of the activation function on the back propagation algorithm. In Proceedings of the International Joint Conference on Neural Networks (Vol. I, pp. 707-710). Piscataway, NJ: IEEE Press.
23. Rigler A. K., Irvine J. M., Vogl T. P. Rescaling of variables in back propagation learning. Neural Networks. 4:1991;225-229.
24. Rumelhart D. E., Hinton G. E., Williams R. J. Learning representations by back-propagating errors. Nature. 323:1986;533-536.
25. van Ooyen A., Nienhuis B. Improving the convergence of the back-propagation algorithm. Neural Networks. 5:1992;465-471.
26. Vogl T. P., Mangis J. K., Rigler A. K., Zink W. T., Alkon D. L. Accelerating the convergence of the back-propagation method. Biological Cybernetics. 59:1988;257-263.
27. Watrous, R. L., (1987). Learning algorithms for connectionist networks: applied gradient methods of nonlinear optimization. In Proceedings of the IEEE Conference on Neural Networks (Vol. II, pp. 619-627). Piscataway, NJ: IEEE Press.
28. Weir M. K. A method for self-determination of adaptive learning rates in back propagation. Neural Networks. 4:1991;371-379.
29. Wessels L. F. A., Barnard E. Avoiding false local minima by proper initialization of connections. IEEE Transactions on Neural Networks. 3:1992;899-905.
30. Xu L., Klasa S., Yuille A. Recent advances on techniques of static feedforward networks with supervised learning. International Journal of Neural Systems. 3:1992;253-290.
31. Yamada, K., Kami, H., Tsukumo, J., & Temma, T. (1989). Handwritten numeral recognition by multilayer neural network with improved learning algorithm. In Proceedings of the International Joint Conference on Neural Networks (Vol. II, pp. 259-266). Piscataway, NJ: IEEE Press.