Optimal learning in artificial neural networks: A review of theoretical results

Neurocomputing

Volumn 13, Issue 2-4, 1996, Pages 313-346

  Bianchini, M ^a   Gori, M ^a  

^a UNIVERSITY OF FLORENCE   (Italy)

Author keywords

connectionist models; learning algorithms; optimal learning

Indexed keywords

LEARNING SYSTEMS; MATHEMATICAL MODELS; NEURAL NETWORKS; OPTIMIZATION; PROBLEM SOLVING;

CONNECTIONIST MODELS; OPTIMAL LEARNING;

LEARNING ALGORITHMS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; LEARNING; PRIORITY JOURNAL; REVIEW;

EID: 0030271834     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/0925-2312(95)00032-1     Document Type: Article

References (81)

1. [1] J. Fodor and Z. Pylyshyn, Connectionism and cognitive architecture: A critical analysis, Connections and Symbols (1989) 3-72. A Cognition Special issue.
2. [2] R. Michalsky, J. Carbonell and T. Mitchell, Machine Learning, an Artificial Intelligence Approach, vol. 1/2 (Morgan Kaufmann, San Mateo, 1983).
3. [3] M. Minsky and S. Papert, Perceptrons - Expanded Edition (MIT Press, Cambridge, 1988).
4. [4] D. Rumelhart, G. Hinton and R. Williams, Learning internal representations by error propagation, in Parallel Distributed Processing (D. Rumelhart and J. McClelland, eds.), vol. 1, ch. 8, pp. 318-362 (MIT Press, Cambridge, 1986). Reprinted in [81].
5. [5] Torn and Zilinkas, Global optimization, Lecture Notes in Computer Sciences (1987).
6. [6] Zhigljavsky, Theory of Global Random Search (Kluwer Academic Publ., 1991).
7. [7] M. Gori and A. Tesi, On the problem of local minima in backpropagation, IEEE Trans. Pattern Analysis and Machine Intelligence, PAMI-14 (Jan. 1992) 76-86.
8. [8] T. Poston, C. Lee, Y. Choie and Y. Kwon, Local minima and backpropagation, in Int. Joint Conf. on Neural Networks, vol. 2, Seattle, WA (IEEE Press, July 1991) 173-176.
9. [9] X. Yu, Can backpropagation error surface not have local minima?, IEEE Trans. Neural Networks 3 (Nov. 1992) 1019-1020.
10. [10] M. Bianchini, P. Frasconi and M. Gori, Learning without local minima in radial basis function networks, to appear in IEEE Trans. Neural Networks.
11. [11] T. Cover, Geometrical and statistical properties of linear threshold devices, PhD thesis, Stanford Electronics Labs., Stanford, CA, May 1964. Also Tech. Rep. 6107-1.
12. [12] R. Brown, Adaptive multiple-output threshold systems and their storage capacities, PhD thesis, Stanford Electronics Labs., Stanford, CA, June 1964. Also Tech. Rep. 6771-1.
13. [13] M. Gori and M. Maggini, Optimal convergence of pattern mode backpropagation, Tech. Rep. DSI-8-94, Dipartimento di Sistemi e Informatica, Università degli Studi di Firenze, Firenze, Italy, April 1994, to appear in IEEE Trans. Neural Networks.
14. [14] S. Hanson and D. Burr, Minkowski-r back-propagation: Learning in connectionist models with non-euclidean error signals, in Advances in Neural Information Processing Systems (D. Anderson, ed.), (Denver, CO, 1987) 348-357.
15. [15] P. Burrascano, A norm selection criterion for the generalized delta rule, IEEE Trans. Neural Networks 2 (Jan. 1991) 125-130.
16. [16] E. Sontag and H. Sussman, Backpropagation separates when perceptrons do, in Int. Joint Conf. on Neural Networks, vol. 1, Washington, DC (IEEE Press, June 1989) 639-642.
17. [17] G. Hinton, Connectionist learning procedures, Artificial Intelligence 40 (1989) 185-234.
18. [18] E. Baum and F. Wilczek, Supervised learning of probability distributions by neural networks, in Advances in Neural Information Processing Systems, D. Anderson, ed. (1988) 52-61.
19. [19] S, Solla, E. Levin and M. Fleisher, Accelerated learning in layered neural networks, Complex Syst. 2 (1988) 625-639.
20. [20] T. Samad, Backpropagation improvements based on heuristic arguments, in Int. Joint Conf. on Neural Networks, vol. 1, San Diego, CA (1990) 565-568.
21. [21] K. Matsuoka and J. Yi, Backpropagation based on the logarithmic error function and elimination of local minima, in Int. Joint Conf. on Neural Networks, Singapore (1991) 1117-1122.
22. [22] J. Moody and C. Darken, Fast learning in networks of locally-tuned processing units, Neural Computat. 1 (1989) 281-294.
23. [23] G. Cottrel, P. Munro and D. Zipser, Learning internal representation of gray scale images: an example of extensional programming, in 9th Annual Cognitive Science Society Conf., Seattle, WA (1987).
24. [24] J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Nat. Acad. Sci. USA 79 (1982) 2554-2558. Also in Neurocomputing (MIT Press, 1988).
25. [25] D. Ackley, G. Hinton and T. Sejnowski, A learning algorithm for Boltzmann machines, Cognitive Sci. 9 (1985) 147-169.
26. [26] N. Nilsson, Learning Machines (McGraw-Hill, New York, 1965). Reissued as Mathematical Foundations of Learning Machines (Morgan Kaufmann, San Mateo, CA, 1990).
27. [27] F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanism (Washington D.C., Spartan Books, 1962).
28. [28] B. Widrow and M. Hoff, Adaptive switching circuits, in 1960 IRE WESCON Convention Record, 4 (New York, IRE, 1960) 96-104.
29. [29] B. Widrow, 30 years of adaptive neural networks: Perceptron, Madaline, and Backpropagation, IEEE Trans. Neural Networks 78 (Sep. 1990) 1415-1442.
30. [30] P. Baldi and K. Hornik, Neural networks and principal component analysis: Learning from examples without local minima, Neural Networks 2 (1989) 53-58.
31. [31] M. Brady, R. Raghavan and J. Slawny, Back-propagation fails to separate where perceptrons succeed, IEEE Trans. Circuits and Systems 36 (1989) 665-674.
32. [32] J. Shynk, Performance surfaces of a single-layer perceptron, IEEE Trans. Neural Networks 1 (Sep. 1990) 268-274.
33. [33] D. Hush and J. Salas, Improving the learning rate of back-propagation with the gradient reuse algorithm, in IEEE Int. Conf. on Neural Networks, vol. 1, San Diego (1988) 441-447 (IEEE, New York, 1988).
34. [34] K. Gouhara and Y. Uchikawa, Memory surface and learning surfaces in multilayer neural networks, Tech. Rep., Naguya University, Naguya, Japan.
35. [35] K. Gouhara, N. Kanai and Y. Uchikawa, Experimental learning surface and learning process in multilayer neural networks, Tech. Rep., Naguya University, Naguya, Japan.
36. [36] F. Jordan and G. Clement, Using the symmetries of multi-layered networks to reduce the weight space, in Int. Joint Conf. on Neural Networks. vol. 2, Seattle (IEEE, July 8-12, 1991) 391-396.
37. [37] G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control, Signals, and Systems 3 (1989) 303-314.
38. [38] K. Funahashi, On the approximate realization of continuous mappings by neural networks, Neural Networks 2 (1989) 183-192.
39. [39] R. Hecht-Nielsen, Theory of the backpropagation neural network, in Int. Joint Conf. on Neural Networks, vol. 1, Washington (1989) (IEEE, New York, 1989) 593-605.
40. [40] K. Hornik, M. Stinchcombe and H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2 (1989) 359-366.
41. [41] S. Huang and Y. Huang, Bounds on the number of hidden neurons in multi-layer perceptrons, IEEE Trans Neural Networks 2 (Jan. 1991) 47-55.
42. [42] Y. Bengio, P. Cosi and R. De Mori, Phonetically-based multi-layered networks for vowel classification, Speech Communication 9 (Feb. 1990) 15-29.
43. [43] Y. le Cun, Generalization and network design strategies, in Connectionism in Perspective (North-Holland, 1989) 143-155.
44. [44] Y. le Cun, A theoretical framework for backpropagation, in 1988 Connectionist Models Summer School, D. Touretzky, G. Hinton and T. Sejnowski, eds., (San Mateo, CA) (Morgan Kauffman, 1988) 21-28.
45. [45] A. Bryson and Y.C. Ho, Applied Optimal Control (Blaisdell, New York, 1969).
46. [46] P. Werbos, Beyond regression: new tools for prediction and analysis in the behavioral sciences, PhD thesis, Harvard University, Cambridge, MA, Nov. 1974.
47. [47] D. Parker, Learning logic, Tech. Rep. TR-47, Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, Cambridge, MA, April 1985.
48. [48] Y. le Cun, Learning process in an asymmetric threshold network, in Disordered Systems and Biological Organization, F.F. Soulie, E. Bienenstock and G. Weisbuch, eds. (Les Houches, France, Springer-Verlag, 1986) 233-240.
49. [49] H. Bourlard and C. Wellekens, Speech pattern discrimination and multi-layered perceptrons, Computer Speech and Language 3 (1989) 1-19.
50. [50] J. Elman and D. Zipser, Learning the hidden structure of speech, J. Acoustical Soc. America 83 (4) (1988) 1615-1626.
51. [51] Y. Bengio, R. De Mori and M. Gori, Learning the dynamic nature of speech with back-propagation for sequences, Pattern Recognition Letters 13 (May 1992) 375-386. Special issue on Artificial Neural Networks.
52. [52] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano and K. Lang, Phoneme recognition using time-delay neural networks, IEEE Trans. Acoustics, Speech, and Signal Processing 37 (3) (1989) 328-339.
53. [53] D. Plaut and G. Hinton, Learning set of filters using back-propagation, Computer Speech and Language 2 (1987) 35-61.
54. [54] R. Jacobs, M. Jordan and A. Barto, Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks, Tech. Rep., COINS, 1990.
55. [55] P. Frasconi, M. Gori and A. Tesi, Backpropagation for linearly separable patterns: a detailed analysis, in IEEE Int. Conf. on Neural Networks, vol. 3, San Francisco, CA (IEEE Press, March-April 1993) 1818-1822.
56. [56] S. Gallant, Perceptron-based learning algorithms, IEEE Trans. Neural Networks 1 (June 1990) 179-192.
57. [57] T. Poggio and F. Girosi, Networks for approximation and learning, Proc. IEEE 78 (9) (1990) 1481-1497.
58. [58] T. Kohonen, The self-organizing map, Proc. IEEE, 78 (9) (1990) 1464-1480. Special Issue on Neural Networks.
59. [59] X. Yu and G. Chen, Further results on the local minima free condition on backpropagation learning, to appear in IEEE Trans. Neural Networks.
60. [60] M. Gori, L. Lastrucci and G. Soda, Neural autoassociators for phoneme-based speaker verification, in Int. Workshop On Automatic Speaker Recognition, Identification, and Verification, Martigny, Switzerland (April 1994) 189-192.
61. [61] T. Kohonen, Self-Organization and Associative Memory (Berlin, Springer-Verlag, 3 ed., 1989).
62. [62] M. Bianchini, P. Frasconi and M. Gori, Learning in multilayered networks used as autoassociators, IEEE Trans. Neural Networks 6 (March 1995) 512-515.
63. [63] H. Bourland and Y. Kamp, Auto-association by multilayer perceptrons and singular value decomposition, Biol. Cybernet. 59 (1988) 291-294.
64. [64] J. McClelland and D. Rumelhart, Explorations in Parallel Distributed Processing, vol. 3 (Cambridge, MIT Press, 1988).
65. [65] R.J. Williams and J. Peng, An efficient gradient-based algorithm for on-line training of recurrent network trajectories, Neural Computat. 2 (4) (1990) 490-501.
66. [66] M. Bianchini, M. Gori and M. Maggini, On the problem of local minima in recurrent neural networks IEEE Trans. Neural Networks, 5 (March 1994) 167-177. Special Issue on Recurrent Neural Networks.
67. [67] P. Frasconi, M. Gori, M. Maggini and G. Soda, Unified integration of explicit rules and learning by example in recurrent networks, IEEE Trans. Knowledge and Data Eng., in press.
68. [68] Y. Lee, S. Oh and M. Kim, The effect of weights on premature saturation in back-propagation learning, in Int. Joint Conf. on Neural Networks, vol. 1, Seattle (July 8-12, 1991) 765-770.
69. [69] K. Doya, Bifurcations of recurrent neural networks in gradient descent learning, Connectionist News Neuroprose.
70. [70] E. Sontag and H. Sussman, Backpropagation can give rise to spurious local minima even for networks without hidden layers, Complex Syst. 3 (1989) 91-106.
71. [71] M. Gori and A. Tesi, Some examples of local minima during learning with backpropagation, in Parallel Architectures and Neural Networks, Vietri sul Mare, Italy (May 1990).
72. [72] J. McInerny, K. Haines, S. Biafore and R. Hecht-Nielsen, Back propagation error surfaces can have local minima, in Int. Joint Conf. on Neural Networks, vol. 2, Washington (1989) (IEEE, New York, 1989) 627.
73. [73] E. Blum, Approximation of boolean functions by sigmoidal networks: Part 1: XOR and other two-variable functions, Neural Networks 1 (1989) 532-540.
74. [74] M. Gori, Apprendimento con supervisione in reti neuronali, PhD thesis, Università degli Studi di Bologna, Feb. 1990. In Italian.
75. [75] Y. Bengio, P. Frasconi and P. Simard, Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Networks (March 1994) 157-166. Special Issue on Recurrent Neural Networks.
76. [76] C. Drago and S. Ridella, Statistically controlled activation weight initialization (SCAWI), IEEE Trans. Neural Networks 3 (4) (1992) 627-631.
77. [77] J. Schmidthuber, Learning complex, extended sequences using the principle of history compression, Neural Computat. 4 (1992) 234-242.
78. [78] M. Gori, M. Maggini and G. Soda, Scheduling of modular architectures for inductive inference of regular grammars, in Workshop on Combining Symbolic and Connectionist Processing, ECAI '94, Amsterdam (Aug. 1994) 78-87.
79. [79] P. Frasconi, M. Gori and G. Soda, Local feedback multi-layered networks, Neural Computat. 4 (1) (1992) 120-130.
80. [80] N. Nilsson, Principles of Artificial Intelligence (Palo Alto, CA, Tioga, 1980).
81. [81] J. Anderson and E. Rosenfeld, eds., Neurocomputing: Foundations of Research (Cambridge, MIT Press, 1988).