Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron

Physics Report

Volumn 342, Issue 4-5, 2001, Pages 263-392

  Gyorgyi G ^a  

^a EÖTVÖS LORÁND UNIVERSITY   (Hungary)

Author keywords

07.05.Mh; 61.43. j; 75.10Nr; 84.35.+i; Neural networks; Pattern storage; Replica symmetry breaking; Spin glasses

Indexed keywords

EID: 0035255691     PISSN: 03701573     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0370-1573(00)00073-9     Document Type: Review

References (235)

1. McCulloch W.S., Pitts W. A logical calculus of ideas immanent in nervous activity. Bull. Math. Biophys. 5:1943;115.
2. Rosenblatt F. Principles of Neurodynamics. 1962;Spartan, New York.
3. Gardner E. Maximum storage capacity in neural networks. Europhys. Lett. 4:1987;481.
4. Gardner E. The space of interactions in neural network models. J. Phys. A. 21:1988;257.
5. Gardner E., Derrida B. Optimal storage properties of neural network models. J. Phys. A. 21:1988;271.
6. Griniasty M., Gutfreund H. Learning and retrieval in attractor neural networks. J. Phys. A. 24:1991;715.
7. Majer P., Engel A., Zippelius A. Perceptrons above saturation. J. Phys. A. 26:1993;7405.
8. Erichsen R., Theumann W.K. Optimal storage of a neural network model: A replica symmetry-breaking solution. J. Phys. A. 26:1993;L61.
9. Bouten M. Replica symmetry instability in perceptron models. J. Phys. A. 27:1994;6021.
10. Derrida B. Reply to the comment of M. Bouten. J. Phys. A. 27:1994;6025.
11. Whyte W., Sherrington D. Replica-symmetry breaking in perceptrons. J. Phys. A. 29:1996;3063.
12. West A.H.L., Saad D. Threshold induced phase transitions in perceptrons. J. Phys. A. 30:1997;3471.
13. Garey M., Johnson D.S. Computers and Intractability; A Guide to the Theory of NP-Completeness. 1979;Freeman, San Francisco.
14. Mézard M., Parisi G., Virasoro M. Spin Glass Theory and Beyond. 1987;World Scientific, Singapore.
15. Monasson R., Zecchina R. Tricritical points in random combinatorics: the (2+p)-SAT case. J. Phys. A. 31:1998;9209.
16. Papadimitriou C.H. Computational Complexity. 1994;Addison-Wesley, Reading, MA.
17. Györgyi G., Reimann P. Parisi phase in a neuron. Phys. Rev. Lett. 79:1997;2746.
18. Györgyi G., Reimann P. Beyond storage capacity in a single model neuron: continuous replica symmetry breaking. J. Stat. Phys. 101:2000;679.
19. Hertz J., Krogh A., Palmer R.G. Introduction to the Theory of Neural Computation. 1991;Addison-Wesley, Reading, MA.
20. Parisi G. A sequence of approximated solutions to the S-K model for spin glasses. J. Phys. A. 13:1980;L115.
21. Parisi G. The order parameter for spin glasses: A function on the interval 0-1. J. Phys. A. 13:1980;1101.
22. Parisi G. Magnetic properties of spin glasses in a new mean field theory. J. Phys. A. 13:1980;1887.
23. Parisi G. The magnetic properties of the Sherrington-Kirkpatrick model for spin glasses. Theory versus Monte Carlo simulations. Philos. Mag. 41:1980;677.
24. Gardner E. Spin glasses with p-spin interactions. Nucl. Phys. B. 257:1985;747.
25. Gross D.J., Kanter I., Sompolinsky H. Mean-field theory of the Potts glass. Phys. Rev. Lett. 55:1985;304.
26. Bray A.J., Moore M.A. Metastable states in the solvable spin glass model. J. Phys. A. 14:1981;L377.
27. de Almeida J.R.L., Lage E.J.S. Internal field distribution in the infinite-range Ising spin glass. J. Phys. C. 16:1983;939.
28. Mézard M., Virasoro M.A. The microstructure of ultrametricity. J. Phys. France. 46:1985;1293.
29. Sommers H.-J., Dupont W. Distribution of frozen fields in the mean-field theory of spin glasses. J. Phys. C. 17:1984;5785.
30. Mézard M., Parisi G., Sourlas N., Toulouse G., Virasoro M. Replica symmetry breaking and the nature of the spin glass phase. J. Phys. France. 45:1984;843.
31. Tokita K. The replica-symmetry-breaking solution of the Hopfiel model at zero temperature: Critical storage capacity and frozen field distribution. J. Phys. A. 27:1994;4413.
32. Sompolinsky H., Zippelius A. Dynamic theory of the spin-glass phase. Phys. Rev. Lett. 47:1981;359.
33. Sompolinsky H., Zippelius A. Relaxational dynamics of the Edwards-Anderson model and the mean field theory of spin-glasses. Phys. Rev. B. 25:1982;6860.
34. Sompolinsky H. Time-dependent order parameters in spin-glasses. Phys. Rev. Lett. 47:1981;935.
35. Sompolinsky H. Spin-glass transition in a magnetic field. Philos. Mag. B. 51:1985;543.
36. Sommers H.-J. Path-integral approach to Ising spin-glass dynamics. Phys. Rev. Lett. 58:1987;1268.
37. Wendemuth A. Performance of robust training algorithms for neural networks. J. Phys. A. 28:1995;5485.
38. Widrow B., Hoff M.E. Adaptive switching circuits. IRE WESCON Conversation Record. 4:1960;96.
39. Widrow B. Generalization and information storage in networks of adaline "neurons" Yovits M.C., Jacobi G.T., Goldstein G.D. Self-Organizing Systems. 1962;Spartan, Washington.
40. Minsky M.L., Papert S.A. Perceptrons. 1969;MIT Press, Cambridge, MA.
41. Rumelhart D.E., Hinton G.E., Williams R.J. Learning representations by backpropagating errors. Nature. 323:1986;533.
42. Hornik K., Stinchcombe M., White W. Multilayer feed-forward networks are universal approximators. Neural Networks. 2:1989;359.
43. Hartman E.J., Keeler J.D., Kowalski J.M. Layered neural networks with Gaussian hidden units as universal approximations. Neural Comput. 2:1990;210.
44. Taylor W.K. Electrical simulations of some nervous system functional activities. Cherry C. Information Theory. 1956;Butterworths, London.
45. K. Steinbuch, Die Lernmatrix, Kybernetik 1 (1961) 36 (in German).
46. Cragg B.G., Temperley H.N.V. The organization of neurons: a cooperative analogy. EEG Clin. Neurophysiol. 6:1954;85.
47. Cragg B.G., Temperley H.N.V. Memory: the analogy with ferromagnetic hysteresis. Brain. 78 II:1955;304.
48. Caianiello E.R. Outline of a theory of thought and thinking machines. J. Theoret. Biol. 1:1961;204.
49. Sompolinsky H., Crisanti A., Sommers H.-J. Chaos in random neural networks. Phys. Rev. Lett. 61:1988;259.
50. Kanter I. Emergence of chaos in asymmetric networks. Phys. Rev. Lett. 77:1996;4844.
51. Little W.A. The existence of persistent states in the brain. Math. Biosci. 19:1974;101.
52. Little W.A., Shaw G.L. A statistical theory of short and long term memory. Behav. Biol. 14:1975;115.
53. Hebb D.O. The Organization of Behavior. 1949;Wiley, New York.
54. Hopfield J.J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA. 79:1982;2554.
55. Hopfield J.J. Neurons with graded response have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. USA. 81:1984;3088.
56. Heerma M., van Leeuwen W.A. Derivation of Hebb's rule. J. Phys. A. 32:1999;263.
57. Binder K., Young A.P. Spin glasses: experimental facts, theoretical concepts, and open questions. Rev. Mod. Phys. 58:1986;801.
58. Toulouse G. Theory of the frustration effect in spin glasses. Commun. Phys. 2:1977;115.
59. Edwards S.F., Anderson P.W. Theory of spin glasses. J. Phys. F. 5:1975;965.
60. Sherrington D., Kirkpatrick S. Solvable model of a spin-glass. Phys. Rev. Lett. 35:1975;1792.
61. Kirkpatrick S., Sherrington D. Infinite-ranged models of spin-glasses. Phys. Rev. B. 11:1978;4384.
62. Derrida B. Random-energy model: an exactly solvable model of disordered systems. Phys. Rev. B. 24:1981;2613.
63. Gross D., Mézard M. The simplest spin glass. Nucl. Phys. B. 240:1984;431.
64. Crisanti A., Sommers H.-J. The spherical p-spin interaction spin glass model: the statics. Z. Phys. B. 87:1992;341.
65. Gabay M., Toulouse G. Coexistence of spin-glass and ferromagnetic ordering. Phys. Rev. Lett. 47:1981;201.
66. Erzan A., Lage E.J.S. The infinite-ranged Potts spin glass model. J. Phys. C. 16:1983;L555.
67. Elderfield D., Sherrington D. The curious case of the Potts spin glass. J. Phys. C. 16:1983;L497.
68. Nieuwenhuizen Th.M. Exactly solvable model of a quantum spin glass. Phys. Rev. Lett. 74:1995;4289.
69. Nieuwenhuizen Th.M. Quantum description of spherical spins. Phys. Rev. Lett. 74:1995;4293.
70. Oppermann R., Rosenow B. Low-energy excitations in fermionic spin glasses: a quantum-dynamical image of Parisi symmetry breaking. Europhys. Lett. 41:1998;525.
71. Villain J., Séméria B., Lancon F., Billard L. A controversial problem: modified Ising model in a random field. J. Phys. C. 16:1983;6153.
72. Kardar M. Replica Bethe ansatz studies of two-dimensional interfaces with quenched random impurities. Nucl. Phys. B. 290:1987;582.
73. Mézard M., Parisi G. Replica field theory for random manifolds. J. Phys. I France. 1:1991;809.
74. Engel A. Replica symmetry breaking in zero dimension. Nucl. Phys. B. 410:1993;617.
75. Le Doussal P., Giamarchi T. Replica symmetry breaking instability in the 2D XY model in a random field. Phys. Rev. Lett. 74:1995;606.
76. Arenzon J.J. Replica theory of granular media. J. Phys. A. 32:1999;L107.
77. Berg J., Engel A. Matrix games, mixed strategies, and statistical mechanics. Phys. Rev. Lett. 81:1998;4999.
78. Garel T., Orland H. Mean-field model for protein folding. Europhys. Lett. 6:1988;307.
79. Shakhnovich E.I., Gutin A.M. Frozen states of a disordered globular heteropolymer. J. Phys. A. 22:1989;1647.
80. Sasai M., Wolynes P.G. Unified theory of collapse, folding, and glass transitions in associative-memory Hamiltonian models of proteins. Phys. Rev. A. 46:1992;7979.
81. Takada S., Wolynes P.G. Statistics, metastable states, and barriers in protein folding: A replica variational approach. Phys. Rev. E. 55:1997;4562.
82. Pagnoni A., Parisi G., Ricci-Tersenghi F. Glassy transition in a disordered model for the RNA secondary structure. Phys. Rev. Lett. 84:1999;2026.
83. van Mourik J., Wong K.Y.M., Bollé D. From shrinking to percolation in an optimization model. J. Phys. A. 33:2000;L53.
84. Fischer K.H., Hertz J.A. Spin Glasses. 1991;Cambridge University Press, Cambridge, UK.
85. Marinari E., Naitza C., Zuliani F., Parisi G., Picco M., Ritort F. General method to determine replica symmetry breaking transitions. Phys. Rev. Lett. 81:1998;1698.
86. Bokil H., Bray A.J., Drossel B., Moore M.A. Comment on "General method to determine replica symmetry breaking transitions" Phys. Rev. Lett. 82:1999;5174.
87. Marinari E., Naitza C., Zuliani F., Parisi G., Picco M., Ritort F. Marinari et al. reply. Phys. Rev. Lett. 82:1999;5175.
88. Moore M.A., Bokil H., Drossel B. Evidence for the droplet picture of spin glasses. Phys. Rev. Lett. 81:1998;4252.
89. Marinari E., Parisi G., Ruiz-Lorenzo J.J., Zuliani F. Comment on "Evidence for the droplet picture of spin glasses" Phys. Rev. Lett. 82:1999;5176.
90. Bokil H., Bray A.J., Drossel B., Moore M.A. Bokil et al. reply. Phys. Rev. Lett. 82:1999;5177.
91. Marinari E., Parisi G., Ruiz-Lorenzo J.J. Numerical simulations of spin glass systems. Young A.P. Spin Glasses and Random Fields. 1998;World Scientific, Singapore.
92. Pastur L.A., Shcherbina M.V. Absence of self-averaging of the order parameter in the Sherrington-Kirkpatrick model. J. Stat. Phys. 62:1991;1.
93. Feng J. The SLLN for the free-energy of a class of neural networks. Helv. Phys. Acta. 68:1995;365.
94. Bovier A., Gayrard V., Picco P. Gibbs states of the Hopfield model with extensively many patterns. J. Stat. Phys. 79:1995;395.
95. van Hemmen J.L., Palmer R.G. The replica method and a solvable spin glass model. J. Phys. A. 12:1979;563.
96. de Almeida J.R.L., Thouless D.J. Stability of the Sherrington-Kirkpatrick solution of a spin glass model. J. Phys. A. 11:1978;983.
97. Thouless D.J., de Almeida J.R.L., Kosterlitz J.M. Stability and susceptibility in Parisi's solution of a spin glass model. J. Phys. C. 13:1980;3271.
98. De Dominicis C., Kondor I. Eigenvalues of the stability matrix for Parisi solution of the long-range spin-glass. Phys. Rev. B. 27:1983;606.
99. Nieuwenhuizen Th.M. To maximize or not to maximize the free energy of glassy systems. Phys. Rev. Lett. 74:1995;3463.
100. Kondor I. Parisi's mean-field solution for spin glasses as an analytic continuation in the replica number. J. Phys. A. 16:1983;L127.
101. Pytte E., Rudnick J. Scaling, equation of state and the instability of the spin-glass phase. Phys. Rev. B. 19:1979;3603.
102. A. Blandin, Theories versus experiments in the spin glass systems, J. Phys. France 39 (1978) C6-1499.
103. Bray A.J., Moore M.A. Replica-symmetry breaking in spin-glass theories. Phys. Rev. Lett. 41:1978;1068.
104. Bray A.J., Moore M.A. Replica symmetry and massless modes in the Ising spin glass. J. Phys. C. 12:1979;79.
105. Blandin A., Gabay M., Garel T. On the mean-field theory of spin glasses. J. Phys. C. 13:1980;403.
106. De Dominicis C., Gabay M., Orland H. Replica derivation of Sompolinsky free energy functional for mean field spin glass. J. Phys. Lett. France. 42:1981;L523.
107. De Dominicis C. Broken symmetry in the mean-field theory of the Ising spin glass: Replica way and no replica way. Van Hemmen J.L., Morgenstern I. Heidelberg Colloquium on Spin Glasses. 1983;Springer, Berlin.
108. Parisi G. Towards a mean field theory for spin glasses. Phys. Lett. A. 73:1979;203.
109. Parisi G. Infinite number of order parameters for spin-glasses. Phys. Rev. Lett. 43:1979;1754.
110. Parisi G. Order parameter for spin glasses. Phys. Rev. Lett. 50:1983;1946.
111. Temesvári T., De Dominicis C., Kondor I. Block diagonalizing ultrametric matrices. J. Phys. A. 27:1994;7569.
112. Thouless D.J., Anderson P.W., Palmer R.G. Solution of "solvable model of a spin glass" Philos. Mag. 35:1977;593.
113. Sommers H.-J. Solution of the long-range Gaussian-random Ising model. Z. Phys. B. 31:1978;301.
114. Sommers H.-J. The Sherrington-Kirkpatrick spin glass model: results of a new theory. Z. Phys. B. 33:1979;173.
115. Bray A.J., Moore M.A. Some observations on the mean-field theory of spin glasses. J. Phys. C. 13:1980;419.
116. Sompolinsky H., Zippelius A. Relaxational dynamics of the Edwards-Anderson model and the mean-field theory of spin-glasses. Phys. Rev. B. 25:1982;6860.
117. De Dominicis C., Gabay M., Duplantier B. A Parisi equation for Sompolinsky's solution of the SK model. J. Phys. A. 15:1982;L47.
118. Mézard M., Parisi G., Virasoro M.A. SK model: the replica solution without replicas. Europhys. Lett. 1:1986;77.
119. Diener G., Brusch L. A new simple version of the replica method. J. Phys. A. 32:1998;585.
120. Domb C., Green M.S. Phase Transitions and Critical Phenomena, Vol. 1. 1972;Academic Press, London.
121. Mackenzie N.D., Young A.P. Lack of ergodicity in the infinite-range Ising spin-glass. Phys. Rev. Lett. 49:1982;301.
122. Houghton A., Jain S., Young A.P. Role of initial conditions in the mean-field theory of spin-glass dynamics. Phys. Rev. B. 28:1983;2630.
123. Young A.P. Direct determination of the probability distribution for the spin-glass order parameter. Phys. Rev. Lett. 51:1983;1206.
124. Mézard M., Parisi G., Virasoro M.A. Random free energies in spin glasses. J. Phys. Lett. 46:1985;L217.
125. Newman C.M., Stein D.L. Simplicity of state and overlap structure in finite-volume realistic spin glasses. Phys. Rev. E. 57:1998;1356.
126. Monasson R., O'Kane D. Domains of solutions and replica symmetry breaking in multilayer neural networks. Europhys. Lett. 27:1994;85.
127. Engel A., Reimers L. Reliability of replica symmetry for the generalization problem of a toy multilayer neural network. Europhys. Lett. 28:1994;531.
128. Bex G.-J., Serneels R., Van den Broeck C. Storage capacity and generalization error for the reversed-wedge Ising perceptron. Phys. Rev. E. 51:1995;6309.
129. Aizenman M., Lebowitz J.L., Ruelle D. Some rigorous results on the Sherrington-Kirkpatrick spin glass model. Commun. Math. Phys. 112:1987;3.
130. Shcherbina M. On the replica symmetric solution for the Sherrington-Kirkpatrick model. Helv. Phys. Acta. 70:1997;838.
131. Guerra F. The cavity method in the mean field spin glass model. Functional representation of thermodynamical variables. Albeverio A., Figari R., Orlandi E., Teta A. Advances in Dynamical Systems and Quantum Physics. 1995;World Scientific, Singapore.
132. Guerra F. Fluctuations and thermodynamic variables in mean field spin glass models. Albeverio A., Cattaneo U., Merlini D. Stochastic Processes, Physics, and Geometry. 1995;World Scientific, Singapore.
133. Ghirlanda S., Guerra F. General properties of overlap probability distributions in disordered spin systems. Towards Parisi ultrametricity. J. Phys. A. 31:1998;9149.
134. Guerra F. About the overlap distribution in mean field spin glass models. Int. J. Mod. Phys. B. 10:1998;1675.
135. Amit D.J., Gutfreund H., Sompolinsky H. Storing infinite number of patterns in a spin-glass model of neural networks. Phys. Rev. Lett. 55:1985;1530.
136. Amit D.J., Gutfreund H., Sompolinsky H. Statistical mechanics of neural networks near saturation. Ann. Phys. 173:1987;30.
137. Amit D.J. Modeling Brain Function. 1989;Cambridge University Press, Cambridge, UK.
138. Crisanti A., Amit D.J., Gutfreund H. Saturation level of the Hopfield model for neural networks. Europhys. Lett. 2:1986;337.
139. Steffan H., Kühn R. Replica symmetry breaking in attractor neural network models. Z. Phys. B. 95:1994;249.
140. G. Parisi, G. Toulouse, J. Phys. Lett. France 41 (1980) L361.
141. Horner H., Bormann D., Frick M., Kinzelbach H., Schmidt A. Transients and basins of attraction in neural network models. Z. Phys. B. 76:1989;381.
142. Stiefvater Th., Müller K.-R., Kühn R. Averaging and finite-size analysis for disorder: The Hopfield model. Physica A. 232:1996;61.
143. Geszti T. Physical Models of Neural Networks. 1990;World Scientific, Singapore.
144. Bouten M. Storage capacity of the Hopfield model. Srinivasa Rao K., Satpathy L. Perspectives in Theoretical Nuclear Physics. 1994;Wiley Eastern, India.
145. Rieger H., Schreckenberg M., Zittartz J. Glauber dynamics of the Little-Hopfield model. Z. Phys. B. 72:1988;523.
146. Coolen A.C.C., Sherrington D. Dynamics of fully connected attractor neural networks near saturation. Phys. Rev. Lett. 23:1993;3886.
147. Shiino M., Fukai T. Self-consistent signal-to-noise analysis of statistical behavior of analog neural networks and enhancement of the storage capacity. Phys. Rev. E. 48:1993;867.
148. Pastur L., Shcherbina M., Tirozzi B. The replica-symmetric solution without replica trick for the Hopfield model. J. Stat. Phys. 74:1994;1161.
149. Winder R.O. Switching circuit theory and logical design. AIEE Special Publ. S-134:1961;321.
150. Cover T.M. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans. Electron. Comput. 14:1965;326.
151. Venkatesh S. Epsilon capacity of neural networks. Denker J.S. Neural Networks for Computing, AIP Conference Proceedings, Vol. 151. 1986;AIP, New York.
152. Baldi P., Venkatesh S. Number of stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Lett. 58:1987;913.
153. Mitchison G.J., Durbin R.M. Bounds on the learning capacity of some multi-layer networks. Biol. Cybernet. 60:1989;345.
154. Amaldi E. On the complexity of training perceptrons. Kohonen T., Mäkisara K., Simula O., Kangas J. Artificial Neural Networks. 1991;North-Holland, Amsterdam.
155. Wendemuth A. Learning the unlearnable. J. Phys. A. 28:1995;5423.
156. Rujan P. Searching for optimal configurations by simulated tunneling. Z. Phys. B. 73:1988;391.
157. Gallant S. Perceptron-based learning algorithms. IEEE Trans. Neural Networks. 1:1990;179.
158. Frean M. A thermal perceptron learning rule. Neural Comput. 4:1992;946.
159. Dittes F.-M. Optimization on rugged landscapes: a new general purpose Monte Carlo approach. Phys. Rev. Lett. 76:1996;4651.
160. Kirkpatrick S., Gelatt C.D. Jr., Vecchi M.P. Optimization by simulated annealing. Science. 220:1983;671.
161. Wenzel W., Hamacher K. Stochastic tunneling approach for global minimization of complex potential energy landscapes. Phys. Rev. Lett. 82:1999;3003.
162. Mézard M. The space of interactions in neural networks: Gardner's calculation with the cavity method. J. Phys. A. 22:1989;2181.
163. Griniasty M. "Cavity-approach" analysis of the neural-network learning problem. Phys. Rev. E. 47:1993;4496.
164. Bouten M., Schietse J., Van den Broeck C. Gradient descent learning in perceptrons: a review of its possibilities. Phys. Rev. E. 52:1995;1958.
165. Wong K.Y.M. Microscopic equations and stability conditions in optimal neural networks. Europhys. Lett. 30:1995;245.
166. Derrida B., Griffiths R.B., Prügel-Bennett A. Finite-size effects and bounds for perceptron models. J. Phys. A. 24:1991;4907.
167. R. Monasson, R. Zecchina, Weight space structure and internal representations: a direct approach to learning and generalization in multilayer neural networks, Phys. Rev. Lett. 75 (1995) 2432; Erratum 76 (1996) 2205.
168. Monasson R., Zecchina R. Learning and generalization theories of large committee-machines. Mod. Phys. Lett. B. 9:1996;1887.
169. Cocco S., Monasson R., Zecchina R. Analytical and numerical study of internal representations in multilayer neural networks with binary weights. Phys. Rev. E. 54:1996;717.
170. Weigt M., Engel A. Multifractality and percolation in the coupling space of perceptrons. Phys. Rev. E. 55:1997;4552.
171. Rieger H., Seung H.S. Vapnik-Chervonenkis entropy of the spherical perceptron. Phys. Rev. E. 55:1996;3283.
172. Van den Broeck C., Bex G.-J. Multifractal a priori probability distribution for the perceptron. Phys. Rev. E. 57:1998;3660.
173. Halsey T.C., Jensen M.H., Kadanoff L.P., Procaccia I., Shraiman B.I. Fractal measures and their singularities: the characterization of strange sets. Phys. Rev. A. 33:1986;1141.
174. Tél T. Fractals, multifractals, and thermodynamics. Z. Naturforsch. 43a:1988;1154.
175. Krauth W., Mézard M. Storage capacity of memory networks with binary couplings. J. Phys. France. 50:1989;3057.
176. Fontanari J.F., Meir R. The statistical mechanics of the Ising perceptron. J. Phys. A. 26:1993;1077.
177. Pitt L., Valiant L.G. Computational limitations on learning from examples. J. ACM. 35:1988;965.
178. G.-J. Bex, Tuning the transfer function: the reversed wedge and beyond, Ph.D. Thesis, L.U.C. Diepenbeek, Belgium, 1996.
179. Nadler W., Fink W. Finite size scaling in neural networks. Phys. Rev. Lett. 78:1997;555.
180. Schröder M., Urbanczik R. Comment on "Finite size scaling in neural networks" Phys. Rev. Lett. 80:1998;4109.
181. Milde G., Kobe S. An exact learning algorithm for autoassociative neural networks with binary couplings. J. Phys. A. 30:1997;2349.
182. Bouten M., Reimers L., van Rompaey B. Learning in the hypercube: A stepping stone to the binary perceptron. Phys. Rev. E. 58:1998;2378.
183. Engel A., Weigt M. Multifractal analysis of the coupling space of feedforward neural networks. Phys. Rev. E. 53:1996;2064.
184. G. Györgyi, unpublished.
185. Saad D. Explicit symmetries and the capacity of multilayer neural networks. J. Phys. A. 27:1994;2719.
186. Barkai E., Kanter I. Storage capacity of a multilayered neural network with binary weights. Europhys. Lett. 14:1991;107.
187. Barkai E., Hansel D., Sompolinsky H. Broken symmetries in multilayered perceptrons. Phys. Rev. A. 45:1992;4146.
188. Kepler T.B., Abbott L.F. Domains of attraction in neural networks. J. Phys. France. 49:1988;1657.
189. Gardner E. Optimal basins of attraction in randomly sparse neural network models. J. Phys. A. 22:1989;1969.
190. Gardner E., Derrida B., Mottishaw P. Zero temperature parallel dynamics for infinite range spin glasses and neural networks. J. Phys. France. 48:1987;741.
191. Forrest B.M. Content-addressability and learning in neural networks. J. Phys. A. 21:1988;245.
192. Krauth W., Nadal J.-P., Mézard M. The roles of stability and symmetry in the dynamics of neural networks. J. Phys. A. 21:1988;2995.
193. Henkel R.D., Opper M. Parallel dynamics of the neural network with the pseudoinverse coupling matrix. J. Phys. A. 24:1991;2201.
194. Derrida B., Gardner E., Zippelius A. An exactly solvable asymmetric neural network model. Europhys. Lett. 4:1987;167.
195. Amit D.J., Evans M.R., Horner H., Wong K.Y.M. Retrieval phase diagrams for attractor neural network with optimal interactions. J. Phys. A. 23:1990;3361.
196. Engel A., Köhler H.M., Tschepke F., Vollmayr H., Zippelius A. Storage capacity and learning algorithms for two-layer neural networks. Phys. Rev. A. 45:1992;7590.
197. Engel A. Correlations of internal representations in feed-forward neural networks. J. Phys. A. 29:1996;L323.
198. Malzahn D., Engel A., Kanter I. Storage capacity of correlated perceptrons. Phys. Rev. E. 55:1997;7369.
199. Malzahn D., Engel A. Correlations between hidden units in multilayer neural networks and replica symmetry breaking. Phys. Rev. E. 60:1999;2097.
200. Watkin T.L.H., Rau A., Biehl M. The statistical mechanics of learning a rule. Rev. Mod. Phys. 65:1993;499.
201. Schottky B. Phase transitions in the generalization behavior of multilayer neural networks. J. Phys. A. 28:1995;4515.
202. West A.H.L., Saad D. The statistical mechanics of constructive algorithms. J. Phys. A. 31:1998;8977.
203. Winther O., Lautrup B., Zhang J.-B. Optimal learning in multilayer neural networks. Phys. Rev. E. 55:1997;836.
204. Amari S. Natural gradient works efficiently in learning. Neural Comput. 10:1998;251.
205. Barkai E., Hansel D., Kanter I. Statistical mechanics of multilayered neural networks. Phys. Rev. Lett. 65:1990;2312.
206. Kwon C., Oh J.-H. Storage capacities of committee machines with overlapping and non-overlapping receptive fields. J. Phys. A. 30:1997;6273.
207. Urbanczik R. Storage capacity of the fully connected committee machine. J. Phys. A. 30:1997;L387.
208. Xiong Y., Oh J.-H., Kwon C. Weight space structure and the storage capacity of a fully connected committee machine. Phys. Rev. E. 56:1997;4540.
209. Xiong Y., Kwon C., Oh J.-H. Storage capacity of a fully-connected parity machine with continuous weights. J. Phys. A. 31:1998;7043.
210. Priel A., Blatt M., Grossman T., Domany E., Kanter I. Computational capabilities of restricted two-layered perceptrons. Phys. Rev. E. 50:1994;577.
211. Kobayashi K. On the capacity of a neuron with a non-monotone output function. Network. 2:1991;237.
212. Watkin T.L.H., Rau A. Learning unlearnable problems with perceptrons. Phys. Rev. A. 45:1992;4102.
213. De Felice P., Marangi C., Nardulli G., Pasquariello G., Tedesco L. Dynamics of neural networks with non-monotone activation function. Network. 4:1993;1.
214. Nishimori H., Opris I. Retrieval process of an associative memory with a general input-output function. Neural Networks. 6:1993;1061.
215. Boffetta G., Monasson R., Zecchina R. Symmetry breaking in non-monotone neural networks. J. Phys. A. 26:1993;L507.
216. Reimers L., Engel A. Weight space structure and generalization in the reversed-wedge perceptron. J. Phys. A. 29:1996;3923.
217. Mietzner A., Opper M., Kinzel W. Maximal stability in unsupervised learning. J. Phys. A. 28:1995;2785.
218. Lootens E., Van den Broeck C. Analysing cluster formation by replica method. Europhys. Lett. 30:1995;381.
219. Bolle D., Erichsen R. Jr. Optimal capacity of graded-response perceptrons: a replica-symmetry-breaking solution. J. Phys. A. 29:1996;2299.
220. Bex G.-J., Van den Broeck C. Domain sizes of the Gardner volume for the Ising reversed wedge perceptron. Phys. Rev. E. 56:1997;870.
221. Gardner E. Optimal basins of attraction in randomly sparse neural network models. J. Phys. A. 22:1989;1969.
222. G. Györgyi, N. Tishby, Statistical mechanics of learning a rule, in: W.K. Theumann, R. Köberle (Eds.), Proceedings of the STATPHYS-17 Workshop on Neural Networks and Spin Glasses, World Scientific, Singapore, 1990.
223. De Dominicis C., Temesvári T., Kondor I. On Ward-Takahashi identities for the Parisi spin glass. J. Phys. IV France. 8:1998;Pr6-13.
224. xx Commun. Pure Appl. Math. 3:1950;201.
225. Cole J.D. Quart. Appl. Math. 9:1951;225.
226. Duplantier B. Comment on Parisi's equation for the SK model for spin glasses. J. Phys. A. 14:1981;283.
227. G. Parisi, On the physical origin of ultrametricity, cond-mat / 9905189, 1999.
228. T. Temesvári, unpublished.
229. C. Itzykson, J.-M. Drouffe, Statistical Field Theory, Cambridge Monographs on Mathematical Physics, Vol. 2, Cambridge University Press, Cambridge, UK, 1989.
230. Crisanti A., Sommers H.-J. The spherical p-spin interaction spin glass model: The statics. Z. Phys. B. 87:1992;341.
231. Oliver P.J. Applications of Lie Groups to Differential Equations. 1986;Springer, New York.
232. Nieuwenhuizen Th.M., Duin C.N.A. Ginzburg-Landau theory of the cluster glass phase. J. Phys. A. 30:1997;L55.
233. Abramowitz M., Stegun I.A. Handbook of Mathematical Functions. 1970;Dover, New York, NY.
234. Derrida B. Random-energy model: limit of a family of disordered models. Phys. Rev. Lett. 45:1980;79.
235. A. Engel, C. Van den Broeck, Statistical Mechanics of Learning, Cambridge University Press, Cambridge UK, 2000.