Notions of associative memory and sparse coding

Neural Networks

Volumn 9, Issue 8, 1996, Pages 1429-1458

  Okada, Masato ^a  

^a OSAKA UNIVERSITY   (Japan)

Author keywords

associative memory; basin of attraction; nonmonotonic output function; sparse coding; storage capacity

Indexed keywords

ASSOCIATIVE STORAGE; MATHEMATICAL MODELS; NEUROPHYSIOLOGY; PATTERN RECOGNITION SYSTEMS; STATISTICAL METHODS;

NEURODYNAMICAL FUNCTIONS; NONMONOTONIC OUTPUT FUNCTION; SPARSE CODING;

BRAIN MODELS;

ARTIFICIAL NEURAL NETWORK; ASSOCIATIVE MEMORY; PRIORITY JOURNAL; REVIEW; STORAGE;

EID: 0030297197     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(96)00044-5     Document Type: Article

References (110)

1. Abeles M., Vaadia E., Bergman H. Firing patterns of single units in the prefrontal cortex and neural network models. Network. 1:1990;13-25.
2. Amari S. Learning patterns and pattern sequence by self-organizing nets of threshold elements. IEEE Transactions on Computers. 21:1972;1197-1206.
3. Amari S. Neural theory of association and concept-formation. Biological Cybernetics. 26:1977;175-185.
4. Amari S. Characteristics of sparsely encoded associative memory. Neural Networks. 2:1989;451-457.
5. Amari, S., & Arbib, M. A. (1977). Competition and cooperation in neural nets. In J. Metzler (Ed.), Systems neuroscience (pp. 119-165). New York: Academic Press.
6. Amari S., Maginu K. Statistical neurodynamics of associative memory. Neural Networks. 1:1988;63-73.
7. Amit D.J., Gutfreund H., Sompolinsky H. Spin-glass models of neural networks. Physical Review A. 32:1985;1007-1018.
8. Amit D.J., Gutfreund H., Sompolinsky H. Storing infinite numbers of patterns in a spin-glass model of neural networks. Physical Review Letters. 55:1985;1530-1533.
9. Amit D.J., Gutfreund H., Sompolinsky H. Information storage in neural networks with low levels of activity. Physical Review A. 35:1987;2293-2303.
10. Amit D.J., Brunel N., Tsodyks M.V. Correlations of cortical Hebbian reverberations: theory versus experiment. Journal of Neuroscience. 14:1994;6435-6445.
11. Anderson J.A. A simple neural network generating interactive memory. Mathematical Biosciences. 14:1972;197-220.
12. Bear M.F., Cooper L.N., Ebner F.F. A physiological basis for a theory of synapse modification. Science. 237:1987;42-48.
13. Binder, K. (1980). In E. G. D. Cohen (Ed.), Fundamental problems in statistical mechanics V (p. 21). Amsterdam: North-Holland.
14. Boffetta G., Monasson R., Zecchina R. Symmetry breaking in non-monotonic neural networks. Journal of Physics A: Mathematical and General. 26:1993;L507-L513.
15. Brunel N. Dynamics of an attractor neural network converting temporal into spatial correlations. Networks. 5:1994;449-470.
16. Buhl E.H., Halasy K., Somogyi P. Diverse source of hippocampal unitary inhibitory postsynaptic potentials and the number of synaptic release sites. Nature. 368:1994;823-828.
17. Buhmann J., Divko R., Schulten K. Associative memory with high information content. Physical Review A. 39:1989;2689-2692.
18. Coolen A.C.C., Sherrington D. Dynamics of fully connected attractor neural networks near saturation. Physical Review Letters. 71:1993;3886-3889.
19. Coolen A.C.C., Franz S. Closure of macroscopic laws in disordered spin systems: a toy model. Journal of Physics A: Mathematical and General. 27:1994;6947-6954.
20. Cover T.M. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers. 14:1965;326-334.
21. Cugliandolo L.F. Correlated attractors from uncorrelated stimuli. Neural Computation. 6:1994;220-224.
22. Cugliandolo L.F., Tsodyks M.V. Capacity of networks with correlated attractors. Journal of Physics A: Mathematical and General. 27:1994;741-753.
23. Derrida B., Gardner E., Zippelius A. An exactly solvable asymmetric neural network model. Europhysics Letters. 4:1987;167-173.
24. Erichsen R. Jr., Theumann W.K. Mixture states and storage with correlated patterns in Hopfield's model. International Journal of Neural Systems. 1:1991;347-353.
25. Fontanari J.F., Köberle R. Information storage and retrieval in synchronous neural networks. Physical Review A. 36:1987;2475-2477.
26. Fu A.M.N. Statistical analysis of an autoassociative memory network. Neural Computation. 6:1994;837-841.
27. Fukai T., Shiino M. Large suppression of spurious states in neural networks of nonlinear analog neurons. Physical Review a. 42:1990;7459-7466.
28. Fukai T., Shiino M. Comparative study of spurious-state distribution in analogue neural networks and Boltzmann machine. Journal of Physics A: Mathematical and General. 25:1992;2873-2887.
29. Fukai T., Kim J., Shiino M. Retrieval properties of analog neural networks and the nonmonotonicity of transfer functions. Neural Networks. 8:1995;391-404.
30. Gardner E. Structure of metastable states in the Hopfield model. Journal of Physics A: Mathematical and General. 19:1986;L1047-L1052.
31. Gardner E. The space of interactions in neural network model. Journal of Physics A: Mathematical and General. 21:1988;257-270.
32. Gardner E., Derrida B., Mottishaw P. Zero temperature parallel dynamics for infinite range spin glasses and neural networks. Journal de Physique. 48:1987;741-755.
33. Geszti, T. (1990). Physical models of neural networks. Singapore: World Scientific.
34. Glauber R.J. Journal of Mathematical Physics. 4:1963;294.
35. Golomb D., Sompolinsky H. Willshaw model: associative memory with sparse coding and low firing rates. Physical Review A. 41:1990;1843-1854.
36. Gomi, S., & Yonezawa, F. (in press). A new perturbation theory for the dynamics of the Little-Hopfield model. Submitted to Journal of Physics A: Mathematical and General.
37. Griniasty M., Tsodyks M.V., Amit D.J. Conversion of temporal correlations between stimuli to spatial correlations between attractors. Neural Computation. 5:1993;1-17.
38. Gulyás A.I., Miles R., Sík A., Tóth K., Tamamaki N., Freund T.F. Hippocampal pyramidal cells excite inhibitory neurons through a single release site. Nature. 366:1993;683-687.
39. Hassoum, M. (1993). Associative neural memories: theory and implementation. Oxford: The University Press.
40. Hebb, D. O. (1949). The organization of behavior. New York: John Wiley.
41. Hertz, J., Krogh, A., & Palmer, R. G. (1991). Introduction to the theory of neural computation. Reading, MA: Addison-Wesley.
42. Hopfield J.J. Neural networks and physical systems with emergent collective computational abilities. Proceedings National Academy of Sciences. 79:1982;2554-2558.
43. Hopfield J.J. Neurons with graded response have collective computational properties like those of two state neurons. Proceedings National Academy of Sciences. 81:1984;3088-3092.
44. Horn D., Ruppin E., Usher M., Herrmann M. Neural network modeling of memory deterioration in Alzheimer's disease. Neural Computation. 5:1993;736-749.
45. Inoue, J., & Nishimori, H. (1994). Gardner's capacity at finite temperature with non-monotonic input-output function (in Japanese). Abstracts of the Meeting of the Physical Society of Japan, 49th Annual Meeting Part 3, 569.
46. Kanter I., Sompolinsky H. Associative recall of memory without errors. Physical Review A. 35:1987;380-392.
47. Kinzel W. Learning and pattern recognition in spin glass model. Z. Physics B. 60:1985;205-213.
48. Kobayashi K. On the capacity of a neuron with a non-monotone output function. Network. 2:1991;237-243.
49. Kohonen T. Correlation matrix memories. IEEE Transactions on Computers. 21:1972;353-359.
50. Kohonen T., Ruohonen M. Representation of associated data by matrix operator. IEEE Transactions on Computers. 22:1973;701-702.
51. Koyama H., Fujie N., Fujiwara T. Analysis of the recalling processes of associative memory by an integral representation of the sign function. Neural Network. 9:1996;737-746.
52. Kühn S., Bös S., van Hemmen J.L. Statistical mechanics for networks of graded-response neurons. Physical Review A. 43:1991;2084-2087.
53. Kurata, K. (1993). Private communication.
54. Laughton S.N., Coolen A.C.C. Order-parameter flow in symmetric and nonsymmetric fully connected attractor neural networks. Physical Review E. 51:1995;2581-2599.
55. Li X.-G., Somogyi P., Ylinen A., Buzsáki G. The hippocampal CA3 network: an in vivo intracellular labeling study. Journal of Comparative Neurology. 339:1994;181-208.
56. Marcus C.M., Waugh F.R., Westervelt R.M. Associative memory in an analog iterated-map neural network. Physical Review A. 41:1990;3355-3364.
57. McCulloch W.S., Pitts W.H. A logical calculus of the ideas immanent in neural nets. Bulletin of Mathematical Biophysics. 5:1943;115-133.
58. McEliece R.J., Posner E.C., Rodemich E.R., Venkatesh S.S. The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory. 33:1987;461-482.
59. Meunier C., Yanai H., Amari S. Sparsely coded associative memories: capacity and dynamical properties. Network. 2:1991;469-487.
60. Mimura, K., Okada, M., & Kurata, K. (1993). Robustness of noise of associative memory using non-monotonic analogue neurons Proceedings of the 1993 International Joint Conference on Neural networks (IJCNN'93 Nagoya) Vol. 3, (pp. 2559-2562).
61. Miyashita Y. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature. 335:1988;817-820.
62. Miyashita Y., Chang H.S. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature. 331:1988;68-70.
63. Morita M. A neural network model of the dynamics of a short-term memory system in the temporal cortex (in Japanese). Transactions of the Institute of Electronics, Information and Communication Engineers of Japane. J74-D-II:1991;54-63.
64. Morita M. Associative memory with nonmonotone dynamics. Neural Networks. 6:1993;115-126.
65. Morita M., Yoshizawa S., Nakano K. Analysis and improvement of the dynamics of autocorrelation type associative memory (in Japanese). Transactions of the Institute of Electronics, Information and Communication Engineers of Japan. J73-D-II:1990;232-242.
66. Nakano K. Association - a model of associative memory. IEEE Transaction on Systems, Man and Cybernetics. 2:1972;381-388.
67. Nishimori H., Opris I. Retrieval process of an associative memory with a general input-output function. Neural Networks. 6:1993;1061-1067.
68. Nishimori H., Ozeki T. Retrieval dynamics of associative memory of the Hopfield type. Journal of Physics A: Mathematical and General. 26:1993;859-871.
69. Noble, J. J. A., & Tsien, R. W. (1975). Electrical current flow in excitable membranes. London: Oxford University Press.
70. Okada, M. (1994). Statistical neurodynamics of associative memory model with non-monotonic units (in Japanese). Proceedings of 1994 Annual Conference of Japanese Neural Networks Society, 76-77.
71. Okada M. A hierarchy of macrodynamical equations for associative memory. Neural Networks. 8:1995;833-838.
72. Okada, M. (1995b). Associative memory model with memory patterns distributed in clusters (in Japanese). Proceedings of 1995 Annual Conference of Japanese Neural Networks, Society, 38-39.
73. Okada, M., Mimura, K., & Kurata, K. (1993a). Sparsely encoded associative memory: Static synaptic noise and static threshold noise. Proceedings of the 1993 International Joint Conference on Neural Networks (IJCNN'93 Nagoya) (Vol. 3, pp. 2624-2627).
74. Okada, M., Mimura, K., & Kurata, K. (1993b). Memory capacity of associative memory and optimal threshold (in Japanese). (Technical Report of IEICE, NC93-17, 41-48.).
75. Okada, M., Minura, K., & Kurata, K. (1994a). Associative memory model using non-monotonic modules with feedforward inhibition (in Japanese). (Technical Report of IEICE, NC94-20, 49-54.).
76. Okada, M., Mimura, K., & Kurata, K. (1994b). Sparse coding and threshold (in Japanese). Transactions of the Institute of Electronics, Information and Communication Engineers of Japan, J77-D-II, 1934-1936.
77. Opris I. Dynamics of associative memory with a general transfer function. Physical Review E. 51:1995;2619-2623.
78. Ozawa, K., Fukai, T., Shiino, M., & Ohba, I. (1994). The distribution of the metastable states in neural network of non-monotonic neurons (in Japanese). Abstracts of the Meeting of the Physical Society of Japan, 49th Annual Meeting Part 3, 569.
79. Ozeki T., Nishimori H. Noise distribution in retrieval dynamics of the Hopfield model. Journal of Physics A: Mathematical and General. 27:1994;7061-7068.
80. Palm G. On associative memory. Biological Cybernetics. 36:1980;19-31.
81. Patrick A.E., Zagrebnov V.A. On the parallel dynamics for the Little-Hopfield Model. Journal of Statistical Physics. 63:1991;59-71.
82. Patrick A.E., Zagrebnov V.A. A probabilistic approach to parallel dynamics for the Little-Hopfield model. Journal of Physics A: Mathematical and General. 24:1991;3413-3426.
83. Pearlmutter B.A. Learning state space trajectories in recurrent neural networks. Neural Computation. 1:1989;263-269.
84. Peretto P. On learning rules and memory storage abilities of asymmetric neural networks. Journal de Physique. 49:1988;711-726.
85. Perez-Vicente C.J., Amit D.J. Optimized network for sparsely coded patterns. Journal of Physics A: Mathematical and General. 22:1989;559-569.
86. Pineda F.J. Dynamics and architecture for neural computation. Journal of Complexity. 4:1988;216-245.
87. Rieger H., Schreckenberg M., Zittartz J. Glauber dynamics of the Little-Hopfield model. Z. Physics B. 72:1988;523-533.
88. Rolls (1987). Information representation, processing, and storage in the brain: analysis at the single neuron level. In J.-P. Changeux & M. Konishi (Eds.). The neural and molecular bases of learning (pp. 503-539). New York: John Wiley.
89. Rolls E.T., Treves A. The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain. Network. 1:1990;407-421.
90. Sherrington D., Kirkpatrick S. Solvable model of a spin-glass. Physical Review Letters. 35:1975;1792-1796.
91. Shigemitsu S., Okada M. Statistical neurodynamics of the sparsely encoded associative memory (in Japanese). The Brain & Neural Networks. 3:1996;58-64.
92. Shiino, M. (1993). Private communication.
93. Shiino M., Fukai T. Replica-symmetric theory of the non-linear analogue neural networks. Journal of Physics A: Mathematical and General. 23:1990;L1009-L1017.
94. Shiino M., Fukai T. Self-consistent signal-to-noise analysis and its application to analogue neural networks with asymmetric connections. Journal of Physics A: Mathematical and General. 25:1992;L375-L381.
95. Shiino M., Fukai T. Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity. Physical Review E. 48:1993;867-897.
96. Shiino, M., & Fukai, T. (1994). Super retrieval phase in analog networks with positive valued nonmonotonic transfer function (in Japanese). Abstracts of the Meeting of the Physical Society of Japan, 49th Annual Meeting Part 3, 570.
97. Shiino, M., & Fukai, T. (1995). Super retrieval states in analog networks and the sparse coding (in Japanese). Abstracts of the Meeting of the Physical Society of Japan, 50th Annual Meeting Part 3, 606.
98. Sik A., Tamamaki N., Freund T.F. Complete axon arborization of a single CA3 pyramidal cell in the rat hippocampus, and its relationship with postsynaptic parvalbumin-containing interneurons. European Journal of Neuroscience. 5:1993;1719-1728.
99. Sompolinsky H. Neural networks with nonlinear synapses and a static noise. Physical Review A. 34:1986;2571-2574.
100. Tamori, Y. (1995). Biological neuronal model based on dendritic morphology (in Japanese). Proceedings of 1995 Annual Conference of Japanese Neural Networks Society, 141-142.
101. Treves A., Rolls E.T. What determines the capacity of autoassociative memories in the brain. Network. 2:1991;371-397.
102. Treves A., Rolls E.T. Computational constraints suggest the need for two distinct input systems to the hippocampal CA3 network. Hippocampus. 2:1992;189-199.
103. Tsodyks M.V., Feigel'man M.V. The enhanced storage capacity in neural networks with low activity level. Europhysics Letters. 6:1988;101-105.
104. Uesaka Y., Ozeki K. Some properties of associative type memories. Journal of Institute of Electrical and Communication Engineers of Japan. 55-D:1972;323-330.
105. Waugh F.R., Marcus C.M., Westervelt R.M. Fixed-point attractors in analog neural computation. Physical Review Letters. 64:1990;1986-1989.
106. Weisbuch G. Scaling laws for the attractors of Hopfield networks. Journal de Physique Letters. 46:1985;623-630.
107. White E.L., Keller A. Intrinsic circuitry involving the local axon collaterals of corticothalamic projection cells in mouse SmI cortex. Journal of Comparative Neurology. 262:1987;13-26.
108. Willshaw, D., & Longuet-Higgins, H. C. (1970). Associative memory model. In B. Meltzer & O. Michie (Eds.), Machine intelligence (Vol. 5). Edinburgh: Edinburgh University Press.
109. Yanai H., Amari S. Auto-associative memory with two-stage dynamics of non-monotonic neurons. IEEE Transactions on Neural Networks. 7:1996;803-815.
110. Yoshizawa S., Morita M., Amari S. Capacity of associative memory using a nonmonotonic neuron model. Neural Networks. 6:1993;167-176.