Computational implications of lognormally distributed synaptic weights

Proceedings of the IEEE

Volumn 102, Issue 4, 2014, Pages 500-512

  Teramae, Jun Nosuke ^a   Fukai, Tomoki ^b  

^a OSAKA UNIVERSITY   (Japan)

^b RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

Author keywords

Associative memory (AM); feedforward networks; network connectivity; neural dynamics; neuromorphic engineering; principal component analysis (PCA); recurrent networks; sparse coding; spike sequence; spike timing dependent plasticity (STDP); stochastic resonance

Indexed keywords

ASSOCIATIVE PROCESSING; CIRCUIT RESONANCE; CODES (SYMBOLS); FEEDFORWARD NEURAL NETWORKS; MAGNETIC RESONANCE; NEURONS; PRINCIPAL COMPONENT ANALYSIS; STOCHASTIC SYSTEMS;

ASSOCIATIVE MEMORY; FEED-FORWARD NETWORK; NETWORK CONNECTIVITY; NEURAL DYNAMICS; NEUROMORPHIC ENGINEERING; RECURRENT NETWORKS; SPARSE CODING; SPIKE SEQUENCES; SPIKE TIMING DEPENDENT PLASTICITIES; STOCHASTIC RESONANCES;

RECURRENT NEURAL NETWORKS;

EID: 84897547385     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2014.2306254     Document Type: Article

References (56)

1. C. Holmgren, T. Harkany, B. Svennenfors, and Y. Zilberter, "Pyramidal cell communication within local networks in layer 2/3 of rat neocortex," J. Physiol., vol. 551, pp. 139-153, 2003.
2. A. Stepanyants, G. Tamás, and D. B. Chklovskii, "Class-specific features of neuronal wiring," Neuron, vol. 43 pp. 251-259, 2004.
3. N. Kalisman, G. Silberberg, and H. Markram, "The neocortical microcircuit as a tabula rasa," Proc. Nat. Acad. Sci., vol. 102, pp. 880-885, 2005.
4. Y. Yoshimura, J. L. M. Dantzker, and E. M. Callaway, "Excitatory cortical neurons form fine-scale functional networks," Nature, vol. 433, pp. 868-873, 2005.
5. R. Perin, T. K. Berger, and H. Markram, "A synaptic organizing principle for cortical neuronal groups," Proc. Nat. Acad. Sci. USA, vol. 108, pp. 5419-5424, 2011.
6. S. Song, P. J. Sjöström, M. Reigl, S. B. Nelson, and D. B. Chklovskii, "Highly nonrandom features of synaptic connectivity in local cortical circuits," PLoS Biol., vol. 3, no. 3, 2005, e68.
7. S. Lefort, C. Tomm, J. C. Floyd Sarria, and C. C. H. Petersen, "The excitatory neuronal network of the C2 barrel column in mouse primary somatosensory cortex," Neuron, vol. 61, pp. 301-316, 2009.
8. Y. Ikegaya, T. Sasaki, D. Ishikawa, N. Honma, K. Tao, N. Takahashi, G. Minamisawa, S. Ujita, and N. Matsuki, "Interpyramid spike transmission stabilizes the sparseness of recurrent network activity," Cereb. Cortex, vol. 23, no. 2, pp. 293-304, 2013.
9. B. Barbour, N. Brunel, V. Hakim, and J. P. Nadal, "What can we learn from synaptic weight distributions?" Trends Neurosci., vol. 30, no. 12, pp. 622-629, 2007.
10. L. Sarid, R. Bruno, B. Sakmann, I. Segev, and D. Feldmeyer, "Modeling a layer 4-to-layer 2/3 module of a single column in rat neocortex: Interweaving in vitro and in vivo experimental observations," Proc. Nat. Acad. Sci. USA, vol. 104, pp. 16353-16358, 2007.
11. J.-N. Teramae, Y. Tsubo, and T. Fukai, "Optimal spike-based communication in excitable networks with strong-sparse and weak-dense links," Sci. Rep., vol. 2, 2012, DOI: 10.1038/srep00485.
12. J. Teramae, Y. Tsubo, and T. Fukai, "Long-tailed statistics of corticocortical EPSPs: Origin and computational role of noise in cortical circuits," in Advances in Cognitive Neurodynamics (III). Amsterdam, The Netherlands: Springer-Verlag, 2013, pp. 161-167.
13. T. Pfeil, T. C. Potjans, S. Schrader, W. Potjans, J. Schemmel, M. Diesmann, and K. Meier, "Is a 4-bit synaptic weight resolution enough?-constraints on enabling spike-timing dependent plasticity in neuromorphic hardware," Front. Neurosci., vol. 6, 2012, DOI: 10.3389/fnins.2012.00090.
14. N. Yasumatsu, M. Matsuzaki, T. Miyazaki, J. Noguchi, and H. Kasai, "Principles of long-term dynamics of dendritic spines," J. Neurosci., vol. 28, pp. 13592-13608, 2008.
15. Y. Loewenstein, A. Kuras, and S. Rumpel, "Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo," J. Neurosci., vol. 31, pp. 9481-9488, 2011.
16. X. J. Wang, "Probabilistic decision making by slow reverberation in cortical circuits," Neuron, vol. 36, pp. 955-968, 2002.
17. A. Destexhe, M. Rudolph, and D. Paré, "The high-conductance state of neocortical neurons in vivo," Nature Rev. Neurosci., vol. 4, pp. 739-751, 2003.
18. R. Jolivet, R. Kobayashi, A. Rauch, R. Naud, S. Shinomoto, and W. Gerstner, "A benchmark test for a quantitative assessment of simple neuron models," J. Neurosci. Methods, vol. 169, pp. 417-424, 2008.
19. J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," Proc. Nat. Acad. Sci. USA, vol. 79, pp. 2554-2558, 1982.
20. J. J. Hopfield, "Neurons with graded response have collective computational properties like those of two-state neurons," Proc. Nat. Acad. Sci. USA, vol. 81, pp. 3088-3092, 1984.
21. K. Nakazawa, M. C. Quirk, R. A. Chitwood, M. Watanabe, M. F. Yeckel, L. D. Sun, A. Kato, C. A. Carr, D. Johnston, M. A. Wilson, and S. Tonegawa, "Requirement for Hippocampal CA3 NMDA receptors in associative memory recall," Science, vol. 297, pp. 211-218, 2002.
22. D. J. Amit, H. Gutfreund, andH. Sompolinsky, "Spin-glass models of neural networks," Phys. Rev. A, vol. 32, pp. 1007-1018, 1985.
23. B. Derrida, E. Gardner, and A. Zippelius, "An exactly solvable asymmetric neural network model," Europhys. Lett., vol. 4, pp. 167-173, 1987.
24. M. V. Tsodyks and M. V. Feigel'man, "The enhanced storage capacity in neural networks with low activity level," Europhys. Lett., vol. 6, pp. 101-105, 1988.
25. M. Shiino and T. Fukai, "Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks & enhancement of the storage capacity," Phys. Rev., vol. E48, pp. 867-897, 1993.
26. A. C. C. Coolen and D. Sherrington, "Dynamics of fully connected attractor neural networks near saturation," Phys. Rev. Lett., vol. 71, pp. 3886-3889, 1993.
27. M. Okada, "A hierarchy of macrodynamical equations for associative memory," Neural Netw., vol. 8, pp. 833-838, 1995.
28. W. Gerstner and J. L. Hemmen, "Associative memory in a networks of spiking neurons," Network, vol. 3, pp. 139-164, 1992.
29. A. Lansner and E. Fransén, "Modeling Hebbian cell assemblies comprised of cortical neurons," Network, vol. 3, pp. 105-119, 1992.
30. A. Treves, "Mean-field analysis of neuronal spike dynamics," Network, vol. 4, pp. 259-284, 1993.
31. W. Maass and T. Natschläger, "Networks of spiking neurons can emulate arbitrary Hopfield nets in temporal coding," Network, vol. 8, pp. 355-371, 1997.
32. D. J. Amit and N. Brunel, "Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex," Cereb. Cortex, vol. 7, pp. 237-252, 1997.
33. F. T. Sommer and T. Wennekers, "Associative memory in networks of spiking neurons," Neural Netw., vol. 14, pp. 825-834, 2001.
34. M. Curti, G. Mongillo, G. L. Camera, and D. J. Amit, "Mean field and capacity in realistic networks of spiking neurons storing sparsely coded random memories," Neural Comput., vol. 16, pp. 2597-2637, 2004.
35. P. Latham and S. Nirenberg, "Computing and stability in cortical networks," Neural Comput., vol. 16, pp. 1385-1412, 2004.
36. Y. Aviel, D. Horn, and M. Abeles, "Memory capacity of balanced networks," Neural Comput., vol. 17, pp. 691-713, 2005.
37. Y. Roudi and P. Latham, "A balanced memory network," PLoS Comput. Biol., vol. 3, 2007, DOI: 10.1371/journal.pcbi.0030141.
38. N. Hiratani, J. N. Teramae, and T. Fukai, "Associative memory model with long-tail-distributed Hebbian synaptic connections," Front. Comput. Neurosci., vol. 6, 2013, DOI: 10.3389/fncom.2012.00102.
39. A. A. Koulakov, T. Hromadka, and A. M. Zador, "Correlated connectivity and the distribution of firing rates in the neocortex," J. Neurosci., vol. 29, pp. 3685-3694, 2009.
40. T. Hromádka, M. R. DeWeese, and A. M. Zador, "Sparse representation of sounds in the unanesthetized auditory cortex," PLoS Biol., vol. 6, 2008, e16.
41. K. Mizuseki and G. Buzsáki, "Preconfigured, skewed distribution of firing rates in the hippocampus and entorhinal cortex," Cell Rep., vol. 4.5, pp. 1010-1021, 2013.
42. T. Knöpfel, "Genetically encoded optical indicators for the analysis of neuronal circuits," Nat. Rev. Neurosci., vol. 13, pp. 687-700, 2012.
43. M. C. W. van Rossum, G. Q. Bi, and G. G. Turrigiano, "Stable Hebbian learning from spike timing-dependent plasticity," J. Neurosci., vol. 20, pp. 8812-8821, 2000.
44. M. Gilson and T. Fukai, "Stability versus neuronal specialization for STDP: Long-tail weight distributions solve the dilemma," PLoS ONE, vol. 6, 2011, e25339.
45. M. Gilson, T. Fukai, and A. N. Burkitt, "Spectral analysis of input spike trains by spike-timing-dependent plasticity," PLoS Comput. Biol., 2012, e1002584.
46. A. Nathan and V. C. Barbosa, "Network algorithmics and the emergence of the cortical synaptic-weight distribution," Phys. Rev. E, vol. 81, no. 2, 2010, 021916.
47. A. Roxin, N. Brunel, D. Hansel, G. Mongillo, and C. van Vreeswijk, "On the distribution of firing rates in networks of cortical neurons," J. Neurosci., vol. 31, pp. 16217-16226, 2011.
48. N. Brunel, "Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons," J. Comput. Neurosci., vol. 8, pp. 183-208, 2000.
49. C. C. Petersen, R. C. Malenka, R. A. Nicoll, and J. J. Hopfield, "All-or-none potentiation at CA3-CA1 synapses," Proc. Nat. Acad. Sci. USA, vol. 95, pp. 4732-4737, 1998.
50. A. Soltani and X. J. Wang, "Synaptic computation underlying probabilistic inference," Nature Neurosci., vol. 13, pp. 112-119, 2010.
51. M. Rudolph and A. Destexhe, "Correlation detection and resonance in neural systems with distributed noise sources," Phys. Rev. Lett., vol. 86, pp. 3662-3665, 2001.
52. J. C. Magee and E. P. Cook, "Somatic EPSP amplitude is independent of synapse location in hippocampal pyramidal neurons," Nature Neurosci., vol. 3, pp. 895-903, 2000.
53. V. Menon, T. F. Musial, A. Liu, Y. Katz, W. L. Kath, N. Spruston, and D. A. Nicholson, "Balanced synaptic impact via distance-dependent synapse distribution and complementary expression of AMPARs and NMDARs in hippocampal dendrites," Neuron, vol. 80, pp. 1451-1463, 2013.
54. M. Morita, "Memory and learning of sequential patterns by nonmonotone neural networks," Neural Netw., vol. 9, pp. 1477-1489, 1996.
55. S. Ishii and M. Sato, "Associative memory based on parametrically coupled chaotic elements," Physica D, Nonlinear Phenomena, vol. 121, pp. 344-366, 1998.
56. P. Berkes, G. Orbán, M. Lengyel, and J. Fiser, "Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment," Science, vosl. 331, pp. 83-87, 2011.