Memristors empower spiking neurons with stochasticity

IEEE Journal on Emerging and Selected Topics in Circuits and Systems

Volumn 5, Issue 2, 2015, Pages 242-253

  Al Shedivat, Maruan ^a   Naous, Rawan ^a   Cauwenberghs, Gert ^b   Salama, Khaled Nabil ^a  

^a KING ABDULLAH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Saudi Arabia)

^b UNIVERSITY OF CALIFORNIA SAN DIEGO   (United States)

Author keywords

Neuromorphic systems; probabilistic inference; probabilistic learning; spiking neurons; stochastic computing; stochastic memristors; winner take all

Indexed keywords

EID: 85027951598     PISSN: 21563357     EISSN: None     Source Type: Journal    
DOI: 10.1109/JETCAS.2015.2435512     Document Type: Article

References (75)

1. W. Gerstner and W. M. Kistler, Spiking Neuron Models: Single Neurons, Populations, Plasticity. Cambridge, U.K.: Cambridge Univ. Press, 2002.
2. A. L. Hodgkin and A. F. Huxley, "A quantitative description of membrane current and its application to conduction and excitation in nerve, " J. Physiol., vol. 117, no. 4, p. 500, 1952.
3. E. M. Izhikevich, Dynamical Systems in Neuroscience. Cambridge, MA: MIT Press, 2007.
4. A. A. Faisal, L. P. Selen, and D. M. Wolpert, "Noise in the nervous system, " Nature Rev. Neurosci., vol. 9, no. 4, pp. 292-303, 2008.
5. M. McDonnell and L. Ward, "The benefits of noise in neural systems: Bridging theory and experiment, " Nature Rev. Neurosci., vol. 12, no. 7, pp. 415-426, 2011.
6. W. Maass, "Noise as a resource for computation and learning in networks of spiking neurons, " Proc. IEEE, vol. 102, no. 5, pp. 860-880, May 2014.
7. R. Jolivet, A. Rauch, H.-R. Lüscher, and W. Gerstner, "Predicting spike timing of neocortical pyramidal neurons by simple threshold models, " J. Comput. Neurosci., vol. 21, no. 1, pp. 35-49, 2006.
8. B. Nessler, M. Pfeiffer, L. Buesing, and W. Maass, "Bayesian computation emerges in generic cortical microcircuits through spike-timing-dependent plasticity, " PLoS Comput. Biol., vol. 9, no. 4, p. e1003037, 2013.
9. L. Buesing, J. Bill, B. Nessler, and W. Maass, "Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons, " PLoS Comput. Biol., vol. 7, no. 11, p. e1002211, 2011.
10. D. Pecevski, L. Buesing, and W. Maass, "Probabilistic inference in general graphical models through sampling in stochastic networks of spiking neurons, " PLoS Comput. Biol., vol. 7, no. 12, p. e1002294, 2011.
11. D. Kappel, B. Nessler, and W. Maass, "STDP installs in winner-take-all circuits an online approximation to hidden Markov model learning, " PLoS Comput. Biol., vol. 10, no. 3, p. e1003511, 2014.
12. B. U. Pedroni, S. Das, E. Neftci, K. Kreutz-Delgado, and G. Cauwen-berghs, "Neuromorphic adaptations of restricted Boltzmann machines and deep belief networks, " in Proc. Int.l Joint Conf. IEEE Neural Netw., 2013, pp. 1-6.
13. E. Neftci, S. Das, B. Pedroni, K. Kreutz-Delgado, and G. Cauwen-berghs, "Event-driven contrastive divergence for spiking neuromor-phic systems, " Front. Neurosci., vol. 7, 2014.
14. P. O'Connor, D. Neil, S.-C. Liu, T. Delbruck, and M. Pfeiffer, "Realtime classification and sensor fusion with a spiking deep belief network, " Front. Neurosci., vol. 7, 2013.
15. C. Mead and M. Ismail, Analog VLSI Implementation of Neural Systems. New York: Springer, 1989.
16. G. Indiveri et al., "Neuromorphic silicon neuron circuits, " Front. Neu-rosci., vol. 5, 2011.
17. H. E. Plesser and W. Gerstner, "Noise in integrate-and-fire neurons: From stochastic input to escape rates, " Neural Comput., vol. 12, no. 2, pp. 367-384, 2000.
18. A. S. Cassidy et al., "Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores, " in Proc. IEEE Int. Joint Conf. Neural Netw., 2013, pp. 1-10.
19. M. A. Petrovici, J. Bill, I. Bytschok, J. Schemmel, and K. Meier, Stochastic inference with deterministic spiking neurons arXiv preprint arXiv:1311.3211, 2013.
20. T. Hamilton, S. Afshar, A. van Schaik, and J. Tapson, "Stochastic electronics: A neuro-inspired design paradigm for integrated circuits, " Proc. IEEE, vol. 102, no. 5, pp. 843-859, May 2014.
21. J. J. Yang, D. B. Strukov, and D. R. Stewart, "Memristive devices for computing, " Nature Nanotechnol., vol. 8, no. 1, pp. 13-24, 2013.
22. F. Alibart, E. Zamanidoost, and D. B. Strukov, "Pattern classification by memristive crossbar circuits using ex situ and in situ training, " Nature Commun., vol. 4, 2013.
23. B. Linares-Barranco and T. Serrano-Gotarredona, "Memristance can explain spike-time-dependent-plasticity in neural synapses, " Nature Precedings, pp. 1-4, 2009.
24. S. H. Jo et al., "Nanoscale memristor device as synapse in neuromor-phic systems, " Nano Lett., vol. 4, pp. 1297-1301, 2010.
25. C. Zamarre no-Ramos et al., "On spike-timing-dependent-plasticity, memristive devices, building a self-learning visual cortex, " Front. Neu-rosci., vol. 5, 2011.
26. T. Serrano-Gotarredona, T. Masquelier, and T. Prodromakis, "STDP and STDP variations with memristors for spiking neuromorphic learning systems, " Front. Neurosci., vol. 5, pp. 1-15, 2013.
27. M. Rahimi Azghadi, N. Iannella, S. F. Al-Sarawi, G. Indiveri, and D. Abbott, "Spike-based synaptic plasticity in silicon: Design, implementation, application, challenges, " Proc. IEEE, vol. 102, no. 5, pp. 717-737, May 2014.
28. G. Indiveri, B. Linares-Barranco, R. Legenstein, G. Deligeorgis, and T. Prodromakis, "Integration of nanoscale memristor synapses in neu-romorphic computing architectures, " Nanotechnology, vol. 24, no. 38, p. 384010, 2013.
29. T. Serrano-Gotarredona, T. Prodromakis, and B. Linares-Barranco, "A proposal for hybrid memristor-CMOS spiking neuromorphic learning systems, " Circuits Syst. Mag., vol. 13, pp. 74-88, 2013.
30. S. Gaba, P. Sheridan, J. Zhou, S. Choi, and W. Lu, "Stochastic memristive devices for computing and neuromorphic applications, " Nanoscale, vol. 5, no. 13, p. 5872, 2013.
31. Q. Li, A. Khiat, I. Salaoru, H. Xu, and T. Prodromakis, "Stochastic switching of TiO2-based memristive devices with identical initial memory states, " Nanoscale Res. Lett., vol. 9, no. 1, pp. 1-5, 2014.
32. Y. Yang et al., "Observation of conducting filament growth in nanoscale resistive memories, " Nature Commun., vol. 3, p. 732, 2012.
33. M. Suri et al., "Bio-inspired stochastic computing using binary CBRAM synapses, " IEEE Trans. Electron Devices, vol. 60, no. 7, pp. 2402-2409, Jul. 2013.
34. S. Yu et al., "Stochastic learning in oxide binary synaptic device for neuromorphic computing, " Front. Neurosci., vol. 7, 2013.
35. J. Bill and R. Legenstein, "A compound memristive synapse model for statistical learning through STDP in spiking neural networks, " Front. Neurosci., vol. 8, 2014.
36. S. Shin, K. Kim, and S.-M. S. Kang, "Memristor macromodel and its application to neuronal spike generation, " in IEEE Eur. Conf. Circuit Theory Design, 2013, pp. 1-4.
37. M. D. Pickett, G. Medeiros-Ribeiro, and R. S. Williams, "A scalable neuristor built with Mott memristors, " Nature Mater., vol. 12, no. 2, pp. 114-117, 2013.
38. M. Hu, Y. Wang, Q. Qiu, Y. Chen, and H. Li, "The stochastic modeling of TiO2 memristor and its usage in neuromorphic system design, " in Proc. 9th Asia South Pacific Design Automat. Conf., Jan. 2014, pp. 831-836.
39. G. Palma, M. Suri, D. Querlioz, E. Vianello, and B. De Salvo, "Stochastic neuron design using conductive bridge RAM, " in Proc. IEEE/ACM Int. Symp. IEEE Nanoscale Archit., 2013, pp. 95-100.
40. Y. Yang and W. Lu, "Nanoscale resistive switching devices: Mechanisms and modeling, " Nanoscale, vol. 5, no. 21, pp. 10 076-10 092, 2013.
41. J. P. Strachan et al., "Direct identification of the conducting channels in a functioning memristive device, " Adv. Mater., vol. 22, no. 32, pp. 3573-3577, 2010.
42. M. A. Zidan et al., "A family of memristor-based reactance-less oscillators, " Int. J. Circuit Theory Appl., 2013.
43. M. D. Pickett et al., "Switching dynamics in titanium dioxide memris-tive devices, " J. Appl. Phys., vol. 106, no. 7, p. 074508, 2009.
44. Z. Biolek, D. Biolek, and V. Biolkova, "Spice model of memristor with nonlinear dopant drift, " Radioengineering, vol. 18, no. 2, 2009.
45. D. Biolek, M. Di Ventra, and Y. V. Pershin, Reliable spice simulations of memristors, memcapacitors and meminductors arXiv preprint arXiv:1307.2717, 2013.
46. T. Prodromakis, B. P. Peh, C. Papavassiliou, and C. Toumazou, "A versatile memristor model with nonlinear dopant kinetics, " IEEE Trans. Electron Devices, vol. 58, no. 9, pp. 3099-3105, Sep. 2011.
47. F. Corinto and A. Ascoli, "A boundary condition-based approach to the modeling of memristor nanostructures, " IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 59, no. 11, pp. 2713-2726, Nov. 2012.
48. A. Ascoli, F. Corinto, V. Senger, and R. Tetzlaff, "Memristor model comparison, " IEEE Circuits Syst. Mag., vol. 13, no. 2, 2013.
49. A. Radwan, M. A. Zidan, and K. Salama, "On the mathematical modeling of memristors, " in Proc. IEEE Int. Conf. Microelectron., 2010, pp. 284-287.
50. F. Garcia, M. Lopez-Vallejo, and P. Ituero, "Building memristor applications: From device model to circuit design, " IEEE Trans. Nan-otechnol., vol. 13, no. 6, pp. 1154-1162, Nov. 2014.
51. A. G. Radwan, M. A. Zidan, and K. Salama, "HP memristor mathematical model for periodic signals and DC, " in Proc. 53rd IEEE Int. Midwest Symp. Circuits Syst., 2010, pp. 861-864.
52. O. Kavehei et al., "Fabrication and modeling of ag/tio2/ito mem-ristor, " in Proc. 54th IEEE Int. Midwest Sympo. Circuits Syst., 2011, pp. 1-4.
53. S. Kim, S. Choi, and W. Lu, "Comprehensive physical model of dynamic resistive switching in an oxide memristor, " ACS Nano, vol. 8, 2014.
54. Q. Lv et al., "Conducting nanofilaments formed by oxygen vacancy migration in Ti/TiO2/TiN/MgO memristive device, " J. Appl. Phys., vol. 110, no. 10, p. 104511, 2011.
55. S. Yu, X. Guan, and H.-S. Wong, "On the stochastic nature of resistive switching in metal oxide RRAM: Physical modeling, Monte Carlo simulation, experimental characterization, " in Proc. IEEE Int. Electron Devices Meet., 2011, pp. 17-3.
56. D. Ielmini, "Modeling the universal set/reset characteristics of bipolar RRAM by field-and temperature-driven filament growth, " IEEE Trans. Electron Devices, vol. 58, no. 12, pp. 4309-4317, Dec. 2011.
57. D. R. Cox and V. Isham, Point Processes. Boca Raton, FL: CRC Press, 1980, vol. 12.
58. S. H. Jo, K.-H. Kim, and W. Lu, "Programmable resistance switching in nanoscale two-terminal devices, " Nano Lett., vol. 9, no. 1, 2008.
59. R. J. Vogelstein, U. Mallik, J. T. Vogelstein, and G. Cauwenberghs, "Dynamically reconfigurable silicon array of spiking neurons with conductance-based synapses, " IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 253-265, Jan. 2007.
60. R. J. Vogelstein et al., "Spike timing-dependent plasticity in the address domain, " Adv. Neural Inf. Process. Syst., 2002.
61. M. Al-Shedivat, R. Naous, E. Neftci, G. Cauwenberghs, and K. N. Salama, "Inherently stochastic spiking neurons for probabilistic neural computation, " in Proc. 7th Int. IEEE/EMBS Conf. IEEE Neural Eng., 2013.
62. D. H. Goldberg, G. Cauwenberghs, and A. G. Andreou, "Probabilistic synaptic weighting in a reconfigurable network of VLSI integrate-and-fire neurons, " Neural Netw., vol. 14, no. 6, pp. 781-793, 2001.
63. A. Renart, N. Brunel, and X.-J. Wang, "Mean-field theory of irregularly spiking neuronal populations and working memory in recurrent cortical networks, " Comput. Neurosci., Comprehensive Approach, pp. 431-490, 2004.
64. W. Gerstner, "Spike-response model, " Scholarpedia, vol. 3, no. 12, 2008.
65. M. Stimberg, D. F. Goodman, V. Benichoux, and R. Brette, "Equation-oriented specification of neural models for simulations, " Front. Neuroinformat., vol. 8, 2014.
66. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the em algorithm, " J. R. Stat. Soc. Ser. B (Methodol.), pp. 1-38, 1977.
67. D. Lowd and P. Domingos, "Naive Bayes models for probability estimation, " in Proc. 22nd ACM Int. Conf. Mach. Learn., 2005, pp. 529-536.
68. T. G. Dietterich, "Ensemble methods in machine learning, " in Multiple Classifier Systems. New York: Springer, 2000, pp. 1-15.
69. D. Querlioz, O. Bichler, and C. Gamrat, "Simulation of a memristor-based spiking neural network immune to device variations, " in Proc. IEEE Int. Joint Conf. Neural Netw., 2011, pp. 1775-1781.
70. S. Zhong, "Efficient online spherical k-means clustering, " in Proc. IEEE Int. Joint Conf. Neural Netw., 2005, vol. 5, pp. 3180-3185.
71. G. Cauwenberghs, "An analog VLSI recurrent neural network learning a continuous-time trajectory, " IEEE Trans. Neural Netw., vol. 7, no. 2, pp. 346-361, Mar. 1996.
72. M. B. Parker and R. Chu, "A VLSI-efficient technique for generating multiple uncorrelated noise sources and its application to stochastic neural networks, " IEEE Trans. Circuits Syst., vol. 38, no. 1, p. 109, Jan. 1991.
73. J. A. Montiel-Nelson, V. Navarro, J. Sosa, and T. Bautista, "Analysis and optimization of dynamically reconfigurable regenerative comparators for ultra-low power 6-bit TC-ADCs in 90 nm CMOS technologies, " Microelectronics, vol. 45, pp. 1247-1253, 2014.
74. E. Chicca and S. Fusi, "Stochastic synaptic plasticity in deterministic aVLSI networks of spiking neurons, " Proc. World Congr. Neuroin-format., 2001.
75. R. Moreno-Bote, "Poisson-like spiking in circuits with probabilistic synapses, " PLoS Comput. Biol., vol. 10, no. 7, p. e1003522, 2014.