Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training

IEEE Transactions on Neural Networks and Learning Systems

Volumn 26, Issue 10, 2015, Pages 2408-2421

  Soudry, Daniel ^a   Di Castro, Dotan ^b   Gal, Asaf ^c   Kolodny, Avinoam ^c   Kvatinsky, Shahar ^d  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b YAHOO RESEARCH   (United States)

^c TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^d Stanford University   (United States)

Author keywords

Backpropagation; hardware; memristive systems; memristor; multilayer neural networks (MNNs); stochastic gradient descent; synapse

Indexed keywords

BACKPROPAGATION; BACKPROPAGATION ALGORITHMS; COMPUTER HARDWARE; HARDWARE; MEMRISTORS; MULTILAYERS; PASSIVE FILTERS; STOCHASTIC SYSTEMS;

CONTINUOUS UPDATING; GRADIENT DESCENT TRAINING; MEMRISTIVE SYSTEMS; MEMRISTOR; MULTIPLY AND ACCUMULATE OPERATIONS; NUMBER OF COMPONENTS; STOCHASTIC GRADIENT DESCENT; SYNAPSE;

MULTILAYER NEURAL NETWORKS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; DEVICES; HUMAN; LEARNING; NERVE CELL; ONLINE SYSTEM; PHYSIOLOGY; SIGNAL PROCESSING; SYNAPSE;

ALGORITHMS; COMPUTER SIMULATION; HUMANS; LEARNING; NEURAL NETWORKS (COMPUTER); NEURONS; ONLINE SYSTEMS; SIGNAL PROCESSING, COMPUTER-ASSISTED; SYNAPSES;

EID: 84943802795     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2014.2383395     Document Type: Article

References (74)

1. Q. V. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng, "Building high-level features using large scale unsupervised learning," in Proc. ICML, Edinburgh, Scotland, Jun. 2012, pp. 81-88.
2. D. Ciresan, U. Meier, and J. Schmidhuber, "Multi-column deep neural networks for image classification," in Proc. CVPR, Providence, RI, USA, Jun. 2012, pp. 3642-3649.
3. J. Dean et al., "Large scale distributed deep networks," in Proc. NIPS, Lake Tahoe, NV, USA, Dec. 2012, pp. 1-9.
4. R. D. Hof, "Deep learning," MIT Technol. Rev., Apr. 2013.
5. D. Hernandez, "Now you can build Google's $1M artificial brain on the cheap," Wired, Jun. 2013, pp. 9-48.
6. Y. A. Le Cun, L. Bottou, G. B. Orr, and K.-R. Müller, "Efficient backprop," in Neural Networks: Tricks of the Trade, G. Montavon, G. B. Orr, and K.-R. Müller, Eds., 2nd ed. Heidelberg, Germany: Springer-Verlag, 2012.
7. J. Misra and I. Saha, "Artificial neural networks in hardware: A survey of two decades of progress," Neurocomputing, vol. 74, nos. 1-3, pp. 239-255, Dec. 2010.
8. A. R. Omondi, "Neurocomputers: A dead end?" Int. J. Neural Syst., vol. 10, no. 6, pp. 475-481, 2000.
9. M. Versace and B. Chandler, "The brain of a new machine," IEEE Spectr., vol. 47, no. 12, pp. 30-37, Dec. 2010.
10. M. M. Waldrop, "Neuroelectronics: Smart connections," Nature, vol. 503, no. 7474, pp. 22-24, Nov. 2013.
11. L. O. Chua, "Memristor-the missing circuit element," IEEE Trans. Circuit Theory, vol. 18, no. 5, pp. 507-519, Sep. 1971.
12. D. B. Strukov, G. S. Snider, D. R. Stewart, and R. S. Williams, "The missing memristor found," Nature, vol. 453, no. 7191, pp. 80-83, Mar. 2008.
13. F. Corinto and A. Ascoli, "A boundary condition-based approach to the modeling of memristor nanostructures," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 59, no. 11, pp. 2713-2726, Nov. 2012.
14. S. Kvatinsky, E. G. Friedman, A. Kolodny, and U. C. Weiser, "TEAM: Threshold adaptive memristor model," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 60, no. 1, pp. 211-221, Jan. 2013.
15. S. Kvatinsky, Y. H. Nacson, Y. Etsion, E. G. Friedman, A. Kolodny, and U. C. Weiser, "Memristor-based multithreading," Comput. Archit. Lett., vol. 13, no. 1, pp. 41-44, Jul. 2014.
16. Z. Guo, J. Wang, and Z. Yan, "Passivity and passification of memristorbased recurrent neural networks with time-varying delays," IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 11, pp. 2099-2109, Nov. 2014.
17. L. Wang, Y. Shen, Q. Yin, and G. Zhang, "Adaptive synchronization of memristor-based neural networks with time-varying delays," IEEE Trans. Neural Netw. Learn. Syst., to be published.
18. G. Zhang and Y. Shen, "Exponential synchronization of delayed memristor-based chaotic neural networks via periodically intermittent control," Neural Netw., vol. 55, pp. 1-10, Jul. 2014.
19. D. Querlioz, O. Bichler, and C. Gamrat, "Simulation of a memristorbased spiking neural network immune to device variations," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), San Jose, CA, USA, Jul./Aug. 2011, pp. 1775-1781.
20. C. Zamarreño-Ramos, L. A. Camuñas-Mesa, J. A. Pérez-Carrasco, T. Masquelier, T. Serrano-Gotarredona, and B. Linares-Barranco, "On spike-timing-dependent-plasticity, memristive devices, and building a self-learning visual cortex," Frontiers Neurosci., vol. 5, p. 26, Jan. 2011.
21. O. Kavehei et al., "Memristor-based synaptic networks and logical operations using in-situ computing," in Proc. 7th Int. Conf. Intell. Sensors, Sensor Netw. Inf. Process., Adelaide, SA, Australia, Dec. 2011, pp. 137-142.
22. A. Nere, U. Olcese, D. Balduzzi, and G. Tononi, "A neuromorphic architecture for object recognition and motion anticipation using burst-STDP," PloS One, vol. 7, no. 5, p. e36958, Jan. 2012.
23. Y. Kim, Y. Zhang, and P. Li, "A digital neuromorphic VLSI architecture with memristor crossbar synaptic array for machine learning," in Proc. IEEE Int. SOC Conf. (SOCC), Niagara Falls, NY, USA, Sep. 2012, pp. 328-333.
24. W. Chan and J. Lohn, "Spike timing dependent plasticity with memristive synapse in neuromorphic systems," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Brisbane, QLD, Australia, Jun. 2012, pp. 1-6.
25. T. Serrano-Gotarredona, T. Masquelier, T. Prodromakis, G. Indiveri, and B. Linares-Barranco, "STDP and STDP variations with memristors for spiking neuromorphic learning systems," Frontiers Neurosci., vol. 7, p. 2, Jan. 2013.
26. D. Querlioz, O. Bichler, P. Dollfus, and C. Gamrat, "Immunity to device variations in a spiking neural network with memristive nanodevices," IEEE Trans. Nanotechnol., vol. 12, no. 3, pp. 288-295, May 2013.
27. R. Legenstein, C. Naeger, and W. Maass, "What can a neuron learn with spike-timing-dependent plasticity?" Neural Comput., vol. 17, no. 11, pp. 2337-2382, Mar. 2005.
28. D. Chabi, W. Zhao, D. Querlioz, and J.-O. Klein, "Robust neural logic block (NLB) based on memristor crossbar array," in Proc. IEEE/ACM Int. Symp. IEEE Nanosc. Archit. (NANOARCH), San Diego, CA, USA, Jun. 2011, pp. 137-143.
29. H. Manem, J. Rajendran, and G. S. Rose, "Stochastic gradient descent inspired training technique for a CMOS/nano memristive trainable threshold gate array," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 59, no. 5, pp. 1051-1060, May 2012.
30. M. Soltiz, D. Kudithipudi, C. Merkel, G. S. Rose, and R. E. Pino, "Memristor-based neural logic blocks for nonlinearly separable functions," IEEE Trans. Comput., vol. 62, no. 8, pp. 1597-1606, Aug. 2013.
31. F. Alibart, E. Zamanidoost, and D. B. Strukov, "Pattern classification by memristive crossbar circuits usingex situ and in situ training," Nature Commun., vol. 4, p. 2072, May 2013.
32. F. Rosenblatt, "The perceptron: A probabilistic model for information storage and organization in the brain," Psychol. Rev., vol. 65, no. 6, pp. 386-408, Nov. 1958.
33. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Control, Signals, Syst., vol. 2, no. 4, pp. 303-314, 1989.
34. L. Bottou and O. Bousquet, "The tradeoffs of large scale learning," in Optimization for Machine Learning, S. Sra, S. Nowozin, and S. J. Wright, Eds. Cambridge, MA, USA: MIT Press, 2011, p. 351.
35. D. Cireşan and U. Meier, "Deep, big, simple neural nets for handwritten digit recognition," Neural Comput., vol. 22, no. 12, pp. 3207-3220, Nov. 2010.
36. S. P. Adhikari, C. Yang, H. Kim, and L. O. Chua, "Memristor bridge synapse-based neural network and its learning," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 9, pp. 1426-1435, Sep. 2012.
37. R. Hasan and T. M. Taha, "Enabling back propagation training of memristor crossbar neuromorphic processors," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Beijing, China, Jul. 2014, pp. 21-28.
38. Z. Vasilkoski et al., "Review of stability properties of neural plasticity rules for implementation on memristive neuromorphic hardware," in Proc. Int. Joint Conf. Neural Netw., San Jose, CA, USA, Jul./Aug. 2011, pp. 2563-2569.
39. The Supplementary Material. [Online]. Available: http://ieeexplore.ieee.org.
40. L. O. Chua and S. M. Kang, "Memristive devices and systems," Proc. IEEE, vol. 64, no. 2, pp. 209-223, Feb. 1976.
41. M. D. Pickett et al., "Switching dynamics in titanium dioxide memristive devices," J. Appl. Phys., vol. 106, no. 7, p. 074508, 2009.
42. J. Strachan et al., "State dynamics and modeling of tantalum oxide memristors," IEEE Trans. Electron Devices, vol. 60, no. 7, pp. 2194-2202, Jul. 2013.
43. B. Widrow and M. E. Hoff, "Adaptive switching circuits," Stanford Electron. Labs, Stanford Univ., Stanford, CA, USA, Tech. Rep., 1960.
44. B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ, USA: Prentice-Hall, 1985.
45. E. Oja, "Simplified neuron model as a principal component analyzer," J. Math. Biol., vol. 15, no. 3, pp. 267-273, Nov. 1982.
46. C. M. Bishop, Pattern Recognition and Machine Learning. Singapore: Springer-Verlag, 2006.
47. S. Shalev-Shwartz, Y. Singer, N. Srebro, and A. Cotter, "Pegasos: Primal estimated sub-gradient solver for SVM," Math. Program., vol. 127, no. 1, pp. 3-30, Oct. 2010.
48. S. Kvatinsky, N. Wald, E. Satat, E. G. Friedman, A. Kolodny, and U. C. Weiser, "Memristor-based material implication (IMPLY) logic: Design principles and methodologies," IEEE Trans. Very Large Scale Integr. (VLSI) Syst., vol. 22, no. 10, pp. 2054-2066, Oct. 2014.
49. C. M. Bishop, Neural Networks for Pattern Recognition. Oxford, U.K.: Oxford Univ. Press, 1995.
50. R. Sarpeshkar, "Analog versus digital: Extrapolating from electronics to neurobiology," Neural Comput., vol. 10, no. 7, pp. 1601-1638, Oct. 1998.
51. The Mosis Service. [Online]. Available: http://www.mosis.com, accessed Nov. 21, 2012.
52. G. Huang, D. C. Sekar, A. Naeemi, K. Shakeri, and J. D. Meindl, "Compact physical models for power supply noise and chip/package co-design of gigascale integration," in Proc. IEEE 57th Electron. Compon. Technol. Conf. (ECTC), Sparks, NV, USA, May/Jun. 2007, pp. 1659-1666.
53. 2 thin-film and spintronic memristors," in Proc. 16th Asia South Pacific Design Autom. Conf., Yokohama, Japan, Jan. 2011, pp. 25-30.
54. Sim Electronics. [Online]. Available: http://www.mathworks.com/products/simelectronics/, accessed Nov. 21, 2012.
55. T. Chang, S.-H. Jo, K.-H. Kim, P. Sheridan, S. Gaba, and W. Lu, "Synaptic behaviors and modeling of a metal oxide memristive device," Appl. Phys. A, vol. 102, no. 4, pp. 857-863, Feb. 2011.
56. K. Bache and M. Lichman. (2013). UCI Machine Learning Repository. [Online]. Available: http://archive.ics.uci.edu/ml
57. P. Y. Simard, D. Steinkraus, and J. C. Platt, "Best practices for convolutional neural networks applied to visual document analysis," in Proc. 7th Int. Conf. Document Anal. Recognit., vol. 1. Edinburgh, Scotland, Aug. 2003, pp. 958-963.
58. D. B. Strukov and K. K. Likharev, "Reconfigurable nano-crossbar architectures," in Nanoelectronics and Information Technology, R. Waser, Ed. New York, NY, USA: Wiley, 2012, pp. 543-562.
59. G. Cauwenberghs, C. F. Neugebauer, and A. Yariv, "Analysis and verification of an analog VLSI incremental outer-product learning system," IEEE Trans. Neural Netw., vol. 3, no. 3, pp. 488-497, May 1992.
60. H. C. Card, C. R. Schneider, and W. R. Moore, "Hebbian plasticity in MOS synapses," IEE Proc. F, Radar Signal Process., vol. 138, no. 1, pp. 13-16, Feb. 1991.
61. C. Schneider and H. Card, "Analogue CMOS Hebbian synapses," Electron. Lett., vol. 27, no. 9, pp. 785-786, Apr. 1991.
62. H. C. Card, C. R. Schneider, and R. S. Schneider, "Learning capacitive weights in analog CMOS neural networks," J. VLSI Signal Process. Syst. Signal, Image Video Technol., vol. 8, no. 3, pp. 209-225, Oct. 1994.
63. M. Valle, D. D. Caviglia, and G. M. Bisio, "An experimental analog VLSI neural network with on-chip back-propagation learning," Analog Integr. Circuits Signal Process., vol. 9, no. 3, pp. 231-245, Apr. 1996.
64. T. Morie and Y. Amemiya, "An all-analog expandable neural network LSI with on-chip backpropagation learning," IEEE J. Solid-State Circuits, vol. 29, no. 9, pp. 1086-1093, Sep. 1994.
65. C. Lu, B.-X. Shi, and L. Chen, "An on-chip BP learning neural network with ideal neuron characteristics and learning rate adaptation," Analog Integr. Circuits Signal Process., vol. 31, no. 1, pp. 55-62, Apr. 2002.
66. T. Shima, T. Kimura, Y. Kamatani, T. Itakura, Y. Fujita, and T. Iida, "Neuro chips with on-chip back-propagation and/or Hebbian learning," IEEE J. Solid-State Circuits, vol. 27, no. 12, pp. 1868-1876, Dec. 1992.
67. C. S. Lindsey and T. Lindblad, "Survey of neural network hardware," Proc. SPIE, vol. 2492, pp. 1194-1205, Apr. 1995.
68. G. Cauwenberghs, "A learning analog neural network chip with continuous-time recurrent dynamics," in Proc. NIPS, Golden, CO, USA, Nov. 1994, pp. 858-865.
69. H. Eguchi, T. Furuta, H. Horiguchi, S. Oteki, and T. Kitaguchi, "Neural network LSI chip with on-chip learning," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Seattle, WA, USA, Jul. 1991, pp. 453-456.
70. C. Schneider and H. Card, "CMOS implementation of analog Hebbian synaptic learning circuits," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), vol. i. Seattle, WA, USA, vol. 1, Jul. 1991, pp. 437-442.
71. F. Alibart, L. Gao, B. D. Hoskins, and D. B. Strukov, "High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm," Nanotechnology, vol. 23, no. 7, p. 075201, Feb. 2012.
72. D. Soudry, I. Hubara, and R. Meir, "Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights," in Proc. NIPS, Montreal, QC, Canada, Dec. 2014, pp. 963-971.
73. M. Milev and M. Hristov, "Analog implementation of ANN with inherent quadratic nonlinearity of the synapses," IEEE Trans. Neural Netw., vol. 14, no. 5, pp. 1187-1200, Sep. 2003.
74. A. C. Torrezan, J. P. Strachan, G. Medeiros-Ribeiro, and R. S. Williams, "Sub-nanosecond switching of a tantalum oxide memristor," Nanotechnology, vol. 22, no. 48, p. 485203, Dec. 2011.