Acceleration of deep neural network training with resistive cross-point devices: Design considerations

Frontiers in Neuroscience

Volumn 10, Issue JUL, 2016, Pages

  Gokmen, Tayfun ^a   Vlasov, Yurii ^a,b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b University of Illinois at Urbana Champaign   (United States)

Author keywords

Artificial neural networks; Deep neural network training; Electronic devices; Machine learning; Materials engineering; Memristive devices; Nanotechnology; Synaptic device

Indexed keywords

ACCELERATION; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATIC SPEECH RECOGNITION; CONTROLLED STUDY; DEEP NEURAL NETWORK; GENERAL DEVICE; INTERMETHOD COMPARISON; MICROPROCESSOR; NATURAL LANGUAGE PROCESSING; RESISTIVE PROCESSING UNIT;

EID: 84983247214     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2016.00333     Document Type: Article

References (44)

1. Alaghi, A., and Hayes, J. (2013). Survey of stochastic computing. ACM Trans. Embed. Comput. Syst. 12, 1-19. doi: 10.1145/2465787.2465794
2. Arima, Y., Mashiko, K., Okada, K., Yamada, T., Maeda, A., Notani, H., et al. (1991). A 336-neuron, 28 K-synapse, self-learning neural network chip with branch-neuron-unit architecture. IEEE J. Solid State Circuits 26, 1637-1644.
3. Bi, G. Q., and Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464-10472.
4. Burr, G. W., Shelby, R. M., di Nolfo, C., Jang, J. W., Shenoy, R. S., Narayanan, P., et al. (2014). "Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element,"in 2014 IEEE International, Electron Devices Meeting (IEDM) (San Francisco, CA).
5. Burr, G. W., Narayanan, P., Shelby, R. M., Sidler, S., Boybat, I., di Nolfo, C., et al. (2015). "Large-scale neural networks implemented with nonvolatile memory as the synaptic weight element: comparative performance analysis (accuracy, speed, and power)," in IEDM (International Electron Devices Meeting) (Washington, DC).
6. Chen, G., Anders, M. A., Kaul, H., Satpathy, S. K., Mathew, S. K., Hsu, S. K., et al. (2014). "A 340 mV-to-0.9 V 20.2 Tb/s source-synchronous hybrid packet/circuit-switched 16 × 16 network-on-chip in 22 nm tri-gate CMOS," in 2014 IEEE International Soild-State Circuits Conference Digest of techincal Papers (ISSCC) (San Francisco, CA), 276-277.
7. Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., et al. (2014). "DaDianNao: a machine-learning supercomputer," in 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture (Cambridge, UK), 609-622.
8. Chua, L. O. (1971). Memristor - the missing circuit element. IEEE Trans. Circuit Theory 18, 507-519. doi: 10.1109/TCT.1971.1083337
9. Coates, A., Huval, B., Wang, T., Wu, D. J., Ng, A. Y., and Catanzaro, B. (2013). Deep Learning with COTS HPC Systems. Atlanta, GA: ICML.
10. Gaines, B. R. (1967). "Stochastic computing," in Proceedings of the AFIPS Spring Joint Computer Conference (Atlantic City, NJ), 149-156.
11. Gokhale, V., Jin, J., Dundar, A., Martini, B., and Culurciello, E. (2014). "A 240 g-ops/s mobile coprocessor for deep neural networks," in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (Columbus, OH), 696-701.
12. Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. (2015). Deep learning with limited numerical precision. arXiv:1502.02551 [cs. LG].
13. Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., et al. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag. 29, 82-97. doi: 10.1109/MSP.2012.2205597
14. Indiveri, G., Linares-Barranco, B., Legenstein, R., Deligeorgis, G., and Prodromakis, T. (2013). Integration of nanoscale memristor synapses in neuromorphic computing architectures. Nanotechnology 24:384010. doi: 10.1088/0957-4484/24/38/384010
15. Jackson, B. L., Rajendran, B., Corrado, G. S., Breitwisch, M., Burr, G. W., Cheek, R., et al. (2013). Nanoscale electronic synapses using phase change devices. ACM J. Emerg. Technol. Comput. Syst. 9, 1-20. doi: 10.1145/2463585.2463588
16. Jensen, K., Gaudet, V. C., and Levine, P. M. (2013). "Noise analysis and measurement of integrator-based sensor interface circuits for fluorescence detection in lab-on-a-chip applications," in International Conference on Noise and Fluctuation (Montpellier).
17. Jo, S. H., Chang, T., Ebong, I., Bhadviya, B. B., Mazumder, P., and Lu, W. (2010). Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 1297-1301. doi: 10.1021/nl904092h
18. Jonsson, B. E. (2011a). "An empirical approach to finding energy efficient ADC architectures," in 2011 International Workshop on ADC Modelling, Testing and Data Converter Analysis and Design and IEEE 2011 ADC Forum (Orvieto), 132-137.
19. Jonsson, B. E. (2011b). "Area Efficiency of ADC Architectures," in 2011 20th European Conference on Circuit Theory and Design (ECCTD) (Linkoping), 560-563.
20. Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). "Imagenet classification with deep convolutional neural networks," in Neural Information Processing Systems (Lake Tahoe, NV), 1097-1105.
21. Kuzum, D., Yu, S., and Wong, H. S. (2013). Synaptic electronics: materials, devices and applications. Nanotechnology 24:382001. doi: 10.1088/0957-4484/24/38/382001
22. Le, Q. V., Ranzato, M. A, Monga, R., Devin, M., Chen, K., Corrado, G. S., et al. (2012). "Building high-level features using large scale unsupervised learning," in International Conference on Machine Learning (Edinburg).
23. LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521, 436-444. doi: 10.1038/nature14539
24. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proce. IEEE 86, 2278-2324. doi: 10.1109/5.726791
25. Lehmann, C., Viredaz, M., and Blayo, F. A. (1993). generic systolic array building block for neural networks with on-chip learning. IEEE Trans. Neural Netw. 4, 400-407.
26. Li, B., Wang, Y., Wang, Y., Chen, Y., and Yang, H. (2014). "Training itself: mixed-signal training acceleration for memristor-based neural network" in 19th Asia and South Pacific Design Automation Conference (ASP-DAC) (Suntec City).
27. Merkel, C., and Kudithipudi, D. (2014). "A stochastic learning algorithm for neuromemristive systems," in 27th IEEE International System-on-Chip Conference (SOCC), (Las Vegas, NV), 359-366.
28. NVIDIA (2012). NVIDIA's next generation CUDA compute architecture: Kepler GK110. Whitepaper.
29. O'Connor, P., and Welling, M. (2016). Deep spiking networks. arXiv:1602.08323 [cs. NE].
30. Poppelbaum, W. J., Afuso, C., and Esch, J. W. (1967). "Stochastic computing elements and systems," in Proceedings of the AFIPS Fall Joint Computer Conference (Anaheim, CA), 635-644.
31. Prezioso, M., Merrikh-Bayat, F., Hoskins, B. D., Adam, G. C., Likharev, K. K., and Strukov, D. B. (2015). Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61-64. doi: 10.1038/nature14441
32. Rumelhart, D. E., Hinton, G. E., and Willims, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533-536.
33. Saighi, S., Mayr, C. G., Serrano-Gotarredona, T., Schmidt, H., Lecerf, G., Tomas, J., et al. (2015). Plasticity in memristive devices for spiking neural networks. Front. Neurosci. 9:51. doi: 10.3389/fnins.2015.00051
34. Seo, J., Lin, B., Kim, M., Chen, P., Kadetotad, D., Xu, Z., et al. (2015). On-chip sparse learning acceleration with CMOS and resistive synaptic devices. IEEE Trans. Nanotechnol. 14, 969-979. doi: 10.1109/TNANO.2015.2478861
35. Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. San Diego, CA: ICLR.
36. Soudry, D., Di Castro, D., Gal, A., Kolodny, A., and Kvatinsky, S. (2015). Memristor-based multilayer neural networks with online gradient descent training. IEEE Trans. Neural Netw. Learn. Syst. 26, 2408-2421. doi: 10.1109/TNNLS.2014.2383395
37. Steinbuch, K. (1961). Die lernmatrix. Kybernetik 1, 36-45.
38. Strukov, D., Snider, G., Stewart, D., and Williams, R. (2008). The missing memristor found. Nature 453, 80-83. doi: 10.1038/nature06932
39. Stuecheli, J. (2013). "Next Generation POWER microprocessor," in Hot Chips Conference (Palo Alto, CA).
40. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., et al. (2015). Going Deeper with Convolutions. Boston, MA: CVPR.
41. Wu, R., Yan, S., Shan, Y., Dang, Q., and Sun, G. (2015). Deep image: scaling up image recognition. arXiv:1501.02876 [cs. CV].
42. Xu, Z., Mohanty, A., Chen, P., Kadetotad, D., Lin, B., Ye, J., et al. (2014). Parallel programming of resistive cross-point array for synaptic plasticity. Procedia Comput. Sci. 41, 126-133. doi: 10.1016/j.procs.2014.11.094
43. Yu, S., Gao, B., Fang, Z., Yu, H., Kang, J., and Wong, H. S. (2013). A low energy oxide-based electronic synaptic device for neuromorphic visual systems with tolerance to device variation. Adv. Mater. 25, 1774-1779. doi: 10.1002/adma.201203680
44. Yu, S., Chen, P., Cao, Y., Xia, L., Wang, Y., and Wu, H. (2015). "Scaling-up resistive synaptic arrays for neuro-inspired architecture: challenges and prospect," in International Electron Devices Meeting (IEDM) (Washington, DC), 451-454. doi: 10.1109/iedm.2015.7409718