Equivalent-accuracy accelerated neural-network training using analogue memory

Nature

Volumn 558, Issue 7708, 2018, Pages 60-67

  Ambrogio, Stefano ^a   Narayanan, Pritish ^a   Tsai, Hsinyu ^a   Shelby, Robert M ^a   Boybat, Irem ^b,c   Di Nolfo, Carmelo ^a,c   Sidler, Severin ^a,c   Giordano, Massimo ^a   Bodini, Martina ^a,c   Farinha, Nathan C P ^a   Killeen, Benjamin ^a   Cheng, Christina ^a   Jaoudi, Yassine ^a   Burr, Geoffrey W ^a  

^a IBM ALMADEN RESEARCH CENTER   (United States)

^b IBM RESEARCH ZURICH   (Switzerland)

^c EPFL   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY ASSESSMENT; ALGORITHM; ARTIFICIAL NEURAL NETWORK; DATA SET; HARDWARE; MEMORY; SOFTWARE;

ACCURACY; ALGORITHM; ARTICLE; CLASSIFICATION; CONDUCTANCE; ENERGY; LEARNING; MACHINE LEARNING; MEMORY; NERVE CELL; NERVE CELL NETWORK; NEURAL NETWORK TRAINING; SIMULATION; SOFTWARE; SYNAPSE;

EID: 85048244412     PISSN: 00280836     EISSN: 14764687     Source Type: Journal    
DOI: 10.1038/s41586-018-0180-5     Document Type: Article

References (44)

1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436-444 (2015).
2. Coates, A. et al. Deep learning with COTS HPC systems. In Proc. 30th International Conference on Machine Learning 1337-1345 (Association for Computing Machinery, 2013).
3. Gupta, S., Agrawal, A., Gopalakrishnan, K. & Narayanan, P. Deep learning with limited numerical precision. In Proc. 30th International Conference on Machine Learning 1737-1746 (Association for Computing Machinery, 2015).
4. Merolla, P., Appuswamy, R., Arthur, J., Esser, S. K. & Modha, D. Deep neural networks are robust to weight binarization and other non-linear distortions. Preprint at https://arxiv.org/abs/1606.01981 (2016).
5. Nurvitadhi, E. et al. Can FPGAs beat GPUs in accelerating next-generation deep neural networks? In Proc. 2017 ACM/SIGSA International Symposium of Field-Programmable Gate Arrays 5-14 (Association for Computing Machinery, 2017).
6. Jouppi, N. P. et al. In-datacenter performance analysis of a tensor processing unit. In Proc. 2017 International Symposium on Computer Architecture 1-12 (Association for Computing Machinery, 2017).
7. Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668-673 (2014).
8. Esser, S. K. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl Acad. Sci. USA 113, 11441-11446 (2016).
9. Morie, T. & Amemiya, Y. An all-analog expandable neural network LSI with on-chip backpropagation learning. IEEE J. Solid-State Circuits 29, 1086-1093 (1994).
10. Burr, G. W. et al. Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element. In 2014 IEEE International Electron Devices Meeting T29.5 (IEEE, 2014).
11. Burr, G. W. et al. Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element. IEEE Trans. Electron Dev. 62, 3498-3507 (2015).
12. Gokmen, T. & Vlasov, Y. Acceleration of deep neural network training with resistive cross-point devices: design considerations. Front. Neurosci. 10, 333 (2016).
13. Burr, G. W. et al. Neuromorphic computing using non-volatile memory. Adv. Physics X 2, 89-124 (2017).
14. Yu, S. et al. Scaling-up resistive synaptic arrays for neuro-inspired architecture: challenges and prospect. In 2015 IEEE International Electron Devices Meeting 17.3 (IEEE, 2015).
15. Gao, L. et al. Fully parallel write/read in resistive synaptic array for accelerating on-chip learning. Nanotechnology 26, 455204 (2015).
16. Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61-64 (2015).
17. Jang, J.-W., Park, S., Burr, G. W., Hwang, H. & Jeong, Y.-H. Optimization of conductance change in Pr1-xCaxMnO3-based synaptic devices for neuromorphic systems. IEEE Electron Device Lett. 36, 457-459 (2015).
18. Jeong, Y. J., Kim, S. & Lu, W. D. Utilizing multiple state variables to improve the dynamic range of analog switching in a memristor. Appl. Phys. Lett. 107, 173105 (2015).
19. Kaneko, Y., Nishitani, Y. & Ueda, M. Ferroelectric artificial synapses for recognition of a multishaded image. IEEE Trans. Electron Dev. 61, 2827-2833 (2014).
20. Nandakumar, S. R. et al. Mixed-precision training of deep neural networks using computational memory. Preprint at https://arxiv.org/abs/1712.01192 (2017).
21. van de Burgt, Y. et al. A non-volatile organic electrochemical device as a low-voltage artificial synapse for neuromorphic computing. Nat. Mater. 16, 414-418 (2017).
22. Agarwal, S. et al. Achieving ideal accuracies in analog neuromorphic computing using periodic carry. In 2017 Symposium on VLSI Technology T13.2 (IEEE, 2017).
23. Lecun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278-2324 (1998).
24. Krizhevsky, A. Learning Multiple Layers of Features From Tiny Images. Ch. 3, https://www.cs.toronto.edu/~kriz/cifar.html (2009).
25. Narayanan, P. et al. Towards on-chip acceleration of the backpropagation algorithm using non-volatile memory. IBM J. Res. Develop. 61, 11 (2017).
26. Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by backpropagating errors. Nature 323, 533-536 (1986).
27. Xu, Z. et al. Parallel programming of resistive cross-point array for synaptic plasticity. Procedia Comput. Sci. 41, 126-133 (2014).
28. Papandreou, N. et al. Programming algorithms for multilevel phase-change memory. In 2011 IEEE International Symposium on Circuits and Systems 329-332 (IEEE, 2011).
29. Alibart, F., Gao, L., Hoskins, B. D. & Strukov, D. B. High-precision tuning of state for memristive devices by adaptable variation-tolerant algorithm. Nanotechnology 23, 075201 (2012).
30. Hu, M. et al. Dot-product engine for neuromorphic computing: programming 1T1M crossbar to accelerate matrix-vector multiplication. In Proc. 53rd Annual Design Automation Conference 19 (Association for Computing Machinery, 2016).
31. Fuller, E. J. et al. Li-ion synaptic transistor for low power analog computing. Adv. Mater. 29, 1604310 (2017).
32. Kim, S., Gokmen, T., Lee, H.-M. & Haensch, W. E. Analog CMOS-based resistive processing unit for deep neural network training. In 2017 IEEE 60th International Midwest Symposium on Circuits and Systems 422-425 (IEEE, 2017).
33. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning Ch. 8 (MIT Press, 2016).
34. Donahue, J. et al. DeCAF: a deep convolutional activation feature for generic visual recognition. In Proc. 31st International Conference on Machine Learning 647-655 (Association for Computing Machinery, 2014).
35. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the Inception architecture for computer vision. Preprint at https://arxiv.org/abs/1512.00567 (2015).
36. Mujtaba, H. Nvidia Volta GV100 12nm FinFET GPU detailed-Tesla V100 specifications include 21 billion transistors, 5120 CUDA cores, 16 GB HBM2 with 900 GB/s bandwidth. Wccftech https://wccftech.com/nvidia-volta-gv 100-gpu-tesla-v100-architecture-specifications-deep-dive/(2017).
37. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735-1780 (1997).
38. Cho, K., van Merrienboer, B., Bahdanau, D. & Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. Preprint at https://arxiv.org/abs/1409.1259 (2014).
39. Burr, G. W. et al. Access devices for 3D crosspoint memory. J. Vac. Sci. Technol. B 32, 040802 (2014).
40. Narayanan, P. et al. Reducing circuit design complexity for neuromorphic machine learning systems based on non-volatile memory arrays. In 2017 IEEE International Symposium on Circuits and Systems 1-4 (IEEE, 2017).
41. Ielmini, D., Lacaita, A. L. & Mantegazza, D. Recovery and drift dynamics of resistance and threshold voltages in phase-change memories. IEEE Trans. Electron Dev. 54, 308-315 (2007).
42. Pelgrom, M. J. M., Duinmaijer, A. C. J. & Welbers, A. P. G. Matching properties of MOS transistors. IEEE J. Solid-State Circuits 24, 1433-1439 (1989).
43. Cao, Y. What is predictive technology model (PTM)? SIGDA Newsl. 39, 1 (2009).
44. Bengio, Y. Louradour, J. Collobert, R. & Weston, J. Curriculum learning. In Proc. 26th Annual International Conference on Machine Learning 41-48 (ACM, 2009).