Toward on-chip acceleration of the backpropagation algorithm using nonvolatile memory

IBM Journal of Research and Development

Volumn 61, Issue 4, 2017, Pages

  Narayanan, P ^a   Fumarola, A ^a   Sanches, L L ^a   Hosokawa, K ^b   Lewis, S C ^c   Shelby, R M ^a   Burr, G W ^a  

^a IBM ALMADEN RESEARCH CENTER   (United States)

^b IBM RESEARCH   (United States)

^c IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85029520494     PISSN: 00188646     EISSN: 21518556     Source Type: Journal    
DOI: 10.1147/JRD.2017.2716579     Document Type: Article

References (29)

1. P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada, F. Akopyan, B. L. Jackson, N. Imam, C. Guo, Y. Nakamura, B. Brezzo, I. Vo, S. K. Esser, R. Appuswamy, B. Taba, A. Amir, M. D. Flickner, W. P. Risk, R. Manohar, and D. S. Modha, "A million spiking-neuron integrated circuit with a scalable communication network and interface," Science, vol. 345, no. 6197, pp. 668-673, 2014.
2. G. W. Burr, R. M. Shelby, C. di Nolfo, J. W. Jang, R. S. Shenoy, P. Narayanan, K. Virwani, E. U. Giacometti, B. Kurdi, and H. Hwang, "Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phasechange memory as the synaptic weight element," in Proc. Int. Electron Devices Meeting, 2014, pp. 29.5.1-29.5.4.
3. G. W. Burr, R. M. Shelby, S. Sidler, C. di Nolfo, J. Jang, I. Boybat, R. S. Shenoy, P. Narayanan, K. Virwani, E. U. Giacometti, B. Kurdi, and H. Hwang, "Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element," IEEE Trans. Electr. Device, vol. 62, no. 11, pp. 3498-3507, Nov. 2015.
4. G. W. Burr, P. Narayanan, R. M. Shelby, S. Sidler, I. Boybat, C. di Nolfo, and Y. Leblebici, "Large-scale neural networks implemented with nonvolatile memory as the synaptic weight element: comparative performance analysis (accuracy, speed, and power)," in Proc. Int. Electron Devices Meeting, 2015, pp. 4.4.1-4.4.4.
5. J.-W. Jang, S. Park, G. W. Burr, H. Hwang, and Y.-H. Jeong, "Optimization of conductance change in Pr1-xCaxMnO3-based synaptic devices for neuromorphic systems," IEEE Electron Device Lett., vol. 36, no. 5, pp. 457-459, May 2015.
6. S. Sidler, I. Boybat, R.M. Shelby, P. Narayanan, J. Jang, A. Fumarola, K. Moon, Y. Leblebici, H. Hwang, and G.W. Burr, "Large-scale neural networks implemented with non-volatile memory as the synaptic weight element: Impact of conductance response," in Proc. 46th Eur. Solid-State Device Res. Conf., 2016, pp. 440-443.
7. A. Fumarola, S. Sidler, P. Narayanan, J. Jang, K. Moon, R. M. Shelby, H. Hwang, and G. W. Burr, "Accelerating machine learning with non-volatile memory: Exploring device and circuit tradeoffs," in Proc. Int. Conf. Rebooting Comput., pp. 1-8, 2016.
8. P. Narayanan, L. L. Sanches, A. Fumarola, R. M. Shelby, J. Jang, H. Hwang, Y. Leblebici, and G. W. Burr, "Reducing circuit design complexity for neuromorphic machine learning systems based on non-volatile memory arrays," Proc. IEEE Int. Symp. Circ. Sys., May 2017.
9. G. Cauwenberghs, C. F. Neugebauer, and A. Yariv, "Analysis and verification of an analog VLSI incremental outer-product learning system," IEEE Trans. Neural Netw., vol. 3, no. 3, pp. 488-497, May 1992.
10. C. Schneider and H. Card, "Analogue CMOS Hebbian synapses," Electron. Lett., vol. 27, no. 9, pp. 785-786, 1991.
11. R. Hasan and T. M. Taha, "Enabling back propagation training of memristor crossbar neuromorphic processors," in Proc. Int. Joint Conf. Neural Netw., 2014, pp. 21-28.
12. M. Prezioso, F. Merrikh-Bayat, B. D. Hoskins, G. C. Adam, K. K. Likharev, and D. B. Strukov, "Training and operation of an integrated neuromorphic network based on metal-oxide memristors," Nature, vol. 521, no. 7550, pp. 61-64, 2015.
13. B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernández-Lobato, G.-Y. Wei, and D. Brooks, "Minerva: Enabling low-power, highly-accurate deep neural network accelerators," in Proc. 43rd Int. Symp. Comput. Archit., 2016, pp. 267-278.
14. P. Chi, S. Li, Z. Qi, P. Gu, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, "PRIME: A novel processing-in-memory architecture for neural network computation in ReRAM-based main memory," in Proc.43rd Annu. Int. Symp. Comput. Archit., 2016, vol. 43, pp. 27-39.
15. A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, "ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars," in Proc. 43rd Annu. Int. Symp. Comput. Archit., 2016, pp. 14-26.
16. S. Yu, P.-Y. Chen, Y. Cao, L. Xia, Y. Wang, and H. Wu, "Scalingup resistive synaptic arrays for neuro-inspired architecture: Challenges and prospect," in Proc. IEEE Int. Electron Devices Meet., 2015, pp. 17.3.1-17.3.4.
17. L. Gao, I.-T. Wang, P.-Y. Chen, S. Vrudhula, J.-S. Seo, Y. Cao, T.-H. Hou, and S. Yu, "Fully parallel write/read in resistive synaptic array for accelerating on-chip learning," Nanotechnology, vol. 26, no. 45, 2015, Art. no. 455204.
18. D. Negrov, I. Karandashev, V. Shakirov, Yu Matveyev, W. L. Dunin-Barkowski, and A. Zenkevich, "An approximate backpropagation learning rule for memristor based neural networks using synaptic plasticity," J. Neurocomp., vol. 237(C), pp. 193-199, 2017.
19. S. B. Eryilmaz, S. Joshi, E. Neftci, W. Wan, G. Cauwenberghs, and H-S. P. Wong, "Neuromorphic architectures with electronic synapses," in Proc. 17th Int. Symp. Quality Electron. Des., 2016, pp. 118-123.
20. T. Gokmen and Y. Vlasov, "Acceleration of deep neural network training with resistive cross-point devices: Design considerations," Frontiers Neurosci., vol. 10, 2016, Art. no. 333.
21. D. Soudry, D. Di Castro, A. Gal, A. Kolodny, and S. Kvatinsky, "Memristor-based multilayer neural networks with online gradient descent training," IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 10, pp. 2408-2421, 2015.
22. G. Indiveri, B. Linares-Barranco, T. J. Hamilton, A. Van Schaik, R. Etienne-Cummings, T. Delbruck, S. C. Liu, P. Dudek, P. Häfliger, S. Renaud, and J. Schemmel, "Neuromorphic silicon neuron circuits," Frontiers Neurosci., vol. 5, 2011, Art. no. 73.
23. S. Kim, M. Ishii, S. Lewis, T. Perri, M. BrightSky, W. Kim, R. Jordan, G.W. Burr, N. Sosa, A. Ray, J.-P. Han, C. Miller, K. Hosokawa, and C. Lam, "NVM neuromorphic core with 64k-cell (256-by-256) phase change memory synaptic array with on-chip neuron circuits for continuous in-situ learning," in Proc. IEEE Int. Electron Devices Meeting, 2015, pp. pp. 17.1.1-17.1.4.
24. R. A. Nawrocki, R. M. Voyles, and S. E. Shaheen, "A mini review of neuromorphic architectures and implementations," IEEE Trans. Electron Devices, vol. 63, no. 10, pp. 3819-3829, Oct. 2016.
25. G. W. Burr, R. S. Shenoy, K. Virwani, P. Narayanan, A. Padilla, B. Kurdi, and H. Hwang, "Access devices for 3D crosspoint memory," J. Vacuum Sci. Technol. B, vol. 32, no. 4, 2014, Art. no. 040802.
26. D. Rumelhart, G. E. Hinton, and J. L. McClelland, "A general framework for parallel distributed processing," in Parallel Distributed Processing, Cambridge, MA, USA: MIT Press, 1986.
27. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
28. P.-Y. Chen, B. Lin, I-T. Wang, T.-H. Hou, J. Ye, S. Vrudhula, J.-S. Seo, Y. Cao, and S. Yu, "Mitigating effects of non-ideal synaptic device characteristics for on-chip learning," in Proc. IEEE/ACM Int. Conf. Comput.-Aided Des., 2015, pp. 194-199.
29. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viegas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, "TensorFlow: Large-scale machine learning on heterogeneous distributed systems," arxiv:1603.04467, 2016.