PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory

Proceedings - 2016 43rd International Symposium on Computer Architecture, ISCA 2016

Volumn , Issue , 2016, Pages 27-39

  Chi, Ping ^a   Li, Shuangchen ^a   Xu, Cong ^b   Zhang, Tao ^c   Zhao, Jishen ^d   Liu, Yongpan ^e   Wang, Yu ^e   Xie, Yuan ^a  

^a University of California   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

^c NVIDIA   (United States)

^d University of California Santa Cruz   (United States)

^e TSINGHUA UNIVERSITY   (China)

Author keywords

neural network; processing in memory; resistive random access memory

Indexed keywords

ARTIFICIAL INTELLIGENCE; BENCHMARKING; COMPUTATION THEORY; COMPUTER ARCHITECTURE; ENERGY CONSERVATION; ENERGY UTILIZATION; LEARNING SYSTEMS; MEMORY ARCHITECTURE; METALS; NETWORK ARCHITECTURE; NEURAL NETWORKS; NONVOLATILE STORAGE; RECONFIGURABLE HARDWARE; RRAM;

MATRIX VECTOR MULTIPLICATION; MICRO ARCHITECTURES; NETWORK COMPUTATIONS; NEURAL NETWORK (NN); PROCESSING IN MEMORY; RESISTIVE RANDOM ACCESS MEMORY; RESISTIVE RANDOM ACCESS MEMORY (RERAM); SOFTWARE DEVELOPER;

RANDOM ACCESS STORAGE;

EID: 84988345727     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ISCA.2016.13     Document Type: Conference Paper

References (84)

1. S. W. Keckler et al., "GPUs and the future of parallel computing," IEEE Micro, vol. 31, no. 5, pp. 7-17, 2011.
2. B. Akin et al., "Data reorganization in memory using 3Dstacked DRAM," in Proc. ISCA, 2015.
3. J. Ahn et al., "A scalable processing-in-memory accelerator for parallel graph processing," in Proc. ISCA, 2015.
4. D. Zhang et al., "TOP-PIM: Throughput-oriented programmable processing in memory," in Proc. HPDC, 2014.
5. S. H. Pugsley et al., "NDC: Analyzing the impact of 3Dstacked memory+ logic devices on mapreduce workloads," in Proc. ISPASS, 2014.
6. J. T. Pawlowski, "Hybrid memory cube: breakthrough DRAM performance with a fundamentally re-architected DRAM subsystem,," in Proc. of Hot Chips Symposium, 2011.
7. H.-S. Wong et al., "Metal-oxide RRAM," Proc. of the IEEE, vol. 100, no. 6, pp. 1951-1970, 2012.
8. A. Vincent et al., "Spin-transfer torque magnetic memory as a stochastic memristive synapse," in Proc. ISCAS, 2014.
9. G. Burr et al., "Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element," in Proc. IEDM, 2014.
10. M. Hu et al., "Hardware realization of BSB recall function using memristor crossbar arrays," in Proc. DAC, 2012.
11. B. Li et al., "Memristor-based approximated computation," in Proc. ISLPED, 2013.
12. M. Prezioso et al., "Training and operation of an integrated neuromorphic network based on metal-oxide memristors," Nature, vol. 521, no. 7550, pp. 61-64, 2015.
13. Y. Kim et al., "A reconfigurable digital neuromorphic processor with memristive synaptic crossbar for cognitive computing," J. Emerg. Technol. Comput. Syst., vol. 11, no. 4, pp. 38:1-38:25, 2015.
14. Z. Chen et al., "Optimized learning scheme for grayscale image recognition in a RRAM based analog neuromorphic system," in Proc. IEDM, 2015.
15. G. W. Burr et al., "Large-scale neural networks implemented with non-volatile memory as the synaptic weight element: Comparative performance analysis (accuracy, speed, and power)," in Proc. IEDM, 2015.
16. A. Coates et al., "Deep learning with COTS HPC systems," in Proc. ICML, 2013.
17. T. Chen et al., "DianNao: A small-footprint high-throughput accelerator for ubiquitous machine-learning," in Proc. ASPLOS, 2014.
18. Y. Chen et al., "DaDianNao: A machine-learning supercomputer," in Proc. MICRO, 2014.
19. P. Merolla et al., "A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm," in Proc. CICC, 2011.
20. C. Xu et al., "Overcoming the challenges of crossbar resistive memory architectures," in Proc. HPCA, 2015.
21. M.-J. Lee et al., "A fast, high-endurance and scalable nonvolatile memory device made from asymmetric Ta2O5-x/TaO2-x bilayer structures," Nature Materials, vol. 10, no. 8, pp. 625-630, 2011.
22. C.-W. Hsu et al., "Self-rectifying bipolar TaOx/TiO2 RRAM with superior endurance over 1012 cycles for 3D high-density storage-class memory," in Proc. VLSIT, 2013.
23. M. K. Qureshi et al., "Enhancing lifetime and security of PCM-based main memory with Start-Gap wear leveling," in Proc. MICRO, 2009.
24. D. Niu et al., "Design trade-offs for high density cross-point resistive memory," in Proc. ISLPED, 2012.
25. A. Kawahara et al., "An 8Mb multi-layered cross-point ReRAM macro with 443MB/s write throughput," in Proc. ISSCC, 2012.
26. S. Yu et al., "3D vertical RRAM-scaling limit analysis and demonstration of 3D array operation," in Proc. VLSIT, 2013.
27. C. Xu et al., "Architecting 3D vertical resistive memory for next-generation storage systems," in Proc. ICCAD, 2014.
28. S. Yu et al., "Investigating the switching dynamics and multilevel capability of bipolar metal oxide resistive switching memory," Applied Physics Letters, vol. 98, p. 103514, 2011.
29. M.-C. Wu et al., "A study on low-power, nanosecond operation and multilevel bipolar resistance switching in ti/zro2/pt nonvolatile memory with 1t1r architecture," Semiconductor Science and Technology, vol. 27, p. 065010, 2012.
30. L. Zhang et al., "SpongeDirectory: Flexible sparse directories utilizing multi-level memristors," in Proc. PACT, 2014.
31. F. Alibart et al., "High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm," Nanotechnology, vol. 23, no. 7, p. 075201, 2012.
32. H. Esmaeilzadeh et al., "Neural acceleration for generalpurpose approximate programs," in Proc. MICRO, 2012.
33. R. St. Amant et al., "General-purpose code acceleration with limited-precision analog computation," in Proc. ISCA, 2014.
34. S. K. Esser et al., "Cognitive computing systems: Algorithms and applications for networks of neurosynaptic cores," in Proc. IJCNN, 2013.
35. J. Seo et al., "A 45nm CMOS neuromorphic chip with a scalable architecture for learning in networks of spiking neurons," in Proc. CICC, 2011.
36. T. M. Taha et al., "Exploring the design space of specialized multicore neural processors," in Proc. IJCNN, 2013.
37. A. Shafiee et al., "ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars," in Proc. ISCA, 2016.
38. M. Jung et al., "Design of a large-scale storage-class RRAM system," in Proc. ICS, 2013.
39. D. C. Cireşan et al., "Flexible, high performance convolutional neural networks for image classification," in Proc. IJCAI, 2011.
40. J. Schmidhuber, "Multi-column deep neural networks for image classification," in Proc. CVPR, 2012.
41. S. Sahin et al., "Neural network implementation in hardware using FPGAs," in Neural Information Processing, vol. 4234, pp. 1105-1112, 2006.
42. C. Farabet et al., "Cnp: An FPGA-based processor for convolutional networks," in Proc. FPL, 2009.
43. J.-Y. Kim et al., "A 201.4 GOPS 496 mW real-time multiobject recognition processor with bio-inspired neural perception engine," JSSC, vol. 45, no. 1, pp. 32-45, 2010.
44. D. Liu et al., "Pudiannao: A polyvalent machine learning accelerator," in Proc. ASPLOS, 2015.
45. S. Liu et al., "Cambricon: An instruction set architecture for neural networks," in Proc. ISCA, 2016.
46. C. Kozyrakis et al., "Scalable processors in the billiontransistor era: IRAM," Computer, vol. 30, no. 9, pp. 75-78, 1997.
47. D. Patterson et al., "Intelligent ram (iram): The industrial setting, applications, and architectures," in Proc. ICCD, 1997.
48. J. Draper et al., "The architecture of the DIVA processingin-memory chip," in Proc. ICS, 2002.
49. M. Gokhale et al., "Processing in memory: The terasys massively parallel PIM array," Computer, vol. 28, no. 4, pp. 23-31, 1995.
50. D. Elliott et al., "Computational RAM: The case for SIMD computing in memory," in Workshop on Mixing Logic and DRAM at ISCA, 1997.
51. T. Yamauchi et al., "A single chip multiprocessor integrated with DRAM," in Workshop on Mixing Logic and DRAM at ISCA, 1997.
52. M. Oskin et al., "Active pages: a computation model for intelligent memory," in Proc. ISCA, 1998.
53. R. Balasubramonian et al., "Near-data processing: Insights from a micro-46 workshop," Micro, IEEE, vol. 34, no. 4, pp. 36-42, 2014.
54. R. Nair et al., "Active memory cube: A processing-in-memory architecture for exascale systems," IBM Journal of Research and Development, vol. 59, no. 2/3, pp. 17:1-17:14, 2015.
55. Z. Guz et al., "Real-time analytics as the killer application for processing-in-memory," in Proc. WoNDP, 2014.
56. N. S. Mirzadeh et al., "Sort vs. hash join revisited for nearmemory execution," in Proc. ASBD, 2015.
57. J. Jeddeloh and B. Keeth, "Hybrid memory cube new DRAM architecture increases density and performance," in Proc. VLSIT, 2012.
58. D. U. Lee et al., "A 1.2V 8Gb 8-channel 128GB/s highbandwidth memory (HBM) stacked DRAM with effective microbump I/O test methods using 29nm process and TSV," in Proc. ISSCC, 2014.
59. S. Li et al., "Pinatubo: A processing-in-memory architecture for bulk bitwise operations in emerging non-volatile memories," in Proc. DAC, 2016.
60. F. Alibart et al., "Hybrid CMOS/nanodevice circuits for high throughput pattern matching applications," in Proc. AHS, 2011.
61. Q. Guo et al., "A resistive TCAM accelerator for dataintensive computing," in Proc. MICRO, 2011.
62. Q. Guo et al., "AC-DIMM: Associative computing with STTMRAM," in Proc. ISCA, 2013.
63. B. Li et al., "RRAM-based analog approximate computing," TCAD, vol. 34, no. 12, pp. 1905-1917, 2015.
64. J. Li et al., "A novel reconfigurable sensing scheme for variable level storage in phase change memory," in Proc. IMW, 2011.
65. L. Gao et al., "A high resolution nonvolatile analog memory ionic devices," in Proc. NVMW, 2013.
66. M. Hu et al., "Dot-product engine: Programming memristor crossbar arrays for efficient vector-matrix multiplication," in ICCAD'15 Workshop on "Towards Efficient Computing in the Dark Silicon Era", 2015.
67. Y. Lecun et al., "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
68. M. Courbariaux et al., "Low precision storage for deep learning," CoRR, vol. abs/1412.7024, 2014.
69. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. ICLR, 2015.
70. F. Alibart et al., "Pattern classification by memristive crossbar circuits using ex situ and in situ training," Nature communications, vol. 4, 2013.
71. M. Hu et al., "BSB training scheme implementation on memristor-based circuit," in Proc. CISDA, 2013.
72. B. Li et al., "Training itself: Mixed-signal training acceleration for memristor-based neural network," in Proc. ASP-DAC, 2014.
73. B. Liu et al., "Digital-assisted noise-eliminating training for memristor crossbar-based analog neuromorphic computing engine," in Proc. DAC, 2013.
74. B. Liu et al., "Reduction and IR-drop compensations techniques for reliable neuromorphic computing systems," in Proc. ICCAD, 2014.
75. P. Chi et al., "Processing-in-memory in ReRAM-based main memory," SEAL-lab Technical Report, no. 2015-001, 2015.
76. V. Seshadri et al., "RowClone: Fast and energy-efficient in-DRAM bulk data copy and initialization," in Proc. MICRO, 2013.
77. B. Verghese et al., "Operating system support for improving data locality on CC-NUMA compute servers," in Proc. ASPLOS, 1996.
78. N. Agarwal et al., "Page placement strategies for GPUs within heterogeneous memory systems," in Proc. ASPLOS, 2015.
79. M. K. Qureshi et al., "Morphable memory system: A robust architecture for exploiting multi-level phase change memories," in Proc. ISCA, 2010.
80. P. Zhou et al., "Dynamic tracking of page miss ratio curve for memory management," in Proc. ASPLOS, 2004.
81. X. Dong et al., "Nvsim: A circuit-level performance, energy, and area model for emerging nonvolatile memory," TCAD, vol. 31, no. 7, pp. 994-1007, 2012.
82. K. Chen et al., "CACTI-3DD: Architecture-level modeling for 3D die-stacked DRAM main memory," in Proc. DATE, 2012.
83. N. P. Jouppi et al., "CACTI-IO: CACTI with off-chip powerarea-timing models," in Proc. ICCAD, 2012.
84. C. Xu et al., "Understanding the trade-offs in multi-level cell ReRAM memory design," in Proc. DAC, 2013.