Designing reconfigurable large-scale deep learning systems using stochastic computing

2016 IEEE International Conference on Rebooting Computing, ICRC 2016 - Conference Proceedings

Volumn , Issue , 2016, Pages

  Ren, Ao ^a   Li, Zhe ^a   Wang, Yanzhi ^a   Qiu, Qinru ^a   Yuan, Bo ^b  

^a Syracuse University   (United States)

^b City University of New York   (United States)

Author keywords

deep learning; large scale; neuron; reconfigurable; Stochastic computing

Indexed keywords

ARTIFICIAL INTELLIGENCE; BACKPROPAGATION; BINARY SEQUENCES; BIT ERROR RATE; COMPUTER HARDWARE; COMPUTER VISION; DATA HANDLING; HARDWARE; INTEGRATED CIRCUIT DESIGN; LEARNING ALGORITHMS; LEARNING SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; NETWORK-ON-CHIP; NEURAL NETWORKS; NEURONS;

ARITHMETIC OPERATIONS; CONVOLUTION FUNCTIONS; DEEP LEARNING; LARGE-SCALE; NATURAL LANGUAGE PROCESSING; PROCESSING TECHNIQUE; RECONFIGURABLE; STOCHASTIC COMPUTING;

STOCHASTIC SYSTEMS;

EID: 85005965765     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICRC.2016.7738685     Document Type: Conference Paper

References (35)

1. Y. Bengio, "Learning deep architectures for AI," Foundations and trends in Machine Learning, 2009.
2. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in Proc. of International Conference on Machine Learning (ICML), 2009.
3. J. Schmidhuber, "Deep learning in neural networks: An overview," Elsevier Neural Networks, 2015.
4. R. Raina, A. Madhavan, and A. Y. Ng, "Large-scale deep unsupervised learning using graphics processors," Annual International Conference on Machine Learning (ICML), 2009.
5. Q. V. Le, "Building high-level features using large scale unsupervised learning," Proc. of ICASSP, 2013.
6. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman, "Project Adam: Building an Efficient and Scalable Deep Learning Training System", Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, 2014.
7. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng, "Large scale distributed deep networks," Proc. of Advances in Neural Information Processing Systems (NIPS), 2012.
8. A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y. Ng, and B. Catanzaro, "Deep learning with COTS HPC systems," Proc. of Advances in Neural Information Processing Systems (NIPS), 2013.
9. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, "Caffe: Convolutional architecture for fast feature embedding," Proc. of ACM International Conference on Multimedia, 2014.
10. D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber, "Deep, big, simple neural nets for handwritten digital recognition," Neural Computation, 2010.
11. J. Bergstra, F. Bastien, O. Breuleux, P. Lamblin, R. Pascanu, O. Delalleau, G. Desjardins, D. Warde-Farley, I. Goodfellow, A. Bergeron, and Y. Bengio, "Theano: Deep learning on GPUs with Python," Journal of Machine Learning Research, 2011.
12. Y. Wang, T. Tang, L. Xia, B. Li, P. Gu, H. Yang, H. Li, Y. Xie, "Energy Efficient RRAM Spiking Neural Network for Real Time Classification." In Proceedings of the 25th edition on Great Lakes Symposium on VLSI, pp. 189-194. ACM, 2015.
13. O. Bichler, D. Querlioz, S. J. Thorpe, J. P. Bourgoin, C. Gamrat, "Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity." Neural Networks 32 (2012): 339-348.
14. S. B. Furber, D. R. Lester, L. A. Plana, J. D. Garside, E. Painkras, S. Temple, A. D. Brown, "Overview of the SpiNNaker System Architecture," in Computers, IEEE Transactions on, vol. 62, no. 12, pp. 2454-2467, Dec. 2013.
15. P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada, F. Akopyan, B. L. Jackson et al. "A million spiking-neuron integrated circuit with a scalable communication network and interface." Science 345, no. 6197 (2014): 668-673.
16. B. Gaines, "Stochastic computing systems," Advances in Information Systems Science, vol. 2, no. 2, pp. 37-172, 1969.
17. B. D. Brown and H. C. Card, "Stochastic neural computation I: computational elements," IEEE Trans. Comput., vol. 50, pp. 891-905, Sept. 2001.
18. W. Qian, X. Li, Marc D. Riedel, K. Bazargan, and D. J. Lilja, "An architecture for fault-tolerant computation with stochastic logic," IEEE Trans. on Computers, vol. 60, no. 1, pp. 93-105, 2011.
19. Y. Liu and K.K. Parhi, "Lattice FIR Digital Filters using Stochastic Computing," in Proc. of 2015 IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 1027-1031, April 2015
20. B. Yuan, Y. Wang and Z. Wang, "Area-Efficient Error-Resilient Discrete Fourier Transformation Design using Stochastic Computing," submitted to Great Lake Symp. on VLSI (GLSVLSI'2016)
21. B. Yuan, C. Zhang and Z. Wang, "Design Space Exploration for Hardware-Efficient Stochastic Computing: A Case Study on Discrete Cosine Transformation", accepted by IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP'2016)
22. A. Ardakani, F. Leduc-Primeau, N. Onizawa, T. Hanyu and W. Gross, "VLSI Implementation of Deep Neural Network using Integral Stochastic Computing,", available in arxiv.org.
23. P. Li and D. Lilja, "A low power fault-tolerance architecture for the kernel density estimation based image segmentation algorithm," in Proc. of IEEE International Conf. on Application-Specific Systems, Architectures and Processors (ASAP), pp. 161-168, 2011.
24. D. Zhang and H. Li, "A stochastic-based FPGA controller for an induction motor drive with integrated neural network algorithms," IEEE Trans. Industrial Electronics, vol. 55, no. 2, pp. 551-561, 2008.
25. M. D. Zeiler and R. Fergus, "Stochastic Pooling for Regularization of Deep Convolutional Neural Networks," available in arxiv.org
26. A. Krizhevsky, I. Sutskever and G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," in Proc. of NIPS, 2012.
27. D. Larkin, A. Kinane, V. Muresan and N. O'Connor, "An efficient hardware architecture for a neural network activation function generator," in Proc. of the ISNN Int. Symp. on Neural Networks, vol. 144, pp. 1319-1327, 2006.
28. S. Brown, R. J. Francis, J. Rose, and Z. G. Vranesic, Field-Programmable Gate Arrays, Springer, 1992.
29. A. B. Kahng, B. Li, L. S. Peh, and K. Samadi, "ORION 2.0: A fast and accurate NoC power and area model for early-stage design space exploration," Proc. of Design, Automation, and Test in Europe (DATE), 2009.
30. A. Alaghi and J. P. Hayes, "Survey of stochastic computing," ACM Trans. Embed. Comput. Syst, vol. 12, no. 92, May 2013.
31. TTD Team, R. Al-Rfou, G. Alain, A. Almahairi, C. Angermueller, D. Bahdanau, N. Ballas et al. "Theano: A Python framework for fast computation of mathematical expressions." arXiv preprint arXiv, 2016.
32. LeCun, Yann, C. Cortes, "The MNIST database of handwritten digits.", 1998.
33. LeCun, Yann. "LeNet-5, convolutional neural networks." URL: http://yann.lecun.com/exdb/lenet, 2015.
34. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." In Advances in neural information processing systems, pp. 1097-1105. 2012.
35. B. Jun, P. Kocher, "The Intel Random Number Generator.", Cryptography Research Inc. White Paper, 1999.