FPGA acceleration of recurrent neural network based language model

Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015

Volumn , Issue , 2015, Pages 111-118

  Li, Sicheng ^a   Wu, Chunpeng ^a   Li, Hai ^a   Li, Boxun ^b   Wang, Yu ^b   Qiu, Qinru ^c  

^a UNIVERSITY OF PITTSBURGH   (United States)

^b TSINGHUA UNIVERSITY   (China)

^c SYRACUSE UNIVERSITY   (United States)

Author keywords

Acceleration; FPGA; Language model; Recurrent neural network (RNN)

Indexed keywords

ACCELERATION; BIOMIMETICS; COMPUTATIONAL LINGUISTICS; COMPUTERS; FIELD PROGRAMMABLE GATE ARRAYS (FPGA); NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS;

BIOLOGICALLY INSPIRED MODELS; COMPUTATION EFFICIENCY; HARDWARE IMPLEMENTATIONS; HISTORICAL INFORMATION; LANGUAGE MODEL; NATURAL LANGUAGE PROCESSING; RECURRENT NEURAL NETWORK (RNN); TRAINING ACCELERATION;

RECURRENT NEURAL NETWORKS;

EID: 84943378851     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/FCCM.2015.50     Document Type: Conference Paper

References (26)

1. I. Sutskever, J. Martens, and G. E. Hinton, "Generating text with recurrent neural networks, " in Proc. of the 28th Int. Conf. on Machine Learning (ICML), 2011, pp. 1017-1024.
2. Q. Qiu, Q. Wu, M. Bishop, R. Pino, and R. Linderman, "A parallel neuromorphic text recognition system and its implementation on a heterogeneous high-performance computing cluster, " IEEE Trans. on Comp. , pp. 886-899, 2013.
3. L. Deng, J. Li, J.-T. Huang, K. Yao, D. Yu, and F. Seide, "Recent advances in deep learning for speech research at microsoft, " in IEEE Conf. on ICASSP, 2013, pp. 8604-8608.
4. A Recursive Recurrent Neural Network for Statistical Machine Translation. ACL, 2014. [Online]. Available: http://research. microsoft. com/apps/pubs/default. aspx?id=215646
5. H. Sak, O. Vinyals, G. Heigold, A. Senior, E. McDermott, R. Monga, and M. Mao, "Sequence discriminative distributed training of long short-term memory recurrent neural networks, " in Interspeech, 2014.
6. J. T. Goodman, "A bit of progress in language modeling, " Comp. Speech & Language, vol. 15, no. 4, pp. 403-434, 2001.
7. A. Pauls and D. Klein, "Faster and smaller n-gram language models, " in Proc. ACL, 2011, pp. 258-267.
8. G. Potamianos and F. Jelinek, "A study of n-gram and decision tree letter language modeling methods, " Speech Communication, vol. 24, no. 3, pp. 171-192, 1998.
9. S. Khudanpur and J. Wu, "A maximum entropy language model integrating n-grams and topic dependencies for conversational speech recognition, " in IEEE Conf. on ICASS, vol. 1, 1999, pp. 553-556 vol. 1.
10. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, "A neural probabilistic language model, " J. Mach. Learn. Res. , vol. 3, pp. 1137-1155, 2003.
11. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. H. Cernocky, "Empirical evaluation and combination of advanced language modeling techniques, " in Interspeech. ISCA, 2011.
12. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, "Strategies for training large scale neural network language models, " in IEEE Workshop ASRU, Dec 2011, pp. 196-201.
13. C. Chelba, J. Schalkwyk, T. Brants, V. Ha, B. Harb, W. Neveitt, C. Parada, and P. Xu, "Query language modeling for voice search, " in Proc. SLT, 2010, pp. 127-132.
14. H. Esmaeilzadeh, A. Sampson, L. Ceze, and D. Burger, "Neural acceleration for general-purpose approximate programs, " Micro, IEEE, vol. 33, no. 3, pp. 16-27, 2013.
15. B. Li, E. Zhou, B. Huang, J. Duan, and Y. Wang, "Large scale recurrent neural network on gpu, " in IJCNN, 2014, pp. 4062-4069.
16. Y. Maeda and M. Wakamura, "Simultaneous perturbation learning rule for recurrent neural networks and its fpga implementation, " IEEE Trans. on Neural Networks, vol. 16, no. 6, pp. 1664-1672, 2005.
17. Convey Reference Manual, Convey computer, 2012, http://www. conveycomputer. com/.
18. NVIDIA CUDA C Programming Guide, NVIDIA Corp.
19. Intel math kernel library, Intel, http://software. intel. com/enus/ intelmkl/.
20. H. Amin, K. Curtis, and B. Hayes-Gill, "Piecewise linear approximation applied to nonlinear function of a neural network, " Circuits, Devices and Systems, IEEE Proc. , vol. 144, no. 6, pp. 313-317, 1997.
21. D. L. Ly and P. Chow, "A high-performance fpga architecture for restricted boltzmann machines, " in Proc. FPGA, 2009, pp. 73-82.
22. G. Zweig and C. J. Burges, "A challenge set for advancing language modeling, " in Proc. NAACL-HLT 2012, pp. 29-36.
23. T. Mikolov, "Statistical language models based on neural networks, " Ph. D. dissertation, Brno Uni. of Technology, 2012.
24. A. Mnih and K. Kavukcuoglu, "Learning word embeddings efficiently with noise-contrastive estimation, " in Proc. NIPS, 2013, pp. 2265-2273.
25. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, "Building a large annotated corpus of english: The penn treebank, " Computational linguistics, vol. 19, no. 2, pp. 313-330, 1993.
26. M. Pietras, "Hardware conversion of neural networks simulation models for neural processing accelerator implemented as fpga-based soc, " in Proc. FPL, 2014, pp. 1-4.