Large scale recurrent neural network on GPU

Proceedings of the International Joint Conference on Neural Networks

Volumn , Issue , 2014, Pages 4062-4069

  Li, Boxun ^a   Zhou, Erjin ^a   Huang, Bo ^a   Duan, Jiayi ^a   Wang, Yu ^a   Xu, Ningyi ^b   Zhang, Jiaxing ^b   Yang, Huazhong ^a  

^a TSINGHUA UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER GRAPHICS; DATA HANDLING; GRAPHICS PROCESSING UNIT; LARGE DATASET; MULTILAYER NEURAL NETWORKS; NETWORK LAYERS; PROGRAM PROCESSORS;

COMPUTATION COMPLEXITY; DATA PROCESSING APPLICATIONS; MASSIVE PARALLELISM; MICROSOFT RESEARCHES; PIPELINE IMPLEMENTATION; PROCESSED INFORMATION; RECURRENT NEURAL NETWORK (RNN); SENTENCE COMPLETIONS;

RECURRENT NEURAL NETWORKS;

EID: 84908481248     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IJCNN.2014.6889433     Document Type: Conference Paper

References (33)

1. B. Brown, M. Chui, and J. Manyika, "Are you ready for the era of big data?" McKinsey Quarterly, vol. 4, pp. 24-35, 2011.
2. N.. Workshop. Modern nonparametric methods in machine learning. [Online]. Available: http://nips2013.sched.org/2013-12-09/list/descriptions/type/workshops
3. R. Menéndez de Llano and J. L. Bosque, "Study of neural net training methods in parallel and distributed architectures", Future Generation Computer Systems, vol. 26, no. 2, pp. 267-275, 2010.
4. I. Basheer and M. Hajmeer, "Artificial neural networks: fundamentals, computing, design, and application", Journal of microbiological methods, vol. 43, no. 1, pp. 3-31, 2000.
5. J. Dean et al., "Large scale distributed deep networks", in NIPS, 2012.
6. K. Yu, "Large-scale deep learning at baidu", in Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 2013, pp. 2211-2212.
7. I. Sutskever, J. Martens, and G. E. Hinton, "Generating text with recurrent neural networks", in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 1017-1024.
8. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks", Neural computation, vol. 1, no. 2, pp. 270-280, 1989.
9. L. Deng, J. Li, J.-T. Huang, K. Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X. He, J. Williams et al., "Recent advances in deep learning for speech research at microsoft", ICASSP 2013, 2013.
10. J. Skaruz and F. Seredynski, "Recurrent neural networks towards detection of sql attacks", in Parallel and Distributed Processing Symposium, 2007. IPDPS 2007. IEEE International. IEEE, 2007, pp. 1-8.
11. J. Martens and I. Sutskever, "Learning recurrent neural networks with hessian-free optimization", in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 1033-1040.
12. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, "Strategies for training large scale neural network language models", in Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop on. IEEE, 2011, pp. 196-201.
13. R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks", Technical Report, Tech. Rep., 2012.
14. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew, "Deep learning with cots hpc systems", in Proceedings of the 30th International Conference on Machine Learning (ICML-13), vol. 28, no. 3, May 2013, pp. 1337-1345.
15. W. W. Fung, I. Sham, G. Yuan, and T. M. Aamodt, "Dynamic warp formation and scheduling for efficient gpu control flow", in Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture. IEEE Computer Society, 2007, pp. 407-420.
16. R. Raina, A. Madhavan, and A. Y. Ng, "Large-scale deep unsupervised learning using graphics processors." in ICML, vol. 9, 2009, pp. 873-880.
17. J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, A. Ng, and Q. V. Le, "On optimization methods for deep learning", in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 265-272.
18. M. Bodén, "A guide to recurrent neural networks and backpropagation", The Dallas project, SICS technical report, 2002.
19. D. Jurafsky and J. H. Martin, Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. MIT Press, vol. 2.
20. A. Pauls and D. Klein, "Faster and smaller n-gram language models." in ACL, 2011, pp. 258-267.
21. F. Jelinek, Statistical methods for speech recognition. MIT press, 1997.
22. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model", in Acoustics, Speech and Signal Processing (ICASSP), 2011.
23. T. Mikolov, M. Karafiát, L. Burget, J. Cernockì, and S. Khudanpur, "Recurrent neural network based language model", in INTERSPEECH, 2010, pp. 1045-1048.
24. Intel, "Intel math kernel library", http://software.intel.com/en-us/intelmkl/.
25. C. Nvidia, "Cublas library", NVIDIA Corporation, Santa Clara, California, vol. 15, 2008.
26. T. Mikolov, S. Kombrink, A. Deoras, L. Burget, and J. Černocky, "Rnnlm-recurrent neural network language modeling toolkit", in Proc. IEEE workshop on Automatic Speech Recognition and Understanding, 2011, p. 16.
27. V. Volkov and J. W. Demmel, "Benchmarking gpus to tune dense linear algebra", in Proceedings of the 2008 ACM/IEEE conference on Supercomputing. IEEE Press, 2008, p. 31.
28. V. Narasiman et al., "Improving gpu performance via large warps and two-level warp scheduling", in Micro. ACM, 2011, pp. 308-317.
29. NVIDIA, "Gpu computing sdk", https://developer.nvidia.com/gpucomputing-sdk.
30. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, "Building a large annotated corpus of english: The penn treebank", Computational linguistics, vol. 19, no. 2, pp. 313-330, 1993.
31. G. Zweig and C. J. Burges, "A challenge set for advancing language modeling", in Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT. Association for Computational Linguistics, 2012, pp. 29-36.
32. T. Mikolov, "Statistical language models based on neural networks", Ph. D. dissertation, Ph. D. thesis, Brno University of Technology, 2012.
33. A. Mnih and K. Kavukcuoglu, "Learning word embeddings efficiently with noise-contrastive estimation", in NIPS, 2013, pp. 2265-2273.