Single stream parallelization of generalized LSTM-like RNNs on a GPU

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 1047-1051

  Hwang, Kyuyeon ^a   Sung, Wonyong ^a  

^a Seoul National University   (South Korea)

Author keywords

generalization; graphics processing unit (GPU); long short term memory (LSTM); parallelization; Recurrent neural network (RNN)

Indexed keywords

AUDIO SIGNAL PROCESSING; BRAIN; COMPUTER GRAPHICS; COMPUTER GRAPHICS EQUIPMENT; DATA HANDLING; GRAPHIC METHODS; LONG SHORT-TERM MEMORY; PROGRAM PROCESSORS; SPEECH COMMUNICATION;

GENERALIZATION; GRAPHICS PROCESSING UNIT (GPU); PARALLEL IMPLEMENTATIONS; PARALLELIZATIONS; RECURRENT NEURAL NETWORK (RNN); RECURRENT NEURAL NETWORK (RNNS); TRAINING ALGORITHMS; TRAINING PROCEDURES;

GRAPHICS PROCESSING UNIT;

EID: 84946011093     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7178129     Document Type: Conference Paper

References (11)

1. Geoffrey E Hinton and Ruslan R Salakhutdinov, "Reducing the dimensionality of data with neural networks, " Science, vol. 313, no. 5786, pp. 504-507, 2006.
2. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82-97, 2012.
3. Tomas Mikolov, Stefan Kombrink, Lukas Burget, JH Cernocky, and Sanjeev Khudanpur, "Extensions of recurrent neural network language model, " in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 5528-5531.
4. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins, "Learning to forget: Continual prediction with LSTM, " Neural computation, vol. 12, no. 10, pp. 2451-2471, 2000.
5. Ronald J Williams and David Zipser, "A learning algorithm for continually running fully recurrent neural networks, " Neural computation, vol. 1, no. 2, pp. 270-280, 1989.
6. Derek Monner and James A Reggia, "A generalized LSTMlike training algorithm for second-order recurrent neural networks, " Neural Networks, vol. 25, pp. 70-83, 2012.
7. Paul J Werbos, "Backpropagation through time: what it does and how to do it, " Proceedings of the IEEE, vol. 78, no. 10, pp. 1550-1560, 1990.
8. Xie Chen, YongqiangWang, Xunying Liu, Mark JF Gales, and Philip C Woodland, "Efficient GPU-based training of recurrent neural network language models using spliced sentence bunch, " in INTERSPEECH, 2014.
9. Robert Tarjan, "Depth-first search and linear graph algorithms, " SIAM journal on computing, vol. 1, no. 2, pp. 146-160, 1972.
10. Ronald J Williams and Jing Peng, "An efficient gradient-based algorithm for on-line training of recurrent network trajectories, " Neural Computation, vol. 2, no. 4, pp. 490-501, 1990.
11. Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu, "Sample size selection in optimization methods for machine learning, " Mathematical programming, vol. 134, no. 1, pp. 127-155, 2012.