Sequence-discriminative training of recurrent neural networks

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Volumn 2015-August, Issue , 2015, Pages 2100-2104

  Voigtlaender, Paul ^a   Doetsch, Patrick ^a   Wiesler, Simon ^a   Schluter, Ralf ^a   Ney, Hermann ^a,b  

^a RWTH AACHEN UNIVERSITY   (Germany)

^b UFR 919 Laboratoire d'Informatique Pour la Mécanique et les Sciences de l'Ingénieur   (France)

Author keywords

handwriting recognition; long shortterm memory; recurrent neural networks; sequence discriminative training

Indexed keywords

AUDIO SIGNAL PROCESSING; CHARACTER RECOGNITION; ENTROPY; LONG SHORT-TERM MEMORY; SPEECH COMMUNICATION; STEREOPHONIC BROADCASTING;

CONTEXTUAL INFORMATION; CROSS ENTROPY; DISCRIMINATIVE TRAINING; FEED-FORWARD NETWORK; HANDWRITING RECOGNITION; MAXIMUM MUTUAL INFORMATION; OBJECTIVE FUNCTIONS;

RECURRENT NEURAL NETWORKS;

EID: 84946067332     PISSN: 15206149     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICASSP.2015.7178341     Document Type: Conference Paper

References (25)

1. Marcus Liwicki, Alex Graves, Horst Bunke, and JUrgen Schmidhuber, "A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks," in Proc. of the Int. Con! on Document Anal. and Recognition (ICDAR), 2007, pp. 367-371.
2. Brian Kingsbury, Tara N. Sainath, and Hagen Soltau, "Scalable minimum bayes risk training of deep neural network acoustic models using distributed hessian-free optimization.," in Proc. of the Ann. Con! of the Int. Speech Commun. Assoc. (Interspeech). 2012,I SCA.
3. Karel Vesely, Arnab Ghoshal, LukMi Burget, and Daniel Povey, "Sequence-discriminative training of deep neural networks," in Proc. of the Ann. Con! of the Int. Speech Commun. Assoc. (Interspeech), August 2013.
4. Hang Su, Gang Li, Dong Yu, and Frank Seide, "Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription," in Proc. of the Int. Con! on Acoust., Speech, and Signal Process. (ICASSP),2013.
5. Georg Heigold, Erik McDermott, Vincent Vanhoucke, Andrew Senior, and Michiel Bacchiani, "Asynchronous stochastic optimization for sequence training of deep neural networks," in Proc. of the Int. Conf. on Acoust., Speech, and Signal Process. (ICASSP), Florence, Italy, May 2014, pp. 5624-5628.
6. L. R. Bahl, P. F. Brown, P. V. de Souza, and R. L. Mercer, "Maximum mutual information estimation of hidden Markov model parameters for speech recognition," in Proc. of the Int. Con! on Acoust., Speech, and Signal Process. (ICASSP), Tokyo, 1986, pp. 49-52.
7. Brian Kingsbury, "Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling," in Proc. of the Int. Con! on Acoust., Speech, and Signal Process. (ICASSP), 2009, pp. 3761-3764.
8. Hasim Sak, Oriol Vinyals, Georg Heigold, Andrew Senior, Erik McDermott, Rajat Monga, and Mark Mao, "Sequence discriminative distributed training of long short-term memory recurrent neural networks," in Proc. of the Ann. Conf. of the Int. Speech Commun. Assoc. (Interspeech), 2014.
9. Herve A. Bourlard and Nelson Morgan, Connectionist Speech Recognition: A Hybrid Approach, Kluwer Academic Publishers, Norwell, MA, USA, 1993.
10. R. J. Williams and D. Zipser, "Gradient-based learning algorithms for recurrent networks and their computational complexity," in Back-propagation: Theory, Architectures and Applications, Y. Chauvin and D. E. Rumelhart, Eds., chapter 13, pp. 433-486. Hillsdale, NJ: Erlbaum, 1995.
11. Sepp Hochreiter, "The vanishing gradient problem during learning recurrent neural nets and problem solutions," Int. J. of Uncertainty, Fuzziness and Knowledge-Based Systems, vo!. 6, no. 2, pp. lO7-116, Apr. 1998.
12. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vo!. 9, no. 8, pp. 1735-1780, 1997.
13. Michal Kozielski, Patrick Doetsch, and Hermann Ney, "Improvements in rwth's system for off-line handwriting recognition," in Proc. of the Int. Con! on Document Anal. and Recognition (ICDAR), 2013, pp. 935-939.
14. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural networks," Trans. on Signal Processing, vo!. 45, no. 11, pp. 2673-2681, Nov. 1997.
15. R. SchlUter, B. Muller, F. Wessel, and H. Ney, "Interdependence of language models and discriminative training," in Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Keystone, CO, USA, Dec. 1999, pp. 119-122.
16. James Bergstra, 01ivier Breuleux, Frederic Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio, "Theano: a CPU and GPU math expression compiler," in Proc. of the Python for Scientific Computing Conference, June 2010.
17. David Rybach, Stefan Hahn, Patrick Lehnen, David Nolden, Martin Sundermeyer, Zoltan TUske, Simon Wiesler, Ralf Schltiter, and Hermann Ney, "Rasr-the rwth aachen university open source speech recognition toolkit," in Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Hawaii, USA, Dec. 2011.
18. u.-V. Marti and H. Bunke, "The lAM-database: an english sentence database for offline handwriting recognition," Int. J. on Document Anal. and Recognition, vo!. 5, no. 1, pp. 39-46, Nov. 2002.
19. Michal Kozielski, David Rybach, Stefan Hahn, Ralf Schliiter, and Hermann Ney, "Open vocabulary handwriting recognition using combined word-level and character-level language models," in Proc. of the Int. Con! on Acoust., Speech, and Signal Process. (ICASSP), Vancouver, Canada, May 2013, pp. 8257-8261.
20. E. Grosicki and H. El Abed, "ICDAR 2009 handwriting recognition competition," in Proc. of the Int. Con! on Document Anal. and Recognition (ICDAR), July 2009, pp. 1398-1402.
21. Patrick Doetsch, Michal Kozielski, and Hermann Ney, "Fast and robust training of recurrent neural networks for offline handwriting recognition," in International Conference on Frontiers in Handwriting Recognition, Crete, Greece, Sept. 2014.
22. Hynek Hermansky, Daniel P. W. Ellis, and Sangita Sharma, 'Tandem connectionist feature extraction for conventional hmm systems," in Proc. of the Int. Con! on Acoust., Speech, and Signal Process. (ICASSP), 2000, pp. 1635-1638.
23. Vu Pham, Thodore Bluche, Christopher Kermorvant, and Jrme Louradour, "Dropout improves recurrent neural networks for handwriting recognition," in Int. Conf. on Frontiers in Handwriting Recognition, 2014.
24. Alex Graves and JUrgen Schmidhuber, "Offline handwriting recognition with multidimensional recurrent neural networks," in Advances in Neural Inform. Process. Syst. (NIPS), Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Leon Bottou, Eds. 2008, pp. 545-552, Curran Associates, Inc.
25. Alex Graves, Santiago Fernandez, Faustino J. Gomez, and JUrgen Schmidhuber, " Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks," in Proc. of the Int. Con! on Mach. Learning (ICML), William W. Cohen and Andrew Moore, Eds. 2006, vol. 148 of ACM International Conference Proceeding Series, pp. 369-376, ACM.