Acoustic modeling with hierarchical reservoirs

IEEE Transactions on Audio, Speech and Language Processing

Volumn 21, Issue 11, 2013, Pages 2439-2450

  Triefenbach, Fabian ^a   Jalalvand, Azarakhsh ^a   Demuynck, Kris ^a   Martens, Jean Pierre ^a  

^a GHENT UNIVERSITY   (Belgium)

Author keywords

Acoustic modeling; automatic speech recognition; recurrent neural networks; reservoir computing

Indexed keywords

ACOUSTIC MODEL; AUTOMATIC SPEECH RECOGNITION; GAUSSIAN MIXTURE MODEL (GMMS); HIDDEN MARKOV MODELS (HMMS); RECURRENT NEURAL NETWORK (RNN); RESERVOIR ARCHITECTURE; RESERVOIR COMPUTING; TRAINING PROCEDURES;

HIDDEN MARKOV MODELS; RECURRENT NEURAL NETWORKS;

SPEECH RECOGNITION;

EID: 84886714036     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2013.2280209     Document Type: Article

References (60)

1. H. Bourlard and C. J. Wellekens, "Links between Markov models and multilayer perceptrons," IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 12, pp. 1167-1178, Dec. 1990.
2. N. Morgan and H. Bourlard, "Continuous speech recognition using multilayer perceptrons with hiddenMarkov models," in Proc. ICASSP, 1990, pp. 413-416.
3. H. Bourlard and N. Morgan, Connectionist speech recognition: a hybrid approach, ser. TheKluwer International Series in Engineering and Computer Science. NewYork,NY, USA: Springer, 1994, vol. 247.
4. A. Robinson, "An application of recurrent neural nets to phone probability estimation," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 298-305, Mar. 1994.
5. N. Morgan and H. Bourlard, "Neural networks for statistical recognition of continuous speech," Proc. IEEE, vol. 83, no. 5, pp. 742-772, May 1995.
6. G. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, Jul. 2006. (Pubitemid 44024729)
7. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layerwise training of deep networks," in Proc. NIPS, , 2007, pp. 153-160.
8. G. Hinton, L. Deng, D. Yu, G. Dahl, A.Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
9. T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P. Novák, and A. r. Mohamed, "Making deep belief networks effective for large vocabulary continuous speech recognition," in Proc. ASRU, 2011, pp. 30-35.
10. A. Mohamed, G. Dahl, and G. Hinton, "Acoustic modeling using deep belief networks," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 14-22, Jan. 2012.
11. G. Dahl,D.Yu, L.Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012.
12. L. Deng, D. Yu, and J. Platt, "Scalable stacking and learning for building deep architectures," in Proc. ICASSP, 2012, pp. 2133-2136.
13. B. Hutchinson, L. Deng, and D. Yu, "A deep architecture with bilinear modeling of hidden representations: applications to phonetic recognition," in Proc. ICASSP, 2012, pp. 4805-4808.
14. N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke, "Application of pretrained deep neural networks to large vocabulary speech recognition," in Proc. Interspeech, 2012.
15. P. Schwarz, P. Matejka, and J. Cernocky, "Hierarchical structures of neural networks for phoneme recognition," in Proc. ICASSP, 2006, pp. 325-328.
16. L. Toth, "A hierarchical, context-dependent neural network architecture for improved phone recognition," in Proc. ICASSP, 2011, pp. 5040-5043.
17. J. Pinto, G. S. V. S. Sivaram, M. Magimai-Doss, H. Hermansky, and H. Bourlard, "Analysis of MLP-based hierarchical phoneme posterior probability estimator," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 2, pp. 225-241, Feb. 2011.
18. G. Sivaram and H. Hermansky, "Sparse multilayer perceptron for phoneme recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 23-29, Jan. 2012.
19. D. E. Rumelhart, G. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, no. 6088, pp. 533-536, Oct. 1986. (Pubitemid 16025374)
20. H. Jaeger, "Tutorial on training recurrent neural networks, covering BPTT, RTRL, EKF and the echo state network approach," GMD Rep. 159, German Nat. Res. Center for Inform. Technol. Tech. Rep., 2002.
21. D.Verstraeten, B. Schrauwen, M. D'Haene, andD. Stroobandt, "An experimental unification of reservoir computing methods," Neural Netw., vol. 20, no. 3, pp. 391-403, Apr. 2007. (Pubitemid 46856109)
22. M. Lukosevi?ius and H. Jaeger, "Reservoir computing approaches to recurrent neural network training," Comput. Sci. Rev., vol. 3, no. 3, pp. 127-149, Aug. 2009.
23. M. Lukosevi?ius, H. Jaeger, and B. Schrauwen, "Reservoir computing trends," KI-Künstliche Intell., vol. 26, no. 4, pp. 365-371, Nov. 2012.
24. G. Holzmann and H. Hauser, "Echo state networks with filter neurons and a delay & sum readout," Neural Netw., vol. 23, no. 2, pp. 244-256, Mar. 2010.
25. M. Skowronski and J. Harris, "Automatic speech recognition using a predictive echo state network classifier," Neural Netw., vol. 20, no. 3, pp. 414-423, Apr. 2007. (Pubitemid 46863403)
26. A. Jalalvand, F. Triefenbach, D. Verstraeten, and J.-P. Martens, "Connected digit recognition by means of reservoir computing," in Proc. Interspeech, 2011, pp. 1725-1728.
27. A. Jalalvand, F. Triefenbach, and J.-P. Martens, "Continuous digit recognition in noise: Reservoirs can do an excellent job," in Proc. Interspeech, 2012, paper ID 644.
28. L. Rabiner, "A tutorial on hiddenMarkov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
29. H. Jiang, "Discriminative training of hmms for automatic speech recognition: A survey," Comput. Speech Lang., vol. 24, no. 4, pp. 589-608, Oct. 2010.
30. B. Merialdo, "Phonetic recognition using hidden Markov models and maximum mutual information training," in Proc. ICASSP, 1988, pp. 111-114. (Pubitemid 18665963)
31. M. Richard and R. Lippmann, "Neural network classifiers estimate posterior probabilities," Neural Comput., vol. 3, no. 4, pp. 461-483, 1991.
32. H. Hermansky and S. Sharma, "Temporal patterns (TRAPS) in ASR of noisy speech," in Proc. ICASSP, 1999, pp. 289-292.
33. Z. Tüske, M. Sundermeyer, R. Schlüter, and H. Ney, "Context-dependent MLPs for LVCSR: TANDEM, hybrid or both?," in Proc. Interspeech, 2012, Paper ID 734.
34. T. Robinson and F. Fallside, "A recurrent error propagation network speech recognition system," Comput. Speech Lang., vol. 5, no. 3, pp. 259-274, Jul. 1991.
35. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural networks," IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673-2681, Nov. 1997. (Pubitemid 127766336)
36. P. Werbos, "Backpropagation through time: what it does and how to do it," Proc. IEEE, vol. 78, no. 10, pp. 1550-1560, Oct. 1990.
37. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Comput., vol. 1, no. 2, pp. 270-280, 1989.
38. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157-166, Mar. 1994.
39. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, Nov. 1997. (Pubitemid 127462305)
40. A. Graves, "Supervised sequence labelling with recurrent neural networks, dissertation," Ph.D., Technische Univ. München, Munich, Germany, 2008.
41. S. Fernández, A. Graves, and J. Schmidhuber, Phoneme recognition in TIMIT with BLSTM-CTC, IDSIA-04-08, Tech. Rep 2008.
42. D. Correa, A. Levada, and J. Saito, "Stabilizing and improving the learning speed of 2-layered LSTM network," in Proc. CSE, , 2008, pp. 293-300.
43. G. Hinton, "Training products of experts by minimizing contrastive divergence," Neural Comput., vol. 14, no. 8, pp. 1771-1800, Aug. 2002.
44. L. Deng and D. Yu, "Deep convex net: A scalable architecture for speech pattern classification," in Proc. Interspeech, 2011, pp. 2285-2288.
45. H. Jaeger, The echo state approach to analysing and training recurrent neural networks-with an erratum note GMD Rep. 148, German Nat. Res. Center for Inform. Technolo. Tech. Rep., 2001.
46. W. Maass, T. Natschlager, and H. Markram, "Real-time computing without stable states: A new framework for neural computation based on perturbations," Neural Comput., vol. 14, no. 11, pp. 2531-2560, Nov. 2002.
47. H. Jaeger, M. Lukosevi?ius, D. Popovici, and U. Siewert, "Optimization and applications of echo state networks with leaky-integrator neurons," Neural Netw., vol. 20, no. 3, pp. 335-352, Apr. 2007. (Pubitemid 46876638)
48. C. Cortes and V. Vapnik, "Support-vector networks," Mach. Learn., vol. 20, no. 3, pp. 273-297, Sep. 1995.
49. C. M. Bishop, "Training with noise is equivalent to Tikhonov regularization," Neural Comput., vol. 7, no. 1, pp. 108-116, Jan. 1994.
50. D. W. Marquardt and R. D. Snee, "Ridge regression in practice," Amer. Statist., vol. 29, no. 1, pp. 3-20, Feb. 1975.
51. R. Penrose, "A generalized inverse for matrices," Proc. Cambridge Philosoph. Soc., vol. 51, pp. 406-413, Jul. 1955.
52. J. Garofolo, L. Lamel, W. Fisher, J. Fiscus, D. Pallett, and N. Dahlgren, The DARPA TIMIT acoustic-phonetic continuous speech corpus CD-ROM NIST, Tech. Rep., 1993.
53. K. F. Lee and H.-W. Hon, "Speaker-independent phone recognition using hidden Markov models," IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 11, pp. 1641-1648, Nov. 1989.
54. S. Davis and P. Mermelstein, "Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-28, no. 4, pp. 357-366, Aug. 1980. (Pubitemid 11464930)
55. F. Triefenbach, A. Jalalvand, B. Schrauwen, and J.-P. Martens, "Phoneme recognition with large hierarchical reservoirs," in Proc. NIPS, 2010, pp. 2307-2315.
56. F. Triefenbach and J.-P. Martens, "Can non-linear readout nodes enhance the performance of reservoir-based speech recognizers?," in Proc. ICI, 2011, pp. 262-267.
57. J. Ming and F. Smith, "Improved phone recognition using bayesian triphone models," in Proc. ICASSP, 1998, pp. 409-412.
58. L. Deng and D. Yu, "Use of differential cepstra as acoustic features in hidden trajectorymodeling for phonetic recognition," in Proc. ICASSP, 2007, pp. 445-448.
59. F. Triefenbach, K. Demuynck, and J.-P. Martens, "Improving large vocabulary continuous speech recognition by combining GMM-based and reservoir-based acoustic modeling," in Proc. SLT, 2012, pp. 107-112.
60. B. Schrauwen, M. Wardermann, D. Verstraeten, J. J. Steil, and D. Stroobandt, "Improving reservoirs using intrinsic plasticity," Neurocomputing, vol. 71, no. 7, pp. 1159-1171, Mar. 2008