Data balancing for efficient training of hybrid ANN/HMM automatic speech recognition systems

IEEE Transactions on Audio, Speech and Language Processing

Volumn 19, Issue 3, 2011, Pages 468-481

  García Moral, Ana Isabel ^a   Solera Ureña, Rubén ^a   Peláez Moreno, Carmen ^a   Díaz De María, Fernando ^a  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

Author keywords

Active learning; additive noise; ANN HMM; artificial neural networks (ANNs); hidden Markov models (HMMs); hybrid automatic speech recognition (ASR); machine learning; MLP HMM; multilayer perceptrons (MLPs); robust ASR

Indexed keywords

ACTIVE LEARNING; ANN/HMM; ARTIFICIAL NEURAL NETWORKS; AUTOMATIC SPEECH RECOGNITION; MACHINE-LEARNING; MLP/HMM; MULTILAYER PERCEPTRONS (MLPS); ROBUST ASR;

ADDITIVE NOISE; CYBERNETICS; HIDDEN MARKOV MODELS; LEARNING SYSTEMS; MULTILAYER NEURAL NETWORKS; MULTILAYERS; OBJECT RECOGNITION; PATTERN RECOGNITION SYSTEMS; PROBABILITY DISTRIBUTIONS;

SPEECH RECOGNITION;

EID: 78149247542     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2010.2050513     Document Type: Article

References (61)

1. E. Trentin and M. Gori, "A survey of hybrid ANN/HMM models for automatic speech recognition", Neurocomputing, vol. 37, no. 1-4, pp. 91-126, April 2001.
2. N. Morgan, B. Chen, Q. Zhu, and A. Stolcke, "Scaling up: Learning large-scale recognition methods from small-scale recognition tasks", in Proc. Special Workshop in Maui (SWIM), Maui, HI, 2004.
3. N. Morgan, Q. Zhu, A. Stolcke, K. Sonmez, S. Sivadas, T. Shinozaki, M. Ostendorf, P. Jain, H. Hermansky, D. Ellis, G. Doddington, B. Chen, Ö. Çetin, H. Bourlard, and M. Athineos, "Pushing the envelope-aside", IEEE Signal Process. Mag., vol. 22, no. 5, pp. 81-88, Sep. 2005.
4. Q. Zhu, A. Stolcke, B. Chen, and N. Morgan, "Using MLP features in SRI's conversational speech recognition system", in Proc. 9th Int. Conf. Speech Commun. Technol. (Interspeech'2005), Lisbon, Portugal, Sep. 2005.
5. R. Solera-Ureña, D. Martín-Iglesias, A. Gallardo- Antolín, C. Peláez-Moreno, and F. Díaz-de-María, "Robust ASR using support vector machines", Speech Commun., vol. 49, no. 4, pp. 253-267, Apr. 2007.
6. R. Solera-Ureña, J. Padrell-Sendra, D. Martín-Iglesias, A. Gallardo-Antolín, C. Peláez-Moreno, and F. Díaz-de- María, "SVMs for automatic speech recognition: A survey", in Progress in Nonlinear Speech Processing, ser. Lecture Notes in Computer Science. Berlin/Heidelberg, Germany: Springer-Verlag, May 2007, vol. 4391, pp. 190-216.
7. J. Padrell-Sendra, D. Martín-Iglesias, and F. Díaz-de- María, "Support vector machines for continuous speech recognition", in Proc. 14th Eur. Signal Process. Conf., Florence, Italy, Sep. 2006.
8. X.-Y. Liu, J. Wu, and Z.-H. Zhou, "Exploratory undersampling for class-imbalance learning", IEEE Trans. Syst., Man, Cybern., Part B: Cybern., vol. 39, no. 2, pp. 539-550, Apr. 2009.
9. G. Weiss, B. Zadrozny, and M. Saar-Tsechansky, "Guest editorial: Special issue on utility-based data mining", Data Mining Knowl. Discov., vol. 17, no. 2, pp. 129-135, Oct. 2008.
10. V. Garcia, J. Sánchez, and R. Mollineda, "An empirical study of the behavior of classifiers on imbalanced and overlapped data sets", in Progress in Pattern Recognition, Image Analysis and Applications, ser. Lecture Notes in Computer Science (LNCS). Berlin/Heidelberg, Germany: Springer-Verlag, 2007, vol. 4756, pp. 397-406.
11. L. Toth and A. Kocsor, "Training HMM/ANN hybrid speech recognizers by probabilistic sampling", in Artificial Neural Networks: Biological Inspirations-ICANN 2005, ser. Lecture Notes in Computer Science (LNCS). Berlin/Heidelberg, Germany: Springer-Verlag, 2005, vol. 3696, pp. 597-603.
12. T. Kamm and G. Meyer, "Automatic selection of transcribed training material", in Proc. IEEE Workshop Autom. Speech Recognition and Understanding 2001 (ASRU'01), Trento, Italy, Dec. 2001, pp. 417-420.
13. T. Kamm and G. Meyer, "Selective sampling of training data for speech recognition", in Proc. 2nd Int. Conf. Human Lang. Technol. Res., San Diego, CA, March 2002, pp. 20-24, Morgan Kaufmann Publisher.
14. T. Kamm, "Active learning for acoustic speech recognition modeling" Ph. D. dissertation, John Hopkins Univ., Baltimore, MD, Jan. 2004 [Online]. Available: http://www.clsp.jhu.edu/people/tkamm/papers/kamm-thesis. pdf
15. Y. Wu, R. Zhang, and A. Rudnicky, "Data selection for speech recognition", in IEEE Workshop Autom. Speech Recognition Understanding (ASRU'07), Kyoto, Japan, Dec. 2007, pp. 562-565.
16. A. Nagroski, L. Boves, and H. Steeneken, "In search of optimal data selection for training of automatic speech recognition systems", in Proc. IEEE Workshop Autom. Speech Recognition and Understanding (ASRU'03), Virgin Islands, USA, Nov.-Dec. 2003, pp. 67-72.
17. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach. Norwell, MA: Kluwer, 1994.
18. P. Pujol, S. Pol, C. Nadeu, A. Hagen, and H. Bourlard, "Comparison and combination of features in a hybrid HMM/MLP and a HMM/GMM speech recognition system", IEEE Trans. Speech Audio Process., vol. 13, no. 1, pp. 14-22, Jan. 2005.
19. H. Bourlard and N. Morgan, "Hybrid HMM/ANN systems for speech recognition: Overview and new research directions", in Adaptive Processing of Sequences and Data Structures, ser. Lecture Notes in Computer Science (LNCS). Berlin/Heidelberg, Germany: Springer-Verlag, 1998, vol. 1387, pp. 389-417.
20. D. Albesano, R. Gemello, and F. Mana, "Hybrid HMM-NN for speech recognition and prior class probabilities", in Proc. 9th Int. Conf. Neural Inf. Process. ICONIP'02, 2002, vol. 5, pp. 2391-2395.
21. B. Chen, Ö. Çetin, G. Doddington, N. Morgan, M. Ostendorf, T. Shinozaki, and Q. Zhu, "A CTS task for meaningful fast-turnaround experiments", in Proc. Rich Transcription Fall Workshop, Palisades, NY, Nov. 2004.
22. N. Morgan and H. Bourlard, "Continuous speech recognition: An introduction to the hybrid HMM/connectionist approach", IEEE Signal Process. Mag., vol. 12, no. 3, pp. 24-42, May 1995.
23. L. Tóth and A. Kocsor, "A segment-based interpretation of HMM/ANN hybrids", Comput. Speech Lang., vol. 21, no. 3, pp. 562-578, Jul. 2007.
24. R. Gemello, F. Mana, S. Scanzio, P. Laface, and R. D. Mori, "Linear hidden transformations for adaptation of hybrid ANN/HMM models", Speech Commun., vol. 49, no. 10-11, pp. 827-835, Oct. 2007.
25. J. Stadermann and G. Rigoll, "Hybrid NN/HMM acoustic modeling techniques for distributed speech recognition", Speech Commun., vol. 48, no. 8, pp. 1037-1046, Aug. 2006.
26. J. Frankel and S. King, "A hybrid ANN/DBN approach to articulatory feature recognition", in Proc. 9th Int. Conf. Speech Commun. Technol. (Interspeech'05), Lisbon, Portugal, Sep. 2005.
27. J. Neto, C. Martins, and L. Almeida, "An incremental speaker-adaptation technique for hybrid hmm-mlp recognizer", in Proc. 4th Int. Conf. Spoken Lang. Process. (ICSLP'96), Philadelphia, PA, 1996, pp. 1289-1292.
28. J. Neto, C. Martins, and L. Almeida, "Speaker-adaptation in a hybrid hmm-mlp recognizer", in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP'96), Washington, DC, May 1996, vol. 6, pp. 3382-3385.
29. S. Haykin, Neural Networks: A Comprehensive Foundation. Upper Saddle River, NJ: Prentice-Hall, 1998.
30. S. Renals, N. Morgan, H. Bourlard, M. Cohen, and H. Franco, "Connectionist probability estimators in HMM speech recognition", Speech and Audio Processing, vol. 2, no. 1, pp. 161-174, Jan. 1994.
31. A. Hagen, "Robust speech recognition based on multi-stream processing" Ph. D. dissertation, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, Dec. 2001 [Online]. Available: http://infoscience.epfl.ch/search.py?recid=32973
32. C. Raymond and G. Riccardi, "Learning with noisy supervision for spoken language understanding", in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP'08), Las Vegas, NV, Mar.-Apr. 2008, pp. 4989-4992.
33. I. Jars and F. Panaget, "Improving spoken language understanding with information retrieval and active learning methods", in IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP'08), Las Vegas, NV, Mar.-Apr. 2008, pp. 5001-5004.
34. A. Bondu, V. Lemaire, and B. Poulain, "Active learning strategies: A case study for detection of emotions in speech", in Advances in Data Mining: Theoretical Aspects and Applications, ser. Lecture Notes in Artificial Intelligence (LNAI). Berlin/Heidelberg, Germany: Springer-Verlag, 2007, vol. 4597, pp. 597-603.
35. D. Farris, C. White, and S. Khudanpur, "Sample selection for automatic language identification", in IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP'08), Las Vegas, NV, Mar.-Apr. 2008, pp. 4225-4228.
36. S. Scanzio, P. Laface, R. Gemello, and F. Mana, "Speeding-up neural network training using sentence and frame selection", in Proc. Interspeech, Jan. 2007, pp. 1725-1728 [Online]. Available: http://www.isca-speech.org/ archive/interspeech-2007/
37. S. Vijayakumar and H. Ogawa, "Improving generalization ability through active learning", IEICE Trans. Inf. Syst., vol. E82-D, no. 2, pp. 480-487, Feb. 1999.
38. G. Tur, D. Hakkani-Tür, and R. Schapire, "Combining active and semi-supervised learning for spoken language understanding", Speech Commun., vol. 45, no. 2, pp. 171-186, Feb. 2005. (Pubitemid 40220192)
39. G. Riccardi and D. Hakkani-Tür, "Active learning: Theory and applications to automatic speech recognition", IEEE Trans. Speech Audio Process., vol. 13, no. 4, pp. 504-511, Jul. 2005.
40. R. Zhang and A. Rudnicky, "A new data selection approach for semisupervised acoustic modeling", in IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP'06), Toulouse, France, May 2006, vol. 1, pp. 421-424.
41. S.-H. Liu, F.-H. Chu, S.-H. Lin, H.-S. Lee, and B. Chen, "Training data selection for improving discriminative training of acoustic models", in Proc. IEEE Workshop Autom. Speech Recognition and Understanding (ASRU'07), Kyoto, Japan, Dec. 2007, pp. 284-289.
42. H. Jiang, F. Soong, and C.-H. Lee, "A dynamic in-search data selection method with its applications to acoustic modeling and utterance verification", IEEE Trans. Speech Audio Process., vol. 13, no. 5, pp. 945-955, Sep. 2005.
43. C. Peláez-Moreno, Q. Zhu, B. Chen, and N. Morgan, "Automatic data selection for MLP-based feature extraction for ASR", in Proc. 9th Int. Conf. Speech Commun. Technol. (Interspeech'2005), Lisbon, Portugal, Sep. 2005, pp. 229-232.
44. N. Chawla, N. Japkowicz, and A. Kotcz, "Editorial: Special issue on learning from imbalanced data sets", ACM SIGKDD Explorations Newslett., vol. 6, no. 1, pp. 1-6, Jun. 2004.
45. A. Moreno, "Speechdat Spanish database for fixed telephone network", Univ. Politecnica de Catalunya, 1997, Tech. Rep..
46. G. Batista, R. Prati, and M. Monard, "A study of the behavior of several methods for balancing machine learning training data", ACM SIGKDD Explorations Newslett., vol. 6, no. 1, pp. 20-29, Jun. 2004.
47. T. Jo and N. Japkowicz, "Class imbalances versus small disjuncts", ACM SIGKDD Explorations Newslett., vol. 6, no. 1, pp. 40-49, Jun. 2004.
48. S. Kotsiantis, D. Kanellopoulos, and P. Pintelas, "Handling imbalanced datasets: A review", GESTS Int. Trans. Comput. Sci. Eng., vol. 30, no. 1, pp. 25-36, 2006.
49. A. Joshi, "Applying the wrapper approach for auto discovery of undersampling and over-sampling percentages on skewed datasets" Ph. D. dissertation, Univ. of South Florida, Tampa, FL, Nov. 2004 [Online]. Available: http://purl.fcla.edu/fcla/etd/SFE0000491
50. K. Veropoulos, C. Campbell, and N. Cristianini, "Controlling the sensitivity of support vector machines", in Proc. 16th Int. Joint Conf. Artif. Intell. (IJCAI99), Stockholm, Sweden, Aug. 1999, pp. 55-60.
51. N. Chawla, D. Cieslak, L. Hall, and A. Joshi, "Automatically countering imbalance and its empirical relationship to cost", Data Mining Knowl. Discov., vol. 17, no. 2, pp. 225-252, Oct. 2008.
52. M. Kubat and S. Matwin, "Addressing the curse of imbalanced training sets: One-sided selection", in Proc. 14th Int. Conf. Mach. Learn., 1997, pp. 179-186, Morgan Kaufmann.
53. N. Chawla, K. Bowyer, L. Hall, and W. Kegelmeyer, "Smote: Synthetic minority over-sampling technique", J. Artif. Intell. Res., vol. 16, pp. 321-357, Feb. 2002.
54. N. Chawla, A. Lazarevic, L. Hall, and K. Bowyer, "SMOTEBoost: Improving prediction of the minority class in boosting", Lecture Notes in Computer Science, vol. 2838, pp. 107-119, 2003.
55. C. Seiffert, T. Khoshgoftaar, J. Van Hulse, and A. Napolitano, "Rusboost: A hybrid approach to alleviating class imbalance", IEEE Trans. Syst., Man, Cybern. A: Syst. Humans, vol. 40, no. 1, pp. 185-197, Jan. 2010.
56. A. Estabrooks, T. Jo, and N. Japkowicz, "A multiple resampling method for learning from imbalanced data sets", Comput. Intell., vol. 20, no. 1, pp. 18-36, Feb. 2004.
57. Z.-H. Zhou and X.-Y. Liu, "Training cost-sensitive neural networks with methods addressing the class imbalance problem", IEEE Trans. Knowl. Data Eng., vol. 18, no. 1, pp. 63-77, Jan. 2006.
58. A. Varga, H. Steenneken, M. Tolimson, and D. Jones, "The NOISEX-92 study on the effect of additive noise on automatic speech recognition", Speech Research Unit, Defence Evaluation and Research Agency (DERA), Malvern, U. K., 1992, Tech. Rep..
59. S. Young, G. Evermann, M. Gales, T. Hain, and D. Kershaw, HTK-Hidden Markov Model Toolkit (Ver 3.4). Cambridge, U. K.: Cambridge Univ., 2006.
60. F. Johansen, N. Warakagoda, B. Lindberg, G. Lehtinen, Z. Ka-Ci-Ce, K. Elenius, and G. Salvi, "The COST 249 SpeechDat multilingual reference recogniser", in COST 249 MCM, Technical Annex, 1999.
61. D. Johnson, ICSI Quicknet Software Package. [Online]. Available: http://www.icsi.berkely.edu/Speech/qn.html 2007