Measuring the gap between HMM-based ASR and TTS

IEEE Journal on Selected Topics in Signal Processing

Volumn 4, Issue 6, 2010, Pages 1046-1058

  Dines, John ^a   Yamagishi, Junichi ^b   King, Simon ^b  

^a IDIAP RESEARCH INSTITUTE   (Switzerland)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Speech recognition; speech synthesis; unified models

Indexed keywords

ACOUSTIC FEATURES; AUTOMATIC SPEECH RECOGNITION; EUROPEAN PROJECT; SPEAKER ADAPTATION; SPEECH-TO-SPEECH TRANSLATION; STATISTICAL MODELING; STATISTICAL MODELS; SYSTEM DESIGN; TEXT TO SPEECH SYNTHESIS; TTS SYSTEMS; UNIFIED MODEL; UNIFIED MODELING;

CHARACTER RECOGNITION; HIDDEN MARKOV MODELS; SPEECH SYNTHESIS; TELEPHONE SYSTEMS;

SPEECH RECOGNITION;

EID: 77953728395     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2010.2079315     Document Type: Article

References (75)

1. M. Ostendorf and I. Bulyko, "The impact of speech recognition on speech synthesis," in Proc. IEEE Workshop Speech Synth., Santa Monica, CA, Sep. 2002, pp. 99-106.
2. M. Gales and S. Young, "The application of hidden Markov models in speech recognition," Foundat. Trends Signal Process., vol. 1, no. 3, pp. 195-304, 2007.
3. H. Zen, K. Tokuda, and A. W. Black, "Statistical parametric speech synthesis," Speech Commun., 2009, 10.1016/j.specom.2009.04.004.
4. S. King, K. Tokuda, H. Zen, and J. Yamagishi, "Unsupervised adaptation for HMM-based speech synthesis," in Proc. Interspeech'08, Sep. 2008, pp. 1869-1872.
5. M. Gibson, "Two-pass decision tree construction for unsupervised adaptation of HMM-based synthesis models," in Proc. Interspeech, Brighton, U.K., Sep. 2009, pp. 1791-1794.
6. H. Liang, J. Dines, and L. Saheer, "A comparison of supervised and unsupervised cross-lingual speaker adaptation approaches for HMM-based speech synthesis," in Proc. ICASSP, Dallas, TX, 2010, pp. 4598-4601.
7. J. Lafferty, A. McCallum, and F. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data," in Proc. ICML, Williamstown, MA, 2001, pp. 282-289.
8. P. Brown, "The Acoustic-Modeling Problem in Automatic Speech Recognition," Ph.D. dissertation, Carneggie-Mellon Univ., Pittsburgh, PA, 1987.
9. D. Povey and P. C. Woodland, "Minimum phone error and I-smoothing for improved discriminative training," in Proc. ICASSP, Orlando, FL, 2002, pp. 105-108.
10. Y.-J. Wu and R.-H. Wang, "Minimum generation error training for HMM-based speech synthesis," in Proc. ICASSP, Toulouse, France, 2006, pp. 89-92.
11. L. R. Rabiner, "A tutorial on hidden Markov models and selected appi-cations in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
12. A. Falaschi, M. Giustiniani, and M. Verola, "A hidden Markov model approach to speech synthesis," in Proc. Eurospeech, Paris, France, 1989, pp. 187-190.
13. S. Furui, "Cepstral analysis technique for automatic speaker verification," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-29, no. 2, pp. 254-272, Apr. 1981.
14. S.-Z. Yu, "Hidden semi-Markov models," Artificial Intell., vol. 174, no. 2, pp. 215-243, Feb. 2009.
15. K. Tokuda, H. Zen, and T. Kitamura, "Trajectory modeling based on HMMs with explicit relationship between static and dynamic features," in Proc. Eurospeech, Geneva, Switzerland, 2003, pp. 865-868.
16. L. Deng, "A generalised hidden Markov model with state-conditioned trend functions of time for the speech signal," Signal Process., vol. 27, pp. 65-78, Apr. 1992.
17. J. Dines,S. Sridharan, andM. Moody, "Trainable speech synthesis with trended hidden Markov models," in Proc. ICASSP, Salt Lake City, UT, 2001, pp. 833-836.
18. C. Wellekens, "Explicit time correlation in hidden Markov models for speech recognition," in Proc. ICASSP, Dallas, TX, 1987, vol. 12, pp. 384-386.
19. M. Shannon and W. Byrne, "Autoregressive HMMs for speech synthesis," in Proc. Interspeech, Brighton, U.K., 2009.
20. L. Deng and J. Ma, "A statistical coarticulatory model for the hidden vocal-tract-resonance dynamics," in Proc. Eurospeech, Budapest, Hungary, 1999, pp. 1499-1502.
21. B. Mesot and D. Barber, "Switching linear dynamical systems for noise robust speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 6, pp. 1850-1858, Aug. 2007.
22. M. Shannon and W. Byrne, "Autoregressive clustering for HMM speech synthesis," in Proc. Interspeech, Makuhari, Japan, 2010.
23. Y. Bengio, Learning Deep Architectures for AI Univ. de Montréal, Montreal, QC, Canada, Tech. Rep. 1312, 2007.
24. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, pp. 1527-1554, 2006.
25. A.-R. Mohamed, G. Dahl, and G. Hinton, "Deep belief networks for phone recognition," in Proc. NIPS Workshop Deep Learn. Speech Recogn. Rel. Applicat., Whistler, Canada, 2009.
26. Y. Bengio, O. Delalleau, and C. Simard, Decision trees do not generalize to new variations Univ. de Montréal, Montreal, QC, Canada, Tech. Rep. 1304, 2006.
27. Y. Nankaku, K. Nakamura, H. Zen, T. Toda, and K. Tokuda, "Acoustic modeling with contextual additive structure for HMM-based speech recognition," in Proc. ICASSP, Las Vegas, NV, 2008, pp. 4469-4472.
28. S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book, 3rd ed. Cambridge, U.K.: Cambridge Univ. Eng. Dept., Dec. 2006.
29. K. Tokuda, H. Zen, J. Yamagishi, T. Masuko, S. Sako, A. Black, and T. Nose, The HMM-Based Speech Synthesis System (HTS). [Online]. Available: http://hts.sp.nitech.ac.jp/
30. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Ki-tamura, "Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis," in Proc. Eurospeech, 1999, pp. 2347-2350.
31. K. Tokuda, H. Zen, and A. W. Black, "HMM-based approach to multilingual speech synthesis," in Text to Speech Synthesis: New Paradigms and Advances, S. Narayanan andA. Alwan, Eds. Upper Saddle River, NJ: Prentice-Hall, 2004.
32. J. Yamagishi, T. Nose, H. Zen, Z.-H. Ling,T. Toda,K.Tokuda, S. King, and S. Renals, "A robust speaker-adaptive HMM-based text-to-speech synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 6, pp. 1208-1230, Aug. 2009.
33. Y. Qian, F. Soong, Y. Chen, and M. Chu, "An HMM-based Mandarin Chinese text-to-speech system," in Proc. ISCSLP'06, Dec. 2006, pp. 223-232.
34. T. Hain, "Hidden model sequence models for automatic speech recognition," Ph.D. dissertation, Cambridge Univ., Cambridge, U.K., 2001.
35. S. Dupont, H. Bourlard, O. Deroo, V. Fontaine, and J.-M. Boite, "Hybrid HMM-ANN systems for training independent tasks: Experiments on phonebook and related improvements," in Proc. ICASSP, Munich, Germany, Apr. 1997, pp. 1767-1770.
36. H. Hermansky, "Perceptual linear predictive (PLP) analysis of speech," J. Acoust. Soc. Amer., vol. 87, no. 4, pp. 1738-1752, 1990.
37. K. Koishida, G. Hirabayashi, K. Tokuda, and T. Kobayashi, "Mel-generalized cepstral analysis\A unified approach to speech spectral estimation," in Proc. ICSLP, Yokohama, Japan, Sep. 1994, vol. 3, pp. 1043-1046.
38. H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Commun., vol. 27, pp. 187-207, 1999.
39. G. Garau and S. Renals, "Combining spectral representations for large vocabulary continuous speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 16, no. 3, pp. 508-518, Mar. 2008.
40. J. Yamagishi and T. Kobayashi, "Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training," IEICE Trans. Inf. Syst, vol. E90-D, no. 2, pp. 533-543, Feb. 2007.
41. T. Irino, Y. Minami, T. Nakatani, M. Tsuzaki, and H. Tagawa, "Evaluation of a speech recognition/generation method based on HMM and STRAIGHT," in Proc. ICSLP, Denver, CO, 2002, pp. 2545-2548.
42. J. J. Odell, "The use of context in large vocabulary continuous speech recognition," Ph.D. dissertation, Queens College, Univ. of Cambridge, Cambridge, U.K., 1995.
43. K. Shinoda and T. Watanabe, "Acoustic modeling based on the MDL criterion for speech recognition," in Proc. Eurospeech, Rhodes, Greece, 1997, vol. 1, pp. 99-102.
44. K. Prahallad, A. W. Black, and R. Mosur, "Sub-phonetic modeling for capturing pronunciation variations for conversational speech synthesis," in Proc. ICASSP, Toulouse, France, 2006, pp. 853-856.
45. K. Shinoda and T.Watanabe, "MDL-based context-dependent subword modeling for speech recognition," J. Acoust. Soc. Japan (E), vol. 21, pp. 79-86, Mar. 2000.
46. J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Isogai, "Analysis of speaker adaptation algorithms for HMM-based speech synthesis and a constrained SMAPLR adaptation algorithm," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 1, pp. 66-83, Jan. 2009.
47. M. Gales, "Maximum likelihood linear transformations for HMM-based speech recognition," Comput. Speech Lang., vol. 12, no. 2, pp. 75-98, 1998.
48. J. Gauvain and C. Lee, "Maximum a posteriori estimation for multi-variate Gaussian mixture observations of Markov chains," IEEE Trans. Speech Audio Process., vol. 2, pp. 291-298, Apr. 1994.
49. V. Digalakis and L. Neumeyer, "Speaker adaptation using combined transformation and Bayesian methods," IEEE Trans. Speech Audio Process., vol. 4, no. 4, pp. 294-300, Jul. 1996.
50. C. Leggetter and P. Woodland, "Flexible speaker adaptation using maximum likelihood linear regression," in Proc. ARPA Spoken Lang. Technol. Workshop, 1995, pp. 104-109, Morgan Kaufmann.
51. O. Siohan, T. Myrvoll, and C.-H. Lee, "Structural maximum a posteriori linear regression for fast hmm adaptation," Computer, Speech and Language, vol. 16, no. 1, pp. 5-24, January 2002.
52. Y. Nakano, M. Tachibana, J. Yamagishi, and T. Kobayashi, "Constrained structural maximum a posteriori linear regression for average-voice-based speech synthesis," in Proc. ICSLP'06, Sep. 2006, pp. 2286-2289.
53. T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, "A compact model for speaker-adaptive training," in Proc. ICSLP'96, Oct. 1996, pp. 1137-1140.
54. K. Oura, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda, "Hidden semi-Markov model based speech recognition system using weighted finite-state transducer," in Proc. ICASSP'06, Toulouse, France, May 2006, pp. 33-36.
55. J. Dines, L. Saheer, and H. Liang, "Speech recognition with speech synthesis models by marginalising over decision tree leaves," in Proc. Interspeech, Brighton, U.K., Sep. 2009, pp. 1395-1398.
56. J. Yamagishi, H. Zen, Y.-J. Wu, T. Toda, and K. Tokuda, "The HTS-2008 system: Yet another evaluation of the speaker-adaptive HMM-based speech synthesis system in the 2008 Blizzard Challenge," in Proc. Blizzard Challenge Workshop, Sep. 2008.
57. H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "A hidden semi-Markov model-based speech synthesis system," IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 825-834, May 2007.
58. K. Tokuda, T. K. T. Masuko, T. Kobayashi, and T. Kitamura, "Speech parameter generation algorithms for HMM-based speech synthesis," in Proc. ICASSP'00, Istanbul, Turkey, 2000, pp. 1315-1318.
59. T. Toda and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 816-824, May 2007.
60. D. Pallet, "DARPA February 1992 pilot corpus CSR "dry run" benchmark test results," in Proc. Workshop Speech and Natural Language, Harriman, NY, Feb. 1992, pp. 382-386.
61. J. Yamagishi, B. Usabaev, S. King, O. Watts, J. Dines, J. Tian, R. Hu, K. Oura, K. Tokuda, R. Karhila, and M. Kurimo, "Thousands of voices for HMM-based speech synthesis," in Proc. Interspeech, Brighton, U.K., Sep. 2009, pp. 420-423.
62. J. Yamagishi, B. Usabaev, S. King, O. Watts, J. Dines, J. Tian, R. Hu, K. Oura, K. Tokuda, R. Karhila, and M. Kurimo, "Thousands of voices for HMM-based speech synthesis-analysis and application of TTS systems built on various ASR corpora," IEEE Trans. Audio, Speech, Lang. Process., vol. 18, no. 5, pp. 984-1004, Jul. 2010.
63. M. Bisani and H. Ney, "Bootstrap estimates for confidence intervals in ASR performance evaluation," in Proc. ICASSP'94, Montreal, QC, Canada, May 1994, vol. 1, pp. 409-412.
64. A. Gray, Jr. and J. Markel, "Distance measures for speech processing," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-24, no. 5, pp. 380-391, Oct. 1976.
65. T. P. Barnwell, III, "Correlation analysis of subjective and objective measures for speech quality," in Proc. ICASSP'80, 1980, pp. 706-709.
66. S. Davis and P. Mermelstein, "Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-28, no. 4, pp. 357-366, Aug. 1980.
67. T. Kitamura, S. Imai, C. Furuichi, and T. Kobayashi, "Speech analysis-synthesis system and quality of synthesized speech using mel-cep-strum," (in Japanese)Electron. Commun. Japan (Part I: Commun.), vol. 69, no. 10, pp. 47-54, 1986.
68. T. Fukada, K. Tokuda, and S. Imai, "An adaptive algorithm for mel-cepstral analysis of speech," in Proc. ICASSP'92, San Francisco, CA, 1992, pp. 137-140.
69. R. Kubichek, "Mel-cepstral distance measure for objective speech quality assessment," in Proc. IEEE Pacific Rim Conf. Commun., Comput., Signal Process., May 1993, vol. 1, pp. 125-128.
70. T. Toda, A. Black, and K. Tokuda, "Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov. 2007.
71. V.Karaiskos,S. King, R. A. J. Clark, and C. Mayo, "The Blizzard Challenge 2008," in Proc. Blizzard Challenge Workshop, Brisbane, Australia, Sep. 2008.
72. The CMU Pronouncing Dictionary. [Online]. Available: http://www. speech.cs.cmu.edu/cgi-bin/cmudict
73. S. Fitt and S. Isard, "Synthesis of regional English using a keyword lexicon," in Proc. Eurospeech, Sep. 1999, vol. 2, pp. 823-826.
74. D. A. Reynolds, "Experimental evaluation of features for robust speaker identification," IEEE Trans. Speech Audio Process., vol. 2, no. 4, pp. 639-643, Oct. 1994.
75. J. Dines, J. Yamagishi, and S. King, "Measuring the gap between HMM-based ASR and TTS," in Proc. Interspeech, Brighton, U.K., Sep. 2009, pp. 1391-1394.