Robust Speaker-Adaptive HMM-Based Text-to-Speech Synthesis

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 6, 2009, Pages 1208-1230

  Yamagishi, Junichi ^a   King, Simon ^a   Renals, Steve ^a   Nose, Takashi ^b   Zen, Heiga ^c   Tokuda, Keiichi ^c   Ling, Zhen Hua ^d   Toda, Tomoki ^e  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^c NAGOYA INSTITUTE OF TECHNOLOGY   (Japan)

^d UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

^e NARA INSTITUTE OF SCIENCE AND TECHNOLOGY   (Japan)

Author keywords

Average voice; HMM Speech Synthesis System; HMM based speech synthesis; HTS; speaker adaptation; speech synthesis; voice conversion

Indexed keywords

EID: 85008006694     PISSN: 15587916     EISSN: 15587924     Source Type: Journal    
DOI: 10.1109/TASL.2009.2016394     Document Type: Article

References (77)

1. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,” in Proc. EUROSPEECH-99, Sep. 1999, pp. 2374–2350.
2. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,” (in Japanese) IEICE Trans., vol. J83-D-II, no. 11, pp. 2099–2107, Nov. 2000.
3. A. Black, H. Zen, and K. Tokuda, “Statistical parametric speech synthesis,” in Proc. ICASSP 2007, Apr. 2007, pp. 1229–1232.
4. K. Tokuda, H. Zen, J. Yamagishi, T. Masuko, S. Sako, A. Black, and T. Nose, The HMM-Based Speech Synthesis System (HTS) Version 2.0.1 [Online]. Available: http://www.hts.sp.nitech.ac.jp/
5. K. Tokuda, T. Kobayashi, and S. Imai, “Speech parameter generation from HMM using dynamic features,” in Proc. ICASSP-95, May 1995, pp. 660–663.
6. K. Tokuda, T. Masuko, T. Kobayashi, and S. Imai “An algorithm for speech parameter generation from HMM using dynamic features,” (in Japanese) J. Acoust. Soc. Japan, vol. 53, no. 3, pp. 192–200, Mar. 1997.
7. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai, “Speech synthesis using HMMs with dynamic features,” in Proc. ICASSP-96, May 1996, pp. 389–392.
8. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai “HMM-based speech synthesis using dynamic features,” (in Japanese) IEICE Trans., vol. J79-D-II, no. 12, pp. 2184–2190, Dec. 1996.
9. K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorigthms for HMM-based speech synthesis,” in Proc. ICASSP 2000, Jun. 2000, pp. 1315–1318.
10. K. Tokuda, T. Masuko, N. Miyazaki, and T. Kobayashi, “Multi-space probability distribution HMM,” IEICE Trans. Inf. Syst., vol. E85-D, no. 3, pp. 455–464, Mar. 2002.
11. H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “A hidden semi-Markov model-based speech synthesis system,” IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 825–834, May 2007.
12. J. Ferguson, “Variable duration models for speech,” in Proc. Symp. Applicat. Hidden Markov Models to Text and Speech, 1980, pp. 143–179.
13. M. Russell and R. Moore, “Explicit modelling of state occupancy in hidden Markov models for automatic speech recognition,” in Proc. ICASSP-85, Mar. 1985, pp. 5–8.
14. S. Levinson, “Continuously variable duration hidden Markov models for automatic speech recognition,” Comput. Speech Lang., vol. 1, no. 1, pp. 29–45, 1986.
15. A. McCree and T. Barnwell, III “A mixed excitation LPC vocoder model for low bit rate speech coding,” IEEE Trans. Speech Audio Process., vol. 3, no. 4, pp. 242–250, Jul. 1995.
16. H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system STRAIGHT,” in Proc. 2nd MAVEBA, Sep. 2001, pp. 13–15.
17. D. W. Griffin and J. S. Lim “Multiband excitation vocoder,” IEEE Trans. Acoust., Speech, Signal Audio Process., vol. 36, no. 8, pp. 1223–1235, Aug. 1988.
18. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Mixed excitation for HMM-based speech synthesis,” in Proc. Eurospeech'01, Sep. 2001, 22632266.
19. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Incorporation of mixed excitation model and postfilter into HMM-based text-to-speech synthesis,” (in Japanese) IEICE Trans., vol. J87-D-II, no. 8, pp. 1565–1571, Aug. 2004.
20. H. Zen, T. Toda, M. Nakamura, and K. Tokuda, “Details of Nitech HMM-based speech synthesis system for the Blizzard Challenge 2005,” IEICE Trans. Inf. Syst., vol. E90-D, no. 1, pp. 325–333, Jan. 2007.
21. A.-H. Ossama, A. S. Mahdy, and R. Mohsen, “Improving Arabic HMM based speech synthesis quality,” in Proc. Interspeech 2006, Sep. 2006, pp. 1332–1335.
22. T. Toda and K. Tokuda, “A speech parameter generation algorithm considering global variance for HMM-based speech synthesis,” IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 816–824, May 2007.
23. H. Zen, T. Toda, and K. Tokuda, “The Nitech-NAIST HMM-based speech synthesis system for the Blizzard Challenge 2006,” IEICE Trans. Inf. Syst., vol. E91-D, no. 6, pp. 1764–1773, Jun. 2008.
24. Z.-H. Ling, Y.-J. Wu, Y.-P. Wang, L. Qin, and R.-H. Wang, “USTC system for Blizzard Challenge 2006 an improved HMM-based speech synthesis method,” in Proc. Blizzard Challenge 2006, Sep. 2006.
25. J. Yamagishi, H. Zen, T. Toda, and K. Tokuda, “Speaker-independent HMM-based speech synthesis system—HTS-2007 system for the Blizzard Challenge 2007,” in Proc. BLZ3-2007 (in Proc. SSW6), Aug. 2007 [Online]. Available: http://festvox.org/blizzard/bc2007/blizzard_2007/blz3_008.html, paper 003.
26. A. Black and K. Tokuda, “The Blizzard Challenge—2005: Evaluating corpus-based speech synthesis on common datasets,” in Proc. Eurospeech 2005, Sep. 2005, pp. 77–80.
27. C. Bennett and A. Black, “The blizzard challenge 2006,” in Proc. Blizzard Challenge 2006, Sep. 2006 [Online]. Available: http://festvox.org/blizzard/bc2006/eval_blizzard2006.pdf
28. M. Fraser and S. King, “The Blizzard Challenge 2007,” in Proc. BLZ3-2007 (in Proc. SSW6), Aug. 2007 [Online]. Available: http://festvox. org/blizzard/bc2007/blizzard_2007/blz3_001.html, paper 001.
29. H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds,” Speech Commun., vol. 27, pp. 187–207, 1999.
30. M. Gales “Semi-tied covariance matrices for hidden Markov models,” IEEE Trans. Speech Audio Process., vol. 7, pp. 272–281, Mar. 1999.
31. R. Gopinath, “Maximum likelihood modeling with Gaussian distributions for classfication,” in Proc. ICASSP-98, May 1998, pp. 661–664.
32. J. Yamagishi and T. Kobayashi, “Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training,” IEICE Trans. Inf. Syst., vol. E90-D, no. 2, pp. 533–543, Feb. 2007.
33. J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Isogai, “Analysis of speaker adaptation algorithms for HMM-based speech synthesis and a constrained SMAPLR adaptation algorithm,” IEEE Trans. Speech, Audio, Lang. Process., vol. 17, no. 1, pp. 66–83, Jan. 2009, 2007.
34. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, “Speaker adaptation for HMM-based speech synthesis system using MLLR,” in Proc. 3rd ESCA/COCOSDA Workshop Speech Synth., Nov. 1998, pp. 273–276.
35. C. Leggetter and P. Woodland “Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models,” Comput. Speech Lang., vol. 9, no. 2, pp. 171–185, 1995.
36. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, “Adaptation of pitch and spectrum for HMM-based speech synthesis using MLLR,” in Proc. ICASSP-01, May 2001, pp. 805–808.
37. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, “Speaker adaptation of pitch and spectrum for HMM-based speech synthesis,” (in Japanese) IEICE Trans., vol. J85-D-II, no. 4, pp. 545–553, Apr. 2002.
38. T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, “A compact model for speaker-adaptive training,” in Proc. ICSLP-96, Oct. 1996, pp. 1137–1140.
39. J. Yamagishi, M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, “A training method of average voice model for HMM-based speech synthesis,” IEICE Trans. Fundamentals, vol. E86-A, no. 8, pp. 1956–1963, Aug. 2003.
40. M. Tachibana, J. Yamagishi, T. Masuko, and T. Kobayashi, “A style adaptation technique for speech synthesis using HSMM and suprasegmental features,” IEICE Trans. Inf. Syst., vol. E89-D, no. 3, pp. 1092–1099, Mar. 2006.
41. L. Qin, Z. Ling, Y. Wu, B. Zhang, and R. Wang, “HMM-based emotional speech synthesis using average emotion model,” in Proc. ISCSLP-06 (Springer LNAI Book), Dec. 2006, pp. 233–240.
42. J. Latorre, K. Iwano, and S. Furui, “New approach to the polyglot speech generation by means of an HMM-based speaker adaptable synthesizer,” Speech Commun., vol. 48, no. 10, pp. 1227–1242, 2006.
43. V. Digalakis and L. Neumeyer “Speaker adaptation using combined transformation and Bayesian methods,” IEEE Trans. Speech Audio Process., vol. 4, pp. 294–300, Jul. 1996.
44. K. Shinoda and C. Lee, “A structural Bayes approach to speaker adaptation,” IEEE Trans. Speech Audio Process., vol. 9, no. 3, pp. 276–287, Mar. 2001.
45. D. Arifianto, T. Tanaka, T. Masuko, and T. Kobayashi, “Robust F0 estimation of speech signal using harmonicity measure based on instantaneous frequency,” IEICE Trans. Inf. Syst., vol. E87-D, no. 12, pp. 2812–2820, Dec. 2004.
46. H. Kawahara, H. Katayose, A. Cheveigne, and R. Patterson, “Fixed point analysis of frequency to instantaneous frequency mapping for accurate estimation of F0 and periodicity,” in Proc. Eurospeech 1999, Sep. 1999, pp. 2781–2784.
47. D. Talkin, “A robust algorithm for pitch tracking (RAPT),” in Speech Coding and Synthesis, W. Kleijn and K. Paliwal, Eds. New York: Elsevier, 1995, pp. 495–518.
48. ESPS Programs Version 5.0. Entropic Research Laboratory Inc., 1993.
49. K. Tokuda, H. Zen, and A. Black, “An HMM-based speech synthesis system applied to English,” in Proc. IEEE Speech Synth. Workshop, Sep. 2002, pp. 227–230.
50. H. Kubozono, “Mora and syllable,” in The handbook of Japanese Linguistics, N. Tsujimura, Ed. Chichester, U.K.: Blackwell, 1995, pp. 31–61.
51. M. Gales “Maximum likelihood linear transformations for HMM-based speech recognition,” Comput. Speech Lang., vol. 12, no. 2, pp. 75–98, 1998.
52. V. Digalakis, D. Rtischev, and L. Neumeyer “Speaker adaptation using constrained reestimation of Gaussian mixtures,” IEEE Trans. Speech Audio Process., vol. 3, no. 5, pp. 357–366, Sep. 1995.
53. A. Dempster, N. Laird, and D. Rubin “Maximum likelihood from incomplete data via the EM algorithm,” J. R. Statist. Soc., Series B, vol. 39, no. 1, pp. 1–38, 1977.
54. J. Yamagishi, K. Onishi, T. Masuko, and T. Kobayashi, “Acoustic modeling of speaking styles and emotional expressions in HMM-based speech synthesis,” IEICE Trans. Inf. Syst., vol. E88-D, no. 3, pp. 503–509, Mar. 2005.
55. K. Shinoda and T. Watanabe, “MDL-based context-dependent subword modeling for speech recognition,” J. Acoust. Soc. Japan (E), vol. 21, pp. 79–86, Mar. 2000.
56. E. Moulines and F. Charpentier “Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones,” Speech Commun., vol. 9, no. 5–6, pp. 453–468, 1990.
57. Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed excitation,” in Proc. Interspeech 2006, Sep. 2006, pp. 2266–2269.
58. O. Watts, J. Yamagishi, K. Berkling, and S. King, “HMM-based synthesis of child speech,” in Proc. 1st Workshop Child, Comput., Interaction (ICMI'08 Post-Conf. Workshop), Oct. 2008.
59. J. Yamagishi, T. Kobayashi, M. Tachibana, K. Ogata, and Y. Nakano, “Model adaptation approach to speech synthesis with diverse voices and styles,” in Proc. ICASSP-07, Apr. 2007, pp. 1233–1236.
60. J. Ni, T. Hirai, H. Kawai, T. Toda, K. Tokuda, M. Tsuzaki, S. Sakai, R. Maia, and S. Nakamura, “ATRECSS—ATR English speech corpus for speech synthesis,” in Proc. BLZ3-2007 (in Proc. SSW6), Aug. 2007.
61. A. Black, P. Taylor, and R. Caley, The Festival Speech Synthesis System. Edinburgh, U.K.: Univ. of Edinburgh, 1999.
62. S.-Z. Yu and H. Kobayashi, “An efficient forward-backward algorithm for an explicit-duration hidden Markov model,” IEEE Signal Process. Lett., vol. 10, no. 1, pp. 11–14, Jan. 2003.
63. C. Mitchell, M. Harper, and L. Jamieson “On the complexity of explicit duration HMM's,” IEEE Trans. Speech Audio Process., vol. 3, no. 3, pp. 213–217, May 1995.
64. H. Zen, K. Tokuda, T. Masuko, T. Yoshimura, T. Kobayashi, and T. Kitamura, “State duration modeling for HMM-based speech synthesis,” IEICE Trans. Inf. Syst., vol. E90-D, no. 3, pp. 692–693, Mar. 2007.
65. H. Zen, Implementing an HSMM-based speech synthesis system using an efficient forward-backward algorithm Nagoya Inst. of Technol., TR-SP-0001, Dec. 2007, Tech. Rep.
66. R. Clark, M. Podsiadlo, M. Fraser, C. Mayo, and S. King, “Statistical analysis of the Blizzard Challenge 2007 listening test results,” in Proc. BLZ3-2007 (in Proc. SSW6), Aug. 2007 [Online]. Available: http://festvox.org/blizzard/bc2007/blizzard_2007/blz3_003.html, paper 003.
67. Z.-H. Ling, L. Qin, H. Lu, Y. Gao, L.-R. Dai, R.-H. Wang, Y. Jiang, Z.-W. Zhao, J.-H.Y.J. Chen, and G.-P. Hu, “The USTC and iFlytek speech synthesis systems for Blizzard Challenge 2007,” in Proc. BLZ3-2007 (in Proc. SSW6), Aug. 2007 [Online]. Available: http://festvox.org/blizzard/bc2007/blizzard_2007/blz3_017.html, paper 017.
68. K. Richmond, V. Strom, R. Clark, J. Yamagishi, and S. Fitt, “Festival Multisyn voices for the 2007 Blizzard Challenge,” in Proc. BLZ3-2007 (in Proc. SSW6), Aug. 2007 [Online]. Available: http://festvox. org/blizzard/bc2007/blizzard_2007/blz3_006.html, paper 006.
69. A. Hunt and A. Black, “Unit selection in a concatenative speech synthesis system using a large speech database,” in Proc. ICASSP-96, May 1996, pp. 373–376.
70. Z.-H. Ling and R.-H. Wang, “HMM-based unit selection using frame sized speech segments,” in Proc. Interspeech 2006, Sep. 2006, pp. 2034–2037.
71. Z.-H. Ling and R.-H. Wang, “HMM-based hierarchical unit selection combining Kullback-Leibler divergence with likelihood criterion,” in Proc. ICASSP-07, Apr. 2007, pp. 1245–1248.
72. R. A. J. Clark, K. Richmond, and S. King, “Multisyn: Open-domain unit selection for the Festival speech synthesis system,” Speech Commun., vol. 49, no. 4, pp. 317–330, 2007.
73. Y. Wu and R.-H. Wang, “Minimum generation error training for HMM-based speech synthesis,” in Proc. ICASSP-06, May 2006, pp. 89–92 [Online]. Available: http://festvox.org/blizzard/bc2008/hts_Blizzard2008.pdf
74. C. Benoit, M. Grice, and V. Hazan, “The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using semantically unpredictable sentences,” Speech Commun., vol. 18, no. 4, pp. 381–392, 1996.
75. S. Fitt and S. Isard, “Synthesis of regional English using a keyword lexicon,” in Proc. Eurospeech 1999, Sep. 1999, vol. 2, pp. 823–826.
76. J. Yamagishi, H. Zen, Y.-J. Wu, T. Toda, and K. Tokuda, “The HTS-2008 system: Yet another evaluation of the speaker-adaptive HMM-based speech synthesis system in the 2008 Blizzard Challenge,” in Proc. Blizzard Challenge 2008, Sep. 2008.
77. M. Aylett and J. Yamagishi, “Combining statistical parametric speech synthesis and unit-selection for automatic voice cloning,” in Proc. LangTech 2008, Feb. 2008 [Online]. Available: http://www. langtech.it/en/poster/03_AYLETT.pdf