Analysis of speaker adaptation algorithms for HMM-based speech synthesis and a constrained SMAPLR adaptation algorithm

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 1, 2009, Pages 66-83

  Yamagishi, Junichi ^a   Kobayashi, Takao ^b   Nakano, Yuji ^b   Ogata, Katsumi ^b   Isogai, Juri ^b  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

Average voice; Hidden Markov model (HMM) based speech synthesis; Speaker adaptation; Speech synthesis; Voice conversion

Indexed keywords

ADAPTATION ALGORITHMS; AVERAGE VOICE; COVARIANCE MATRICES; HIDDEN MARKOV MODEL (HMM)-BASED SPEECH SYNTHESIS; HMM-BASED SPEECH SYNTHESIS; INDEPENDENT MODEL; LINEAR REGRESSION ALGORITHMS; MAP ADAPTATION; MAXIMUM A POSTERIORI; MEAN VECTOR; MODEL CONSTRUCTION; OBJECTIVE EVALUATION; PIECEWISE LINEAR REGRESSION; ROBUST ESTIMATION; SIMULTANEOUS USE; SPEAKER ADAPTATION; SPEECH SYNTHESIS SYSTEM; TRAINING DATA; TRANSFORM FUNCTION; VOICE CONVERSION;

COVARIANCE MATRIX; ESTIMATION; HIDDEN MARKOV MODELS; LINEAR REGRESSION; MATHEMATICAL TRANSFORMATIONS; MOBILE TELECOMMUNICATION SYSTEMS; PIECEWISE LINEAR TECHNIQUES; SPEECH RECOGNITION; SPEECH SYNTHESIS;

ALGORITHMS;

EID: 67650854725     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2006647     Document Type: Article

References (67)

1. A. Black and N. Cambpbell, "Optimising selection of units from speech database for concatenative synthesis," in Proc. EUROSPEECH'95, Sep. 1995, pp. 581-584.
2. A. Hunt and A. Black, "Unit selection in a concatenative speech synthesis system using a large speech database," in Proc. ICASSP'96, May 1996, pp. 373-376.
3. R. Donovan and P. Woodland, "A hidden Markov-model-based trainable speech synthesizer," Comput. Speech Lang., vol.13, no.3, pp. 223-241, 1999.
4. A. Syrdal, C. Wightman, A. Conkie, Y. Stylianou, M. Beutnagel, J. Schroeter, V. Storm, K. Lee, and M. Makashay, "Corpus-based techniques in the AT&T NEXTGEN synthesis system," in Proc. ICSLP'00, Oct. 2000, pp. 411-416.
5. A. Black, "Unit selection and emotional speech," in Proc. Eurospeech' 03, Sep. 2003, pp. 1649-1652.
6. K. Tokuda, T. Kobayashi, and S. Imai, "Speech parameter generation from HMM using dynamic features," in Proc. ICASSP'95, May 1995, pp. 660-663.
7. K. Tokuda, T. Masuko, T. Kobayashi, and S. Imai, "An algorithm for speech parameter generation from HMM using dynamic features," (in Japanese) J. Acoust. Soc. Jpn., vol.53, no.3, pp. 192-200, Mar. 1997.
8. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai, "Speech synthesis using HMMs with dynamic features," in Proc. ICASSP'96, May 1996, pp. 389-392.
9. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai, "HMM-based speech synthesis using dynamic features," (in Japanese) IEICE Trans., vol.J79-D-II, no.12, pp. 2184-2190, Dec. 1996.
10. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis," in Proc. Eurospeech'99, Sep. 1999, pp. 2374-12350
11. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Simultaneous modeling of spectrum, pitch and duration in HMMbased speech synthesis," (in Japanese) IEICE Trans., vol.J83-D-II, no.11, pp. 2099-2107, Nov. 2000.
12. K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, "Speech parameter generation algorithms for HMM-based speech synthesis," in Proc. ICASSP'00, Jun. 2000, pp. 1315-1318. (Pubitemid 30956411)
13. J.Yamagishi, K. Onishi, T. Masuko, and T.Kobayashi, "Acoustic modeling of speaking styles and emotional expressions in HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol.E88-D, no.3, pp. 503-509, Mar. 2005.
14. M. Tachibana, J. Yamagishi, T. Masuko, and T. Kobayashi, "A style adaptation technique for speech synthesis using HSMM and suprasegmental features," IEICE Trans. Inf. Syst., vol.E89-D, no.3, pp. 1092-1099, Mar. 2006.
15. M. Tachibana, J. Yamagishi, T. Masuko, and T. Kobayashi, "Speech synthesis with various emotional expressions and speaking styles by style interpolation and morphing," IEICE Trans. Inf. Syst., vol.E88-D, no.11, pp. 2484-2491, Nov. 2005.
16. T. Nose, J. Yamagishi, and T. Kobayashi, "A style control technique for HMM-based expressive speech synthesis," IEICE Trans. Inf. Syst., vol.E90-D, no.9, pp. 1406-1413, Sep. 2007.
17. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai, "Voice characteristics conversion for HMM-based speech synthesis system," in Proc. ICASSP'97, Apr. 1997, pp. 1611-1614.
18. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, "Adaptation of pitch and spectrum for HMM-based speech synthesis using MLLR," in Proc. ICASSP'01, May 2001, pp. 805-808.
19. J. Yamagishi, M. Tamura, T. Masuko, K. Tokuda, and T.Kobayashi, "A training method of average voice model for HMM-based speech synthesis," IEICE Trans. Fundamentals, vol.E86-A, no.8, pp. 1956-1963, Aug. 2003.
20. J.Yamagishi and T.Kobayashi, "Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training," IEICE Trans. Inf. Syst., vol.E90-D, no.2, pp. 533-543, Feb. 2007.
21. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, "Speaker adaptation for HMM-based speech synthesis system using MLLR," in Proc. 3rd ESCA/COCOSDA Workshop Speech Synth., Nov. 1998, pp. 273-276.
22. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai, "Voice characteristics conversion for HMM-based speech synthesis system using MAP-VFS," (in Japanese) IEICE Trans., vol.J83-D-II, no.12, pp. 2509-2516, Dec. 2000.
23. C. Leggetter and P.Woodland, "Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models," Comput. Speech Lang., vol.9, no.2, pp. 171-185, 1995.
24. J. Gauvain and C. Lee, "Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains," IEEE Trans. Speech Audio Process., vol.2, no.2, pp. 291-298, Apr. 1994.
25. M. Tonomura, T. Kosaka, and S. Matsunaga, "Speaker adaptation based on transfer vector field smoothing using maximum a posteriori probability estimation," Comput. Speech Lang., vol.10, no.2, pp. 117-132, 1995.
26. J. Takahashi and S. Sagayama, "Vector-field-smoothed bayesian learning for fast and incremental speaker/telephone-channel adaptation," Comput. Speech Lang., vol.11, no.2, pp. 127-146, 1997.
27. K. Tokuda, T. Masuko, N. Miyazaki, and T. Kobayashi, "Multi-space probability distribution HMM," IEICE Trans. Inf. Syst., vol.E85-D, no.3, pp. 455-464, Mar. 2002.
28. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, "Speaker adaptation of pitch and spectrum for HMM-based speech synthesis," (in Japanese) IEICE Trans., vol.J85-D-II, no.4, pp. 545-553, Apr. 2002.
29. H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "A hidden semi-Markov model-based speech synthesis system," IEICE Trans. Inf. & Syst., vol.E90-D, no.5, pp. 825-834, May 2007.
30. J. Ferguson, "Variable duration models for speech," in Proc. Symp. Applicat. Hidden Markov Models to Text and Speech, 1980, pp. 143-179.
31. M. Russell and R. Moore, "Explicit modelling of state occupancy in hidden Markov models for automatic speech recognition," in Proc. ICASSP'85, Mar. 1985, pp. 5-8.
32. S. Levinson, "Continuously variable duration hidden Markov models for automatic speech recognition," Comput. Speech Lang., vol.1, no.1, pp. 29-45, 1986. (Pubitemid 17552445)
33. T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, "A compact model for speaker-adaptive training," in Proc. ICSLP'96, Oct. 1996, pp. 1137-1140.
34. J.Yamagishi, M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, "A context clustering technique for average voice models," IEICE Trans. Inf. Syst., vol.E86-D, no.3, pp. 534-542, Mar. 2003.
35. L. Qin, Z. Ling, Y. Wu, B. Zhang, and R. Wang, "HMM-based emotional speech synthesis using average emotion model," in Proc. ISCSLP'06 (Springer LNAI Book), Dec. 2006, pp. 233-240.
36. J. Latorre, K. Iwano, and S. Furui, "New approach to the polyglot speech generation by means of an HMM-based speaker adaptable synthesizer," Speech Commun., vol.48, no.10, pp. 1227-1242, 2006.
37. V. Digalakis, D. Rtischev, and L. Neumeyer, "Speaker adaptation using constrained reestimation of Gaussian mixtures," IEEE Trans. Speech Audio Process., vol.3, no.5, pp. 357-366, Sep. 1995.
38. M. Gales, "Maximum likelihood linear transformations for HMM-based speech recognition," Comput. Speech Lang., vol.12, no.2, pp. 75-98, 1998.
39. K. Shinoda and C. Lee, "A structural Bayes approach to speaker adaptation," IEEE Trans. Speech Audio Process., vol.9, pp. 276-287, Mar. 2001.
40. O. Shiohan, T. Myrvoll, and C. Lee, "Structural maximum a posteriori linear regression for fast HMM adaptation," Comput. Speech Lang., vol.16, no.3, pp. 5-24, 2002.
41. V. Digalakis and L. Neumeyer, "Speaker adaptation using combined transformation and Bayesian methods," IEEE Trans. Speech Audio Process., vol.4, no.3, pp. 294-300, Jul. 1996.
42. J. Isogai, J. Yamagishi, and T. Kobayashi, "Model adaptation and adaptive training using ESAT algorithm for HMM-based speech synthesis," in Proc. Eurospeech'05, Sep. 2005, pp. 2597-2600.
43. J. Yamagishi, K. Ogata, Y. Nakano, J. Isogai, and T. Kobayashi, "HSMM-based model adaptation algorithms for average-voice-based speech synthesis," in Proc. ICASSP'06, May 2006, pp. 77-80.
44. Y. Nakano, M. Tachibana, J. Yamagishi, and T. Kobayashi, "Constrained structural maximum a posteriori linear regression for average-voice-based speech synthesis," in Proc. ICSLP'06, Sep. 2006, pp. 2286-2289.
45. K. Ogata, M. Tachibana, J. Yamagishi, and T. Kobayashi, "Acoustic model training based on linear transformation and MAP modification for HSMM-based speech synthesis," in Proc. ICSLP'06, Sep. 2006, pp. 1328-1331.
46. J. Yamagishi, T. Kobayashi, M. Tachibana, K. Ogata, and Y. Nakano, "Model adaptation approach to speech synthesis with diverse voices and styles," in Proc. ICASSP'07, Apr. 2007, pp. 1233-1236.
47. S. Imai, "Cepstral analysis synthesis on the Mel frequency scale," in Proc. ICASSP'83, Apr. 1983, pp. 93-96.
48. K. Tokuda, T. Kobayashi, T. Fukada, H. Saito, and S. Imai, "Spectral estimation of speech based on Mel-cepstral representation," (in Japanese) IEICE Trans. Fundamentals, vol.J74-A, no.8, pp. 1240-1248, Aug. 1991.
49. Speech Signal Processing Toolkit (SPTK) Version 3.1. 2007 [Online]. Available: http://www.sp-tk.sourceforge.net/
50. D. Arifianto, T. Tanaka, T. Masuko, and T. Kobayashi, "Robust F0 estimation of speech signal using harmonicity measure based on instantaneous frequency," IEICE Trans. Inf. Syst., vol.E87-D, no.12, pp. 2812-2820, Dec. 2004.
51. T. Toda and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol.E90-D, no.5, pp. 816-824, May 2007.
52. J. Yamagishi, T. Kobayashi, S. Renals, S. King, H. Zen, T. Toda, and K. Tokuda, "Improved average-voice-based speech synthesis using gender-mixed modeling and a parameter generation algorithm considering GV," in Proc. 6th ISCA Workshop Speech Synth., Aug. 2007, pp. 125-130.
53. J. Yamagishi, T. Nose, H. Zen, T. Toda, K. Tokuda, S. King, and S. Renals, "A speaker-adaptive HMM-based speech synthesis for the Blizzard Challenge 2007," IEEE Audio, Speech, Lang. Process., 2008, submitted for publication.
54. M. Gales and P. Woodland, "Mean and variance adaptation within the MLLRframework," Comput. Speech Lang., vol.10, no.4, pp. 249-264, 1996.
55. A. Dempster, N. Laird, and D. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc., Series B, vol.39, no.1, pp. 1-38, 1977.
56. H. Zen, T. Toda, and K. Tokuda, "The Nitech-NAIST HMM-based speech synthesis system for the Blizzard Challenge 2006," IEICE Trans. Inf. Syst., vol.E91-D, no.6, pp. 1764-1773, Jun. 2008.
57. M. Gales, "Semi-tied covariance matrices for hidden Markov models," IEEE Trans. Speech Audio Process., vol.7, no.2, pp. 272-281, Mar. 1999.
58. R. Gopinath, "Maximum likelihood modeling with Gaussian distributions for classification," in Proc. ICASSP'98, May 1998, pp. 661-664.
59. M. Rahim and B. Juang, "Signal bias removal by maximum likelihood estimation for robust telephone speech recognition," IEEE Trans. Speech Audio Process., vol.4, no.1, pp. 19-30, Jan. 1996.
60. M. Gales, "Multiple-cluster adaptive training schemes," in Proc. ICASSP'01, May 2001, pp. 361-364.
61. A. Gupta and T. Varga, Elliptically Contoured Models in Statistics. Norwell, MA: Kluwer, 1993.
62. J. Chien, H.Wang, and C. Lee, "Improved Bayesian learning of hidden Markov models for speaker adaptation," in Proc. ICASSP'97, Apr. 1997, pp. 1027-1030.
63. T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai, "An adaptive algorithm for Mel-cepstral analysis of speech," in Proc. ICASSP'92, Mar. 1992, pp. 137-140.
64. K. Shinoda and T.Watanabe, "MDL-based context-dependent subword modeling for speech recognition," J. Acoust. Soc. Japan (E), vol.21, pp. 79-86, Mar. 2000.
65. Y. Sagisaka, K. Takeda, M. Abel, S. Katagiri, T. Umeda, and H. Kuwabara, "A large-scale Japanese speech database," in Proc. ICSLP'96, Nov. 1990, pp. 1089-1092.
66. K. Tokuda, H. Zen, J. Yamagishi, T. Masuko, S. Sako, A. Black, and T. Nose, The HMM-Based Speech Synthesis System (HTS). [Online]. Available: http://www.hts.sp.nitech.ac.jp/
67. H. Zen, T. Nose, J. Yamagishi, S. Sako, and K. Tokuda, "The HMMbased speech synthesis system (HTS) version 2.0," in Proc. 6th ISCA Workshop Speech Synth., Aug. 2007, pp. 294-299.