Thousands of voices for HMM-based speech synthesis - Analysis and application of TTS systems fuilt on various ASR corpora

IEEE Transactions on Audio, Speech and Language Processing

Volumn 18, Issue 5, 2010, Pages 984-1004

  Yamagishi, Junichi ^a   Usabaev, Bela ^b   King, Simon ^a   Watts, Oliver ^a   Dines, John ^c   Tian, Jilei ^d   Guan, Yong ^d   Hu, Rile ^d   Oura, Keiichiro ^e   Wu, Yi Jian ^e,f   Tokuda, Keiichi ^e   Karhila, Reima ^g   Kurimo, Mikko ^g  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b UNIVERSITY OF TÜBINGEN   (Germany)

^c IDIAP RESEARCH INSTITUTE   (Switzerland)

^d NOKIA RESEARCH CENTER   (United States)

^e NAGOYA INSTITUTE OF TECHNOLOGY   (Japan)

^f TTS Group   (China)

^g AALTO UNIVERSITY   (Finland)

Author keywords

Automatic speech recognition (ASR); Average voice; H Triple S (HTS); Hidden Markov model (HMM) based speech synthesis; Speaker adaptation; Speech synthesis; SPEECON database; Voice conversion; WSJ database

Indexed keywords

AUTOMATIC SPEECH RECOGNITION; AUTOMATIC SPEECH RECOGNITION (ASR); AVERAGE VOICE; SPEAKER ADAPTATION; VOICE CONVERSION;

DATABASE SYSTEMS; HIDDEN MARKOV MODELS; SPEECH SYNTHESIS;

SPEECH RECOGNITION;

EID: 77953708096     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2010.2045237     Document Type: Article

References (70)

1. J. Yamagishi et al., "Thousands of voices for HMM-based speech synthesis," in Proc. Interspeech-99, Brighton, U.K., Sep. 2009, pp. 420-423.
2. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Simultaneous modeling of spectrum, pitch and duration in HMMbased speech synthesis," in Proc. EUROSPEECH-99, Budapest, Hungary, Sep. 1999, pp. 2374-12350
3. K. Tokuda, H. Zen, J. Yamagishi, T. Masuko, S. Sako, A. B. Black, and T. Nose, "The HMM-Based Speech Synthesis System (HTS) Version 2.1." [Online]. Available: http://hts.sp.nitech.ac.jp/
4. H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. W. Black, and K. Tokuda, "The HMM-based speech synthesis system (HTS) version 2.0," in Proc. 6th ISCA Workshop Speech Synth. (SSW-6), Bonn, Germany, Aug. 2007.
5. J.Yamagishi, T. Nose, H. Zen, Z.-H. Ling, T. Toda, K. Tokuda, S. King, and S. Renals, "A robust speaker-adaptive HMM-based text-to-speech synthesis," IEEE Trans. Speech, Audio, Lang. Process., vol.17, no.6, pp. 1208-1230, Aug. 2009.
6. J. Yamagishi, Z.-H. Ling, and S. King, "Robustness of HMM-based speech synthesis," in Proc. Interspeech-08, Brisbane, Australia, Sep. 2008, pp. 581-584.
7. H. Zen, K. Tokuda, and A. W. Black, "Statistical parametric speech synthesis," Speech Commun., vol.51, no.11, pp. 1039-1064, Nov. 2009.
8. D. B. Paul and J. M. Baker, "The design for thewall street journal-based CSR corpus," in Proc.Workshop Speech Natural Lang., Harriman, NY, 1992, pp. 357-362.
9. T. Robinson, J. Fransen, D. Pye, J. Foote, and S. Renals, "WSJCAM0: A British English speech corpus for large vocabulary continuous speech recognition," in Proc. ICASSP-95, Detroit, MI, May 1995, pp. 81-84.
10. D. S. Pallet, J. G. Fiscus, and J. S. Garofolo, "DARPAresource management bench," in Proc. Workshop Speech Natural Lang., Hidden Valley, PA, Jun. 1990, pp. 298-305.
11. T. Schultz, "GlobalPhone: A multilingual speech and text database developed at Karlsruhe university," in Proc. ICSLP'02, Denver, CO, Sep. 2002, pp. 345-348.
12. D. Iskra, B. Grosskopf, K. Marasek, H. V. D. Heuvel, F. Diehl, and A. Kiessling, "SPEECON-speech databases for consumer devices: Database specification and validation," in Proc. LREC'02, Canary Islands, Spain, May 2002, pp. 329-333.
13. M. J. F. Gales and S. J. Young, "The application of hidden Markov models in speech recognition," Foundations Trends R Signal Process., vol.1, no.3, pp. 195-304, 2008.
14. K. Itou, M. Yamamoto, K. Takeda, T. Takezawa, T. Matsuoka, T. Kobayashi, K. Shikano, and S. Itahashi, "The design of the newspaper- based Japanese large vocabulary continuous speech recognition corpus," in Proc. ICSLP-98, Sydney, Australia, Dec. 1998, pp. 3261-3264.
15. H. Kubozono, "Mora and syllable," in The Handbook of Japanese Linguistics, N. Tsujimura, Ed. New York: Blackwell, 1995, pp. 31-61.
16. S. Fitt and S. Isard, "Synthesis of regional English using a keyword lexicon," in Proc. Eurospeech-99, Budapest, Hungary, Sep. 1999, vol.2, pp. 823-826.
17. R. A. J. Clark, K. Richmond, and S. King, "Multisyn: Open-domain unit selection for the Festival speech synthesis system," Speech Commun., vol.49, no.4, pp. 317-330, 2007.
18. [Online]. Available: http://www.lc-star.com
19. Y. Qian, F. Soong, Y. Chen, and M. Chu, "An HMM-based Mandarin Chinese text-to-speech system," in Proc. ISCSLP'06, Singapore, Dec. 2006, pp. 223-232.
20. Deliverable Report D2.1 EMIME Project, 2008.
21. Y. Guan, J. Tian, Y.-J. Wu, J. Yamagishi, and J. Nurminen, "An efficient and unified approach of Mandarin HTS system," in Proc. ICASSP'10, Dallas, TX, Mar. 2010.
22. H. Kawai, T. Toda, J. Ni, M. Tsuzaki, and K. Tokuda, "XIMERA: A new TTS from ATR based on corpus-based technologies," in Proc. ISCA 5th Speech Synth. Workshop, Pittsburgh, PA, Jun. 2004, pp. 179-184.
23. H. Kawai, T. Toda, J. Yamagishi, T. Hirai, J. Ni, N. Nishizawa, M. Tsuzaki, and K. Tokuda, "XIMERA: A concatenative speech synthesis system with large scale corpora," IEICE Trans. Inf. Syst., vol.J89-D-II, no.12, pp. 2688-2698, Dec. 2006.
24. S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J.-S. Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto, "The ATR multilingual speech-to-speech translation system," IEEE Trans. Speech, Audio, Lang. Process., vol.14, no.2, pp. 365-376, Mar. 2006.
25. R. Barra-Chicote, J. Yamagishi, J. Montero, S. King, S. Lutfi, and J. Macias-Guarasa, "Generacion de una voz sintetica en Castellano basada en HSMM para la Evaluacion Albayzin 2008: Conversion texto a voz," in V Jornadas en Tecnologia del Habla (in Spanish), Bilbao, Spain, Nov. 2008, pp. 115-118 [Online]. Available: http://www.cstr.inf.ed.ac.uk/downloads/ publications/ 2008/tts-jth08.pdf
26. K. Tokuda, H. Zen, and A. W. Black, "HMM-based approach to multilingual speech synthesis," in Text to Speech Synthesis: New Paradigms and Advances, S. Narayanan and A. Alwan, Eds. Upper Saddle River, NJ: Prentice-Hall, 2004.
27. S. J. Young, J. J. Odell, and P. C. Woodland, "Tree-based state tying for high accuracy acoustic modeling," in Proc. ARPA Human Lang. Technol. Workshop, Plainsboro, NJ, Mar. 1994, pp. 307-312.
28. J. Yamagishi, H. Zen, Y.-J. Wu, T. Toda, and K. Tokuda, "The HTS- 2008 system: Yet another evaluation of the speaker-adaptive HMMbased speech synthesis system in the 2008 Blizzard Challenge," in Proc. Blizzard Challenge 2008, Brisbane, Australia, Sep. 2008.
29. M. Fraser and S. King, "The Blizzard Challenge 2007," in Proc. BLZ3- 2007 (in Proc. SSW6), Bonn, Germany, Aug. 2007.
30. V. Karaiskos, S. King, R. A. J. Clark, and C. Mayo, "The Blizzard Challenge 2008," in Proc. Blizzard Challenge 2008, Brisbane, Australia, Sep. 2008.
31. H. Kawahara, I. Masuda-Katsuse, and A. Cheveigné, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Commun., vol.27, pp. 187-207, 1999.
32. H. Zen, T. Toda, M. Nakamura, and K. Tokuda, "Details of Nitech HMM-based speech synthesis system for the Blizzard Challenge 2005," IEICE Trans. Inf. Syst., vol.E90-D, no.1, pp. 325-333, Jan. 2007.
33. H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "A hidden semi-Markov model-based speech synthesis system," IEICE Trans. Inf. Syst., vol.E90-D, no.5, pp. 825-834, May 2007.
34. A. Dempster, N. Laird, and D. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Statist. Soc., Series B, vol.39, no.1, pp. 1-38, 1977.
35. K. Shinoda and T.Watanabe, "MDL-based context-dependent subword modeling for speech recognition," J. Acoust. Soc. Japan (E), vol.21, pp. 79-86, Mar. 2000.
36. Y. Guan and J. Tian, "Evaluation of flat start labeling for phoneme based Mandarin HTS system," in Proc. ORIENTAL-COCOSDA-09, Aug. 2009, pp. 187-190.
37. T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, "A compact model for speaker-adaptive training," in Proc. ICSLP-96, Philadelphia, PA, Oct. 1996, pp. 1137-1140.
38. M. J. F. Gales, "Maximum likelihood linear transformations for HMMbased speech recognition," Comput. Speech Lang., vol. 12, no. 2, pp. 75-98, 1998.
39. J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Isogai, "Analysis of speaker adaptation algorithms for HMM-based speech synthesis and a constrained SMAPLR adaptation algorithm," IEEE Trans. Speech, Audio, Lang. Process., vol. 17, no. 1, pp. 66-83, Jan. 2009.
40. T. Toda and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol.E90-D, no.5, pp. 816-824, May 2007.
41. E. Moulines and F. Charpentier, "Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones," Speech Commun., vol.9, no.5-6, pp. 453-468, 1990.
42. T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai, "An adaptive algorithm for mel-cepstral analysis of speech," in Proc. ICASSP-92, San Francisco, CA, Mar. 1992, pp. 137-140.
43. S. King and V. Karaiskos, "The Blizzard Challenge 2009," in Proc. Blizzard Challenge Workshop, Edinburgh, U.K., Sep. 2009.
44. J.Yamagishi and T.Kobayashi, "Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training," IEICE Trans. Inf. Syst., vol.E90-D, no.2, pp. 533-543, Feb. 2007.
45. J. Dines, J. Yamagishi, and S. King, "Measuring the gap between HMM-based ASR and TTS," in Proc. Interspeech-09, Brighton, U.K., Sep. 2009, pp. 1391-1394.
46. J. Dines, J. Yamagishi, and S. King, "Measuring the gap between HMM-based ASR and TTS," IEEE J. Sel. Topics Signal Process., 2010, to be published.
47. D. S. Pallett, J. G. Fiscus, W. M. Fisher, J. S. Garofolo, B. A. Lund, and M. A. Przybocki, "1993 benchmark tests for the ARPA spoken language program," in Proc. HLT '94: Workshop Human Lang. Technol., Morristown, NJ, 1994, pp. 49-74.
48. Y.-J. Wu, S. King, and K. Tokuda, "Cross-lingual speaker adaptation for HMM-based speech synthesis," in Proc. ISCSLP-08, Kunming, China, 2008, pp. 9-12.
49. Y.-J. Wu and K. Tokuda, "State mapping based method for cross-lingual speaker adaptation in HMM-based speech synthesis," in Proc. Interspeech- 09, Brighton, U.K., Sep. 2009, pp. 528-531.
50. J. A. Gray and J. Markel, "Distance measures for speech processing," IEEE Trans. Acoust., Speech, Signal Process., vol.ASSP-24, no.5, pp. 380-391, Oct. 1976.
51. T. P. Barnwell, III, "Correlation analysis of subjective and objective measures for speech quality," in Proc. ICASSP-80, Denver, CO, 1980, pp. 706-709.
52. T. Masuko, K. Tokuda, T. Kobayashi, and S. Imai, "Speech synthesis using HMMs with dynamic features," in Proc. ICASSP-96, Atlanta, GA, May 1996, pp. 389-392.
53. A. W. Black and J. Kominek, "Optimizing segment label boundaries for statistical speech synthesis," in Proc. ICASSP-09, Taipei, Taiwan, Apr. 2009, pp. 3785-3788.
54. R. A. J. Clark, M. Podsiadlo, M. Fraser, C. Mayo, and S. King, "Statistical analysis of the Blizzard Challenge 2007 listening test results," in Proc. BLZ3-2007 (in Proc. SSW6), Bonn, Germany, Aug. 2007.
55. M. Shozakai and G. Nagino, "Analysis of speaking styles by two-dimensional visualization of aggregate of acoustic models," in Proc. ICSLP-04, Jeju Island, Korea, Oct. 2004, pp. 717-720.
56. A. Maier, M. Schuster, U. Eysholdt, T. Haderlein, T. Cincarek, S. Steidl, A. Batliner, S. Wenhardt, and E. Noth, "QMOS-A robust visualization method for speaker dependencies with different microphones," J. Pattern Recognition Res., vol.1, pp. 32-51, 2009.
57. T. Cox and M. Cox, Multidimensional Scaling. London, U.K.: Chapman & Hall, 2001.
58. S. Tsakalidis and W. Byrne, "Acoustic training from heterogeneous data sources: Experiments in Mandarin conversational telephone speech transcription," in Proc. ICASSP-05, 18-23, 2005, vol.1, pp. 461-464.
59. S. Tsakalidis and W. Byrne, "Cross-corpus normalization of diverse acoustic training data for robustHMMtraining," Cambridge Univ. Eng. Dept., Cambridge, U.K., 2005.
60. K. Tokuda, H. Zen, and T. Kitamura, "Reformulating the HMM as a trajectory model," IEICE Tech. Rep. Natural Lang. Understanding Models of Commun., vol.104, no.538, pp. 43-48, Dec. 2004.
61. D. Erro, "Intra-lingual and cross-lingual voice conversion using harmonic plus stochastic models," Ph.D. dissertation, Univ. Politecnica de Catalunya, Barcelona, Spain, 2008.
62. J. H. Langlois and L. A. Roggman, "Attractive faces are only average," Psychol. Sci., vol.1, no.2, pp. 115-121, 1990.
63. J. Yamagishi, M. Lincoln, S. King, J. Dines, M. Gibson, J. Tian, and Y. Guan, "Analysis of unsupervised and noise-robust speaker-adaptive HMM-based speech synthesis systems toward a unified ASR and TTS framework," in Proc. Blizzard Challenge Workshop, Edinburgh, U.K., Sep. 2009.
64. K. Tokuda, T. Kobayashi, T. Masuko, and S. Imai, "Mel-generalized cepstral analysis-A unified approach to speech spectral estimation," in Proc. ICSLP-94, Yokohama, Japan, Sep. 1994, pp. 1043-1046.
65. M. J. F. Gales, "Adaptive training for robust ASR," in Proc. IEEE Workshop Autom. Speech Recognition Understanding, Madonna di Campiglio, Italy, 2001, pp. 15-20.
66. P. C. Woodland, "The development of the HTK broadcast news transcription system: An overview," Speech Commun., vol.37, no.1-2, pp. 47-67, 2002.
67. S. Creer, P. Green, S. Cunningham, and J. Yamagishi, "Building personalised synthesised voices for individuals with dysarthria using the HTS toolkit," in Computer Synthesized Speech Technologies: Tools for Aiding Impairment, J. W. Mullennix and S. E. Stern, Eds. Hershey, PA: IGI Global, Jan. 2010.
68. T. Masuko, T. Hitotsumatsu, K. Tokuda, and T. Kobayashi, "On the security of HMM-based speaker verification systems against imposture using synthetic speech," in Proc. Eurospeech-99, Budapest, Hungary, Sep. 1999, pp. 1223-1226.
69. T. Masuko, K. Tokuda, and T. Kobayashi, "Imposture using synthetic speech against speaker verification based on spectrum and pitch," in Proc. ICSLP-00, Beijing, China, Oct. 2000, pp. 302-305.
70. P. L. De Leon, V. R. Apsingekar, M. Pucher, and J. Yamagishi, "Revisiting the security of speaker verification systems against imposture using synthetic speech," in Proc. ICASSP-10, Dallas, TX, Mar. 2010.