Postfilters to modify the modulation spectrum for statistical parametric speech synthesis

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 24, Issue 4, 2016, Pages 755-767

  Takamichi, Shinnosuke ^a   Toda, Tomoki ^b   Black, Alan W ^c   Neubig, Graham ^a   Sakti, Sakriani ^a   Nakamura, Satoshi ^a  

^a NARA INSTITUTE OF SCIENCE AND TECHNOLOGY   (Japan)

^b NAGOYA UNIVERSITY   (Japan)

^c Carnegie Mellon University ^*  (United States)

Author keywords

Clustergen; Global variance; GMM based voice conversion; Hmm based text to speech; Modulation spectrum; Over smoothing; Post filter; Statistical parametric speech synthesis

Indexed keywords

COMMUNICATION CHANNELS (INFORMATION THEORY); GAUSSIAN DISTRIBUTION; HIDDEN MARKOV MODELS; MARKOV PROCESSES; MODULATION; PLASMA DIAGNOSTICS; SPEECH PROCESSING; SPEECH SYNTHESIS; STATISTICS; TRAJECTORIES; TRELLIS CODES;

CLUSTERGEN; GLOBAL VARIANCE; MODULATION SPECTRUM; OVER-SMOOTHING; POSTFILTERS; STATISTICAL PARAMETRIC SPEECH SYNTHESIS; TEXT TO SPEECH; VOICE CONVERSION;

SPEECH;

EID: 84962834006     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2016.2522655     Document Type: Article

References (66)

1. J. Yamagishi, C. Veaux, S. King, and S. Renals, "Speech synthesis technologies for individuals with vocal diabilities: Voice banking and reconstruction, " Acoust. Sci. Technol., vol. 33, pp. 1-5, 2012.
2. K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, "An evaluation of excitation feature prediction in a hybrid approach to electrolaryngeal speech enhancement, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 2014, pp. 4521-4525.
3. H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, "Singing voice conversion method based on many-to-many eigenvoice conversion and training data generation using a singing-to-singing synthesis system, " in Proc. Asia Pac. Signal Inf. Process. Assoc. Annu. Summit Conf. (APSIPA ASC), Hollywood, CA, USA, Nov. 2012, pp. 1-6.
4. N. Hattori, T. Toda, H. Kawai, H. Saruwatari, and K. Shikano, "Speakeradaptive speech synthesis based on eigenvoice conversion and languagedependent prosodic conversion in speech-to-speech translation, " in Proc. INTERSPEECH, Florence, Italy, Aug. 2011, pp. 2769-2772.
5. P. K. Muthukumar and A. W. Black, "Automatic discovery of a phonetic inventory for unwritten languages for statistical speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process., Florence, Italy, May 2014, pp. 2594-2598.
6. S. Sitaram, G. Anumanchipalli, J. Chiu, A. Parlikar, and A. W. Black, "Text to speech in new languages without a standardized orthography, " in Proc. 8th Speech Synth. Workshop, Barcelona, Spain, Aug. 2013, pp. 95- 100.
7. K. Kobayashi et al., "Regression approaches to perceptual age control in singing voice conversion, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 2014, pp. 7954-7958.
8. Z.-H. Ling et al., "Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends, " IEEE Signal Process. Mag., vol. 32, no. 3, pp. 35-52, May 2015.
9. Y. Sagisaka, "Speech synthesis by rule using an optimal selection of non-uniform synthesis units, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), New York, NY, USA, Apr. 1988, pp. 679-682.
10. Y. Stylianou, O. Cappe, and E. Moulines, "Continuous probabilistic transform for voice conversion, " IEEE Trans. Speech Audio Process., vol. 6, no. 2, pp. 131-142, Mar. 1988.
11. H. Zen, K. Tokuda, and A. Black, "Statistical parametric speech synthesis, " Speech Commun., vol. 51, no. 11, pp. 1039-1064, 2009.
12. K. Tokuda, T. Kobayashi, and S. Imai, "Speech parameter generation from HMM using dynamic features, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Detroit, MI, USA, May 1995, pp. 660-663.
13. K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura, "Speech synthesis based on hidden Markov models, " Proc. IEEE, vol. 101, no. 5, pp. 1234-1252, May 2013.
14. T. Toda, A. W. Black, and K. Tokuda, "Voice conversion based on maximum likelihood estimation of spectral parameter trajectory, " IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov. 2007.
15. A.W. Black, "CLUSTERGEN: A statistical parametric synthesizer using trajectory modeling, " in Proc. INTERSPEECH, Pittsburgh, PA, USA, Sep. 2006.
16. T. Koriyama, T. Nose, and T. Kobayashi, "Statistical parametric speech synthesis based on Gaussian process regression, " IEEE J. Sel. Topics Signal Process., vol. 8, no. 2, pp. 173-183, Apr. 2014.
17. E. Helander, T. V. H. Silen, and M. Gabbouj, "Voice conversion using dynamic kernel partial least squares regression, " IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 3, pp. 806-817, Mar. 2012.
18. H. Zen and A. Senior, "Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 2014, pp. 3872-3876.
19. T. Nakashika, R. Takashima, T. Takiguchi, and Y. Ariki, "Voice conversion in high-order Eigen space using deep belief nets, " in Proc. INTERSPEECH, Lyon, France, Aug. 2013, pp. 369-372.
20. A. J. Hunt and A. Black, "Unit selection in a concatenative speech synthesis system using a large speech database, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Atlanta, GA, USA, May 1996, pp. 373-376.
21. K. Oura, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda, "An investigation of implementation performance analysis of DNN based speech synthesis system, " in Proc. INTERSPEECH, Brighton, U.K., 2014, pp. 577-582.
22. J. Yamagishi and T. Kobayashi, "Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training, " IEICE Trans. Inf. Syst., vol. E90-D, no. 2, pp. 533-543, 2007.
23. T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi, "A style control technique for HMM-based expressive speech synthesis, " IEICE Trans. Inf. Syst., vol. E90-D, no. 9, pp. 1406-1413, 2007.
24. L. Chen, M. J. F. Gales, L. Chen, K. Chin, K. Knull, and M. Akamine, "Exploring rich expressive information from audiobook data using cluster adaptive training, " in Proc. INTERSPEECH, Portland, OR, USA, Sep. 2012.
25. S. King and V. Karaiskos, "The Blizzard challenge 2011, " in Proc. Blizzard Challenge Workshop, Turin, Italy, Sep. 2011.
26. Y. Stylianou, "Voice transformation: A survey, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Taipei, Taiwan, Apr. 2009, pp. 3585-3588.
27. H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds, " Speech Commun., vol. 27, no. 3-4, pp. 187-207, 1999.
28. M. Morise, "An attempt to develop a singing synthesizer by collaborative creation, " in Proc. Stockholm Music Acoust. Conf. (SMAC), Stockholm, Sweden, Aug. 2013, pp. 287 292.
29. Y. Agiomyrgiannakis, "Vocaine the vocoder and applications in speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Brisbane, QLD, Australia, Apr. 2015, pp. 4230-4234.
30. P. K. Muthukumar, A. W. Black, and H. T. Bunnell, "Optimizations and fitting procedures for the Liljencrants-Fant model for statistical parametric speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Vancouver, BC, Canada, May 2013, pp. 397-401.
31. T. Toda and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis, " IEICE Trans., vol. E90-D, no. 5, pp. 816-824, 2007.
32. S. Takamichi, T. Toda, Y. Shiga, S. Sakti, G. Neubig, and S. Nakamura, "Parameter generation methods with rich context models for highquality and flexible text-to-speech synthesis, " IEEE J. Sel. Topics Signal Process., vol. 8, no. 2, pp. 239-250, Apr. 2014.
33. T. Nose, V. Chunwijitra, and T. Kobayashi, "A parameter generation algorithm using local variance for HMM-based speech synthesis, " IEEE J. Sel. Topics Signal Process., vol. 8, no. 2, pp. 221-228, Apr. 2014.
34. M. Shannon and W. Byrne, "Fast, low-artifact speech synthesis considering global variance, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Vancouver, BC, Canada, May 2013, pp. 7869-7873.
35. T. Toda, T. Muramatsu, and H. Banno, "Implementation of computationally efficient real-time voice conversion, " in Proc. INTERSPEECH, Portland, OR, USA, Sep. 2012.
36. R. Drullman, J. M. Festen, and R. Plomp, "Effect of reducing slow temporal modulations on speech reception, " J. Acoust. Soc. Amer., vol. 95, pp. 2670-2680, 1994.
37. S. Thomas, S. Ganapathy, and H. Hermansky, "Phoneme recognition using spectral envelop and modulation frequency features, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Taipei, Taiwan, Apr. 2009, pp. 4453-4456.
38. S. Gergen, A. Nagathil, and R. Martin, "Reduction of reverberation effects in the MFCC modulation spectrum for improved classification of acoustic signals, " in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 1992-1995.
39. Z. Wu, X. Xiao, E. S. Chng, and H. Li, "Synthetic speech detection using temporal modulation feature, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Vancouver, BC, Canada, May 2013, pp. 7234-7238.
40. R. Drullman, J. M. Festen, and R. Plomp, "Effect of temporal envelope smearing on speech perception, " J. Acoust. Soc. Amer., vol. 95, pp. 1053- 1064, 1994.
41. T. Arai, M. Pavel, H. Hermansky, and C. Avendano, "Intelligibility of speech with filtered time trajectories of spectral envelopes, " in Proc. 4th Int. Conf. Spoken Lang. (ICSLP), 1996, vol. 4, pp. 2490-2493.
42. T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, "Lowdelay voice conversion based on maximum likelihood estimation of spectral parameter trajectory, " in Proc. INTERSPEECH, Brisbane, QLD, Australia, Sep. 2008, pp. 1076-1079.
43. H. Zen and H. Sak, "Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Brisbane, QLD, Australia, Apr. 2015, pp. 4470-4474.
44. S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, "A postfilter to modify modulation spectrum in HMM-based speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 2014, pp. 290-294.
45. S. Takamichi, T. Toda, A. W. Black, and S. Nakamura, "Modified modulation spectrum-based post-filter for HMM-based speech synthesis, " in Proc. GlobalSIP, Atlanta, GA, USA, Dec. 2014, pp. 710-714.
46. S. Takamichi, T. Toda, A. W. Black, and S. Nakamura, "Modulation spectrum-based post-filter for GMM-based voice conversion, " in Proc. Annu. Summit Conf. Asia-Pac. Signal Inf. Process. Assoc. (APSIPA ASC), Siem Reap, Cambodia, Dec. 2014, pp. 1-4.
47. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis, " in Proc. EUROSPEECH, Budapest, Hungary, Apr. 1999, pp. 2347-2350.
48. K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, "Speech parameter generation algorithms for HMM-based speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Istanbul, Turkey, Jun. 2000, pp. 1315-1318.
49. K. Shinoda and T. Watanabe, "MDL-based context-dependent subword modeling for speech recognition, " J. Acoust. Soc. Jpn. (E), vol. 28, no. 3, pp. 140-146, 2007.
50. L. Atlas and S. A. Shamma, "Joint acoustic and modulation frequency, " EURASIP J. Appl. Signal Process., vol. 7, pp. 668-675, 2003.
51. S. Pan, J. Tao, and Y. Wang, "A state duration generation algorithm considering global variance for HMM-based speech synthesis, " in Proc. Annu. Summit Conf. Asia-Pac. Signal Inf. Process. Assoc. (APSIPA ASC), Xi'an, China, 2011.
52. K. Yu and S. Young, "Continuous F0 modeling for HMM based statistical parametric speech synthesis, " IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 5, pp. 1071-1079, Jul. 2011.
53. K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, "A hybrid approach to electrolaryngeal speech enhansement based on spectral subtraction and statistical voice conversion, " in Proc. Interspeech, Lyon, France, Sep. 2013, pp. 3067-3071.
54. P. Taylor, Text-To-Speech Synthesis. Cambridge, U.K.: Cambridge Univ. Press, 2009.
55. F. Eyben and Y. Agiomyrgiannakis, "A frequency-weighted post-filtering transform for compensation of the over-smoothing effect in HMM-based speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Florence, Italy, May 2014, pp. 275-279.
56. K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda, "The effect of neural networks in statistical parametric speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Brisbane, QLD, Australia, Apr. 2015, pp. 4455-4459.
57. L.-H. Chen, T. Raitio, C.-V. Botinhao, J. Yamagishi, and Z.-H. Ling, "DNN-based stochastic postfilter for HMM-based speech synthesis, " in Proc. INTERSPEECH, MAX Atria, Singapore, May 2014, pp. 1954- 1958.
58. H. Zen, M. J. F. Gales, Y. Nankaku, and K. Tokuda, "Product of experts for statistical parametric speech synthesis, " IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 3, pp. 794-805, Mar. 2011.
59. T. Nose and A. Ito, "Analysis of spectral enhancement using global variance in HMM-based speech synthesis, " in Proc. INTERSPEECH, MAX Atria, Singapore, May 2014, pp. 2917-2921.
60. H. Zen, K. Tokuda, T. K. T. Masuko, and T. Kitamura, "Hidden semi- Markov model based speech synthesis system, " IEICE Trans. Inf. Syst., E90-D, no. 5, pp. 825-834, 2007.
61. Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, and H. Kuawhara, "A large-scale Japanese speech database, " in Proc. Int. Conf. Spoken Lang. (ICSLP'90), Kobe, Japan, Nov. 1990, pp. 1089-1092.
62. H. Kawahara, J. Estill, and O. Fujimura, "Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system STRAIGHT, " in Proc. Int. Workshop Models Anal. Vocal Emissions Biomed. Appl. (MAVEBA), Firentze, Italy, Sep. 2001, pp. 1-6.
63. Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, "Maximum likelihood voice conversion based on GMM with STRAIGHT mixed excitation, " in Proc. INTERSPEECH, Pittsburgh, PA, USA, Sep. 2006, pp. 2266-2269.
64. V. Tyagi, I. McCowan, H. Misra, and H. Bourlard, "Mel-cepstrum modulation spectrum (MCMS) features for robust ASR, " in Proc. Automat. Speech Recog. Understand. (ASRU), MAX Atria, Singapore, Nov. 2003, pp. 399-404.
65. Speech Signal Processing Toolkit (SPTK) [Online]. Available: http://sp-tk.sourceforge.net/.
66. Amazon Mechanical Turk [Online]. Available: https://www.mturk.com/.