Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 26, Issue 1, 2018, Pages 84-96

  Saito, Yuki ^a   Takamichi, Shinnosuke ^a   Saruwatari, Hiroshi ^a  

^a UNIVERSITY OF TOKYO   (Japan)

Author keywords

deep neural networks; generative adversarial networks; over smoothing; Statistical parametric speech synthesis; text tospeech synthesis; voice conversion

Indexed keywords

DEEP NEURAL NETWORKS; SPEECH PROCESSING; SPEECH SYNTHESIS;

ADVERSARIAL NETWORKS; MINIMUM GENERATION ERRORS; OVER-SMOOTHING; QUALITY DEGRADATION; SPECTRAL PARAMETERS; STATISTICAL PARAMETRIC SPEECH SYNTHESIS; TRAINING ALGORITHMS; VOICE CONVERSION;

SPEECH;

EID: 85031781820     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2017.2761547     Document Type: Article

References (53)

1. H. Zen, K. Tokuda, A. Black, "Statistical parametric speech synthesis, " Speech Communi., vol. 51, no. 11, pp. 1039-1064, 2009.
2. Y. Sagisaka, "Speech synthesis by rule using an optimal selection of non-uniform synthesis units, " in Proc. Int. Conf. Acoust., Speech, Signal Process., New York, NY, USA, Apr. 1988, pp. 679-682.
3. Y. Stylianou, O. Cappé, E. Moulines, "Continuous probabilistic transform for voice conversion, " IEEE Trans. Speech Audio Process., vol. 6, no. 2, pp. 131-142, Mar. 1988.
4. Z.-H. Ling, et al., "Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends, " IEEE Signal Process. Mag., vol. 32, no. 3, pp. 35-52, May 2015.
5. K. Tokuda, Y Nankaku, T. Toda, H. Zen, J. Yamagishi, K. Oura, "Speech synthesis based on hiddenMarkovmodels, " Proc. IEEE, vol. 101, no. 5, pp. 1234-1252, Apr. 2013.
6. T. Toda, A. W. Black, K. Tokuda, "Voice conversion based on maximum likelihood estimation of spectral parameter trajectory, " IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov. 2007.
7. Y. J. Wu and R. H. Wang, "Minimum generation error training for HMMbased speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process., Toulouse, France, May 2006, pp. 89-92.
8. Z. Wu and S. King, "Improving trajectory modeling for DNN-based speech synthesis by using stacked bottleneck features and minimum trajectory error training, " IEEE Trans. Audio, Speech, Lang. Process., vol. 24, no. 7, pp. 1255-1265, Jul. 2016.
9. T. Toda, et al., "The voice conversion challenge 2016, " in Proc. INTERSPEECH, San Francisco, CA, USA, Sep. 2016, pp. 1632-1636.
10. Y. Ijima, T. Asami, H. Mizuno, "Objective evaluation using association between dimensions within spectral features for statistical parametric speech synthesis, " in Proc. INTERSPEECH, San Francisco, CA, USA, Sep. 2016, pp. 337-341.
11. Y. Ohtani, M. Tamura, M. Morita, T. Kagoshima, M. Akamine, "Histogram-based spectral equalization for HMM-based speech synthesis using mel-LSP, " in Proc. INTERSPEECH, Portland, OR, USA, Sep. 2012, pp. 1155-1158.
12. S. Takamichi, T. Toda, A. W. Black, G. Neubig, S. Sakti, S. Nakamura, "Postfilters to modify the modulation spectrum for statistical parametric speech synthesis, " IEEE Trans. Audio, Speech, Lang. Process., vol. 24, no. 4, pp. 755-767, Apr. 2016.
13. S. Takamichi, T. Toda, A. W. Black, S. Nakamura, "Modulation spectrum-constrained trajectory training algorithm for GMM-based voice conversion, " in Proc. Int. Conf. Acoust., Speech, Signal Process., Brisbane, QLD, Australia, Apr. 2015, pp. 4859-4863.
14. K. Hashimoto, K. Oura, Y. Nankaku, K. Tokuda, "Trajectory training considering global variance for speech synthesis based on neural networks, " in Proc. Int. Conf. Acoust., Speech, Signal Process., Shanghai, China, Mar. 2016, pp. 5600-5604.
15. T. Nose and A. Ito, "Analysis of spectral enhancement using global variance in HMM-based speech synthesis, " in Proc. INTERSPEECH, MAX Atria, Singapore, May 2014, pp. 2917-2921.
16. H. Zen, A. Senior, M. Schuster, "Statistical parametric speech synthesis using deep neural networks, " in Proc. Int. Conf. Acoust., Speech, Signal Process, Vancouver, BC, Canada, May 2013, pp. 7962-7966.
17. I. Goodfellow, et al., "Generative adversarial nets, " in Proc. Int. Conf. Neural Inf. Process. Syst., 2014, pp. 2672-2680.
18. Z. Wu, et al., "Anti-spoofing for text-independent speaker verification: An initial database, comparison of countermeasures, human performance, " IEEE Trans. Audio, Speech, Lang. Process., vol. 24, no. 4, pp. 768-783, Apr. 2016.
19. N. Chen, Y. Qian, H. Dinkel, B. Chen, K. Yu, "Robust deep feature for spoofing detection the SJTU system for ASVspoof 2015 Challenge, " in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 2097-2101.
20. K. Yu and S. Young, "Continuous F0 modeling for HMM based statistical parametric speech synthesis, " IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 5, pp. 1071-1079, Jul. 2011.
21. E. Grabe and E. L. Low, "Durational variability in speech and the rhythm class hypothesis, " in Papers in Laboratory Phonology 7. Berlin, Germany: Mouton de Gruyter, 2002, pp. 515-546.
22. S. Nowozin, B. Cseke, R. Tomioka, "f-GAN: Training generative neural samplers using variational divergence minimization, " in Proc. Int. Conf. Neural Inf. Process. Syst., Dec. 2016, pp. 271-279.
23. M. Arjovsky, S. Chintala, L. Bottou, "Wasserstein GAN, " in Proc. 34th Int. Conf. Mach. Learn., PMLR 70, 2017, pp. 214-223.
24. X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, S. P. Smolley, "Least squares generative adversarial networks, " IEEE Int. Conf. Comput. Vision (ICCV), 2017, pp. 2794-2802.
25. D. D. Lee and H. S. Seung, "Algorithms for non-negative matrix factorization, " in Proc. Int. Conf. Neural Inf. Process. Syst., 2000, pp. 556-562.
26. R. Kompass, "A generalized divergence measure for nonnegative matrix factorization, " Neural Comput., vol. 19, no. 3, pp. 780-891, Mar. 2007.
27. I. Csiszár and P. C. Shields, "Information theory and statistics: A tutorial, " Found. Trends Commun. Inf. Theory, vol. 1, no. 4, pp. 417-518, 2004.
28. Cédric Vilani, Optimal Transport: Old and New. New York, NY, USA: Springer-Verlag, 2009.
29. B. Huang, D. Ke, H. Zheng, B. Xu, Y. Xu, K. Su, "Multi-task learning deep neural networks for speech feature denoising, " in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 2464-2468.
30. S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, H. Lee, "Generative adversarial text-to-image synthesis, " in Proc. Int. Conf. Mach. Learn., 2016, pp. 1060-1069.
31. H. Zen and H. Sak, "Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process., Brisbane, QLD, Australia, Apr. 2015, pp. 4470-4474.
32. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks, " Science, vol. 313, no. 5786, pp. 504-507, 2006.
33. M. Tulio Ribeiro, S. Singh, C. Guestrin, "Why should I trust you': Explaining the predictions of any classifier, " in Proc. 22nd ACMSIGKDD Int. Conf. Knowl. Discovery Data Mining, San Francisco, CA, USA, Aug. 2016, pp. 1135-1164.
34. Y. Li, K. Swersky, R. Zemel, "Generative moment matching networks, " in Proc. 32nd Int. Conf. Mach. Learn., 2015, pp. 1718-1727.
35. I. Goodfellow, "NIPS 2016 tutorial: Generative adversarial networks, " arXiv:1701. 00160, 2017.
36. D. N. Reshef, et al., "Detecting novel associations in large data sets, " Science, vol. 334, no. 6062, pp. 1518-1524, 2011.
37. X. Glorot, A. Bordes, Y. Bengio, "Deep sparse rectifier neural networks, " in Proc. 14th Int. Conf. Artif. Intell. Statist., Lauderdale, FL, USA, Apr. 2011, pp. 315-323.
38. K. Tanaka, T. Toda, G. Neubig, S. Sakti, S. Nakamura, "A hybrid approach to electrolaryngeal speech enhancement based on noise reduction and statistical excitation generation, " IEICE Trans. Inf. Syst., vol. E97-D, no. 6, pp. 1429-1437, Jun. 2014.
39. S. Kang and H. Meng, "Statistical parametric speech synthesis using weighted multi-distribution deep belief network, " in Proc. INTERSPEECH, Max Atria, Singapore, Sep. 2014, pp. 1959-1963.
40. H. Zhang, et al., "StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks, " IEEE Int. Conf. Comput. Vision (ICCV), 2017, pp. 5907-5915.
41. X. Yin, et al., "Modeling f0 trajectories in hierarchically structured deep neural networks, " Speech Commun., vol. 76, pp. 82-92, 2016.
42. T. Kaneko, H. Kameoka, N. Hojo, Y. Ijima, K. Hiramatsu, K. Kashino, "Generative adversarial network-based postfilter for statistical parametric speech synthesis, " in Proc. Int. Conf. Acoust., Speech, Signal Process., New Orleans, LA, USA, Mar. 2017, pp. 4910-4914.
43. H. Zen, Y. Agiomyrgiannakis, N. Egberts, F. Henderson, P. Szczepaniak, "Fast, compact, high quality LSTM-RNN based statistical parametric speech synthesizer for mobile devices, " in Proc. INTERSPEECH, San Francisco, CA, USA, Sep. 2016, pp. 2273-2277.
44. K. Tokuda and H. Zen, "Directly modeling voiced and unvoiced components in speech waveforms by neural networks, " in Proc. Int. Conf. Acoust., Speech, Signal Process., Shanghai, China, Mar. 2016, pp. 5640-5644.
45. A. Oord, et al., "WaveNet: A generative model for raw audio, " arXiv:1609. 03499, 2016.
46. Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, H. Kawahara, "A large-scale Japanese speech database, " in Proc. Int. Conf. Spoken Lang. Process., Kobe, Japan, Nov. 1990, pp. 1089-1092.
47. H. Kawahara, Jo Estill, O. Fujimura, "Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system STRAIGHT, " in Proc. Int. Workshop Models Anal. Vocal Emissions Biomed. Appl., Firentze, Italy, Sep. 2001, pp. 1-6.
48. Y. Ohtani, T. Toda, H. Saruwatari, K. Shikano, "Maximum likelihood voice conversion based on GMM with STRAIGHT mixed excitation, " in Proc. INTERSPEECH, Pittsburgh, PA, USA, Sep. 2006, pp. 2266-2269.
49. H. Kawahara, I. Masuda-Katsuse, A. D. Cheveigne, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds, " Speech Commun., vol. 27, no. 3/4, pp. 187-207, Apr. 1999.
50. S. Takamichi, K. Kobayashi, K. Tanaka, T. Toda, S. Nakamura, "The NAIST text-to-speech system for the Blizzard Challenge 2015, " in Proc. Blizzard Challenge Workshop, Berlin, Germany, Sep. 2015.
51. T. Toda, T. Muramatsu, H. Banno, "Implementation of computationally efficient real-time voice conversion, " in Proc. INTERSPEECH, Portland, OR, USA, Sep. 2012, pp. 94-97.
52. M. Sahidullah, T. Kinnunen, C. Hanilçi, "A comparison of features for synthetic speech detection, " in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 2087-2091.
53. B. Bollepalli, L. Juvela, P. Alku, "Generative adversarial networkbased glottal waveform model for statistical parametric speech synthesis, " in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 3394-3398.