Improving Trajectory Modelling for DNN-Based Speech Synthesis by Using Stacked Bottleneck Features and Minimum Generation Error Training

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 24, Issue 7, 2016, Pages 1255-1265

  Wu, Zhizheng ^a   King, Simon ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

acoustic modelling; bottleneck; deep neural network; Minimum generation error; Speech synthesis

Indexed keywords

COMPUTATIONAL LINGUISTICS; LINGUISTICS; RECURRENT NEURAL NETWORKS; SPEECH SYNTHESIS;

ACOUSTIC MODELLING; BOTTLENECK; DEEP NEURAL NETWORKS; IMPROVE PERFORMANCE; LINGUISTIC REPRESENTATIONS; LONG SHORT TERM MEMORY; MINIMUM GENERATION ERRORS; TRAJECTORY MODELLING;

ERRORS;

EID: 84978086501     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2016.2551865     Document Type: Article

References (49)

1. H. Zen, K. Tokuda, A. W. Black, "Statistical parametric speech synthesis," Speech Commun., vol. 51, no. 11, pp. 1039-1064, 2009.
2. S. King, "Measuring a decade of progress in text-to-speech," Loquens, vol. 1, no. 1, 2014.
3. A. J. Hunt and A. W. Black, "Unit selection in a concatenative speech synthesis system using a large speech database," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., vol. 1, 1996, pp. 373-376.
4. Y.-J. Wu and R.-H. Wang, "Minimum generation error training for HMMbased speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2006, pp. 89-92.
5. H. Zen, K. Tokuda, T. Kitamura, "Reformulating the HMM as a trajectory model by imposing explicit relationships between static and dynamic feature vector sequences," Comput. Speech Language, vol. 21, no. 1, pp. 153-173, 2007.
6. T. Toda and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inform. Syst., vol. 90, no. 5, pp. 816-824, 2007.
7. S. Takamichi, T. Toda, G. Neubig, S. Sakti, S. Nakamura, "A postfilter to modify the modulation spectrum in HMM-based speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2014, pp. 290-294.
8. T. Merritt, J. Latorre, S. King, "Attributing modelling errors in HMM synthesis by stepping gradually from natural tomodelled speech," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 4220-4224.
9. G. Hinton et al.,"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
10. Z.-H. Ling et al., "Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends," IEEE Signal Process. Mag., vol. 32, no. 3, pp. 35-52, May 2015.
11. O. Karaali, G. Corrigan, N. Massey, C. Miller, O. Schnurr, A. Mackie, "A high quality text-to-speech system composed of multiple neural networks," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., vol. 2, 1998, pp. 1237-1240.
12. T. Weijters and J. Thole, "Speech synthesiswith artificial neural networks," in Proc. Int. Conf. Neural Netw., 1993, pp. 1764-1769.
13. G. Cawley and P. Noakes, "LSP speech synthesis using backpropagation networks," in Proc. 3rd Int. Conf. Artif. Neural Netw., 1993, pp. 291-294.
14. C. Tuerk and T. Robinson, "Speech synthesis using artificial neural networks trained on cepstral coefficients." in Proc. Eur. Conf. Speech Commun. Technol., 1993, pp. 4-7.
15. M. Riedi, "A neural-network-based model of segmental duration for speech synthesis," in Proc. Eur. Conf. Speech Commun. Technol., 1995, pp. 599-602.
16. Z.-H. Ling, L. Deng, D. Yu, "Modeling spectral envelopes using restricted Boltzmann machines and deep belief networks for statistical parametric speech synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 10, pp. 2129-2139, Oct. 2013.
17. S. Kang, X. Qian, H. Meng, "Multi-distribution deep belief network for speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process, 2013, pp. 8012-8016.
18. S. Kang and H. Meng, "Statistical parametric speech synthesis using weighted multi-distribution deep belief network," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2014, pp. 1959-1963.
19. H. Zen and A. Senior, "Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2014, pp. 3844-3848.
20. B. Uria, I. Murray, S. Renals, C. Valentini, "Modelling acoustic feature dependencies with artificial neural networks: Trajectoryrnade," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 4465-4469.
21. L.-H. Chen, T. Raitio,C. Valentini-Botinhao, Z.-H. Ling, J. Yamagishi, "A deep generative architecture for postfiltering in statistical parametric speech synthesis," IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 23, no. 11, pp. 2003-2014, Nov. 2015.
22. H. Zen, A. Senior, M. Schuster, "Statistical parametric speech synthesis using deep neural networks," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2013, pp. 7962-7966.
23. H. Lu, S. King, O. Watts, "Combining a vector space representation of linguistic contextwith a deep neural network for text-to-speech synthesis," in Proc. 8th Int. Speech Commun. Assoc. Speech Synthesis Workshop, 2013, pp. 281-285.
24. Y. Qian, Y. Fan,W. Hu, F. K. Soong, "On the training aspects of deep neural network (DNN) for parametric TTS synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2014, pp. 3829-3833.
25. Z. Wu, C. Valentini-Botinhao, O. Watts, S. King, "Deep neural networks employing multi-task learning and stacked bottleneck features for speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 4460-4464.
26. O. Watts, G. E. Henter, T. Merritt, Z. Wu, S. King, "From HMMs to DNNs:Where do the improvements come from?" in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2016.
27. S. King, "A reading list of recent advances in speech synthesis," in Proc. 18th Int. Congr. Phon. Sci., 2015.
28. C. Valentini-Botinhao, Z.Wu, S. King, "Towards minimum perceptual error training for DNN-based speech synthesis," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 869-873.
29. Q. Hu, Z. Wu, K. Richmond, J. Yamagishi, Y. Stylianou, R. Maia, "Fusion of multiple parameterisations for DNN-based sinusoidal speech synthesis with multi-task learning," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 854-858.
30. Y. Fan, Y. Qian, F. Xie, F. K. Soong, "TTS synthesis with bidirectional LSTM based recurrent neural networks," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2014, pp. 1964-1968.
31. H. Zen and H. Sak, "Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 4470-4474.
32. Z. Wu and S. King, "Investigating gated recurrent neural networks for speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2016.
33. K. Tokuda, H. Zen, A. W. Black, "An HMM-based speech synthesis system applied to english," in Proc. IEEE Workshop Speech Synthesis, 2002, pp. 227-230.
34. Z. Wu and S. King, "Minimum trajectory error training for deep neural networks, combined with stacked bottleneck features," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 309-313.
35. Y. Fan, Y. Qian, F. K. Soong, L. He, "Sequence generation error (SGE) minimization based deep neural networks training for text-tospeech synthesis," in Proc. Annu. Conf. Int. SpeechCommun. Assoc., 2015, pp. 864-868.
36. F.-L. Xie, Y. Qian, Y. Fan, F. K. Soong, H. Li, "Sequence error (SE) minimization training of neural network for voice conversion," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2014, pp. 2283-2287.
37. K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, T. Kitamura, "Speech parameter generation algorithms for HMM-based speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2000, vol. 3, pp. 1315-1318.
38. D. E. Rumelhart, G. E. Hinton, R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, no. 6088, pp. 533-536, 1986.
39. G. Hinton, "A practical guide to training restricted Boltzmann machines," Univ. Toronto, Toronto, ON, Canada, Tech. Rep. UTML TR 2010-003, 2010.
40. Y. Chen, Z.-J. Yan, F. K. Soong, "A perceptual study of acceleration parameters in HMM-based TTS," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2010, pp. 426-429.
41. K. Hashimoto, K. Oura, Y. Nankaku, K. Tokuda, "The effect of neural networks in statistical parametric speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process, 2015, pp. 4455-4459.
42. F. Grézl and P. Fousek, "Optimizing bottle-neck features for LVCSR," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2008, pp. 4729-4732.
43. D. Yu and M. L. Seltzer, "Improved bottleneck features using pretrained deep neural networks," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2011, pp. 237-240.
44. T. N. Sainath, B. Kingsbury, B. Ramabhadran, "Auto-encoder bottleneck features using deep belief networks," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2012, pp. 4153-4156.
45. H. Kawahara, I. Masuda-Katsuse, A. de Cheveigné, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Commun., vol. 27, no. 3, pp. 187-207, 1999.
46. C. Veaux, J. Yamagishi, S. King, "The voice bank corpus: Design, collection and data analysis of a large regional accent speech database," in Proc. Int. Conf. Oriental COCOSDA, 2013, pp. 1-4.
47. D. B. Paul and J. M. Baker, "The design for the Wall Street journalbased CSR corpus," in Proc. Workshop Speech Natural Lang., 1992, pp. 357-362.
48. T. Merritt, J. Yamagishi, Z. Wu, O. Watts, S. King, "Deep neural network context embeddings for model selection in rich-context HMM synthesis," in Proc. Annu. Conf. Int. Speech Commun. Assoc., 2015, pp. 2207-2211.
49. T. Merritt, R. A. Clark, Z. Wu, J. Yamagishi, S. King, "Deep neural network-guided unit selection synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2016.