Voice conversion using RNN pre-trained by recurrent temporal restricted boltzmann machines

IEEE Transactions on Audio, Speech and Language Processing

Volumn 23, Issue 3, 2015, Pages 580-587

  Nakashika, Toru ^a   Takiguchi, Tetsuya ^a   Ariki, Yasuo ^a  

^a KOBE UNIVERSITY   (Japan)

Author keywords

Deep Learning; recurrent neural network; recurrent temporal restricted Boltzmann machine (RTRBM); speaker specific features; voice conversion

Indexed keywords

RECURRENT NEURAL NETWORKS;

DEEP LEARNING; GAUSSIAN MIXTURE MODEL; NEURAL NETWORK (NN); OBJECTIVE CRITERIA; PROBABILISTIC MODELS; RESTRICTED BOLTZMANN MACHINE; SPEAKER-SPECIFIC FEATURES; VOICE CONVERSION;

SPEECH PROCESSING;

EID: 84923867813     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2014.2379589     Document Type: Article

References (42)

1. A. Kain and M. W. Macon, "Spectral voice conversion for text-to-speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1998, pp. 285-288.
2. C. Veaux and X. Robet, "Intonation conversion from neutral to expressive speech," in Proc. Interspeech, 2011, pp. 2765-2768.
3. K. Nakamura, T. Toda, H. Saruwatari, and K. Shikano, "Speaking-aid systems using GMM-based voice conversion for electrolaryngeal speech," Speech Commun., vol. 54, no. 1, pp. 134-146, 2012.
4. L. Deng, A. Acero, L. Jiang, J. Droppo, and X. Huang, "High-performance robust speech recognition using stereo training data," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2001, pp. 301-304.
5. A. Kunikoshi, Y. Qiao, N. Minematsu, and K. Hirose, "Speech generation from hand gestures based on space mapping," in Proc. Interspeech, 2009, pp. 308-311.
6. M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, "Voice conversion through vector quantization," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1988, pp. 655-658.
7. H. Valbret, E. Moulines, and J.-P. Tubach, "Voice transformation using PSOLA technique," Speech Commun., vol. 11, no. 2, pp. 175-187, 1992.
8. Y. Stylianou, O. Cappé, and E. Moulines, "Continuous probabilistic transform for voice conversion," IEEE Trans. Speech Audio Process., vol. 6, no. 2, pp. 131-142, Mar. 1998.
9. C.-H. Lee and C.-H. Wu, "MAP-based adaptation for speech conversion using adaptation data selection and non-parallel training," in Proc. Interspeech, 2006, pp. 2254-2257.
10. T. Toda, Y. Ohtani, and K. Shikano, "Eigenvoice conversion based on gaussian mixture model," in Proc. Interspeech, 2006, pp. 2446-2449.
11. D. Saito, K. Yamamoto, N. Minematsu, and K. Hirose, "One-to-many voice conversion based on tensor representation of speaker space," in Proc. Interspeech, 2011, pp. 653-656.
12. D. Saito, S. Watanabe, A. Nakamura, and N. Minematsu, "Probabilistic integration of joint density model and speaker model for voice conversion,"in Proc. Interspeech, 2010, pp. 1728-1731.
13. S. Desai, E. V. Raghavendra, B. Yegnanarayana, A. W. Black, and K. Prahallad, "Voice conversion using artificial neural networks," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2009, pp. 3893-3896.
14. Y.-J. Wu, H. Kawai, J. Ni, and R.-H. Wang, "Minimum segmentation error based discriminative training for speech synthesis application,"in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2004, pp. 629-632.
15. E. McDermott, T. J. Hazen, J. Le Roux, A. Nakamura, and S. Katagiri, "Discriminative training for large-vocabulary speech recognition using minimum classification error," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 1, pp. 203-223, Jan. 2007.
16. T. Tomoki and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol. 90, no. 5, pp. 816-824, 2007.
17. Z.-H. Ling and L.-R. Dai, "Minimum kullback-leibler divergence parameter generation for HMM-based speech synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 5, pp. 1492-1502, Jul. 2012.
18. Z.-H. Ling, Y.-J. Wu, Y.-P. Wang, L. Qin, and R.-H. Wang, "USTC system for blizzard challenge 2006. An improved HMM-based speech synthesis method," in Proc. Blizzard Challenge Workshop, 2006.
19. Z. Jian and Z. Yang, "Voice conversion using canonical correlation analysis based on gaussian mixture model," in Proc. 8thACIS Int. Conf. IEEE Software Eng., Artif. Intell., Netw., Parallel/Distrib. Comput. (SNPD '07), 2007, pp. 210-215.
20. R. Takashima, T. Takiguchi, and Y. Ariki, "Exemplar-based voice conversion in noisy environment," in Proc. IEEE Spoken Lang. Technol. Workshop (SLT), 2012, pp. 313-317.
21. Z. Wu, T. Virtanen, T. Kinnunen, E. S. Chng, and H. Li, "Exemplar-based voice conversion using non-negative spectrogram deconvolution,"in Proc. 8th ISCA Speech Synth. Workshop, 2013.
22. T. Nakashika, R. Takashima, T. Takiguchi, and Y. Ariki, "Voice conversion in high-order eigen space using deep belief nets," in Proc. Interspeech, 2013, pp. 369-372.
23. P. Smolensky, "Information processing in dynamical systems: Foundations of harmony theory," Parallel Distrib. Process., vol. 1, 1986.
24. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006.
25. Z.-H. Ling, L. Deng, and D. Yu, "Modeling spectral envelopes using restricted Boltzmann machines and deep belief networks for statistical parametric speech synthesis," IEEE Trans. Audio, Speech, Lang. Process., no. 10, pp. 2129-2139, Oct. 2013.
26. A.-r. Mohamed, G. E. Dahl, and G. Hinton, "Acoustic modeling using deep belief networks," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 14-22, Jan. 2012.
27. V. Nair and G. Hinton, "3-d object recognition with deep belief nets,"Adv. Neural Inf. Process. Syst., vol. 22, pp. 1339-1347, 2009.
28. T. Deselaers, S. Hasan, O. Bender, and H. Ney, "A deep learning approach to machine transliteration," in Proc. 4th Workshop Statist. Mach. Translat. Assoc. Comput. Linguist., 2009, pp. 233-241.
29. I. Sutskever, G. Hinton, and G. Taylor, "The recurrent temporal restricted Boltzmann machine," NIPS, vol. 19, pp. 1601-1608, 2008.
30. N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription," in Proc. Int. Conf. Mach. Learn., 2012.
31. Z. Wu, E. S. Chng, and H. Li, "Conditional restricted Boltzmann machine for voice conversion," in Proc. IEEE China Summit and Int. Conf. Signal Inf. Process. (ChinaSIP), 2013, pp. 104-108.
32. L.-H. Chen, Z.-H. Ling, Y. Song, and L.-R. Dai, "Joint spectral distribution modeling using restricted Boltzmann machines for voice conversion,"in Proc. Interspeech, 2013, pp. 3052-3056.
33. G. W. Taylor, G. E. Hinton, and S. T. Roweis, "Modeling human motion using binary latent variables," Adv. Neural Inf. Process. Syst., pp. 1345-1352, 2006.
34. Y. Freund and D. Haussler, Unsupervised Learning of Distributions of Binary Vectors Using Two Layer Networks. Santa Cruz, CA, USA: Computer Research Laboratory, 1994.
35. K. Cho, A. Ilin, and T. Raiko, "Improved learning of gaussian-bernoulli restricted Boltzmann machines," Artif. Neur. Netw. Mach. Learn., pp. 10-17, 2011.
36. R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks," arXiv preprint arXiv:1211.5063, 2012.
37. T. Toda, A. W. Black, and K. Tokuda, "Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory,"IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov. 2007.
38. A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri, H. Kuwabara, and K. Shikano, "ATR japanese speech database as a tool of speech recognition and synthesis," Speech Commun., vol. 9, no. 4, pp. 357-363, 1990.
39. H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino, and H. Banno, "TANDEM-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, f0, and aperiodicity estimation," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2008, pp. 3933-3936.
40. B. Milner and X. Shao, "Speech reconstruction from mel-frequency cepstral coefficients using a source-filter model," in Proc. Interspeech, 2002, pp. 2421-2424.
41. A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," Comput. Sci. Dept., Univ. of Toronto, Toronto, ON, USA, Tech. Rep., 2009.
42. T. Nakashika, T. Takiguchi, and Y. Ariki, "Voice conversion in time-invariant speaker-independent space," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2014, pp. 7939-7943.