Voice conversion using deep neural networks with layer-wise generative training

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 22, Issue 12, 2014, Pages 1859-1872

  Chen, Ling Hui ^a   Ling, Zhen Hua ^a   Liu, Li Juan ^a   Dai, Li Rong ^a  

^a National Engineering Laboratory for Speech and Language Information Processing   (China)

Author keywords

Bidirectional associative memory; Deep neural network; Gaussian mixture model; Restricted Boltzmann machine; Spectral envelope conversion; Voice conversion

Indexed keywords

ASSOCIATIVE PROCESSING; ASSOCIATIVE STORAGE; GAUSSIAN DISTRIBUTION; IMAGE SEGMENTATION; MAPPING; NETWORK LAYERS; SPEECH PROCESSING;

BI-DIRECTIONAL ASSOCIATIVE MEMORY; DEEP NEURAL NETWORKS; GAUSSIAN MIXTURE MODEL; RESTRICTED BOLTZMANN MACHINE; SPECTRAL ENVELOPES; VOICE CONVERSION;

PHOTOMAPPING;

EID: 84921735339     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2014.2353991     Document Type: Article

References (43)

1. M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, "Voice conversion through vector quantization," in Proc. ICASSP, 1988, pp.655-658.
2. H. Doi, T. Toda, K. Nakamura, H. Saruwatari, and K. Shikano, "Alaryngeal speech enhancement based on one-to-many eigenvoice conversion," IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 22, no. 1, pp. 172-183, Jan. 2014.
3. T. L. New, M. Dong, P. Chan, X. Wang, B. Ma, and H. Li, "Voice conversion: From spoken vowels to singing vowels," in Proc. IEEE Int. Conf. Multimedia Expo (ICME), 2010, pp. 1421-1426.
4. T. Toda, M. Nakagiri, and K. Shikano, "Statistical voice conversion techniques for body-conducted unvoiced speech enhancement," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 9, pp. 2505-2517, Nov.. 2012.
5. Z. W. Shuang, R. Bakis, S. Shechtman, D. Chazan, and Y. Qin, "Frequency warping based on mapping formant parameters," in Proc. Interspeech, 2006, pp. 2290-2293.
6. D. Erro and A. Moreno, "Weighted frequency warping for voice conversion," in Proc. Interspeech, 2007, pp. 1965-1968.
7. Y. Stylianou, O. Cappe, and E. Moulines, "Continuous probabilistic transform for voice conversion," IEEE Trans. Speech Audio Process., vol. 6, no. 2, pp. 131-142, Mar. 1998.
8. A. Kain and M. Macon, "Spectral voice conversion for text-to-speech synthesis," in Proc. ICASSP, 1998, pp. 285-288.
9. H. Ye and S. Young, "Quality-enhanced voice morphing using maximum likelihood transformations," IEEE Trans. Audio, Speech, Lang.Process., vol. 14, no. 4, pp. 1301-1312, Jul. 2006.
10. Y. Chen,M. Chu, E. Chang, J. Liu, and R. Liu, "Voice conversion with smoothed GMM and MAP adaptation," Eurospeech, pp. 2413-2416, 2003.
11. T. Toda, A. Black, and K. Tokuda, "Voice conversion based on maximum- likelihood estimation of spectral parameter trajectory," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov. 2007.
12. H. Zen, Y. Nankaku, and K. Tokuda, "Continuous stochastic feature mapping based on trajectory HMMs," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 2, pp. 417-430, Feb. 2011.
13. T. Dutoit, A. Holzapfel, M. Jottrand, A. Moinet, J. Perez, and Y.Stylianou, "Towards a voice conversion system based on frame selection," in Proc. ICASSP, 2007, vol. 4, pp. IV-513-IV-516.
14. Z.Wu, T. Virtanen, T. Kinnunen, E. Chng, and H. Li, "Exemplar-based unit selection for voice conversion utilizing temporal information," in Proc. Interspeech, 2013.
15. E. Helander, H. Silen, T. Virtanen, and M. Gabbouj, "Voice conversion using dynamic kernel partial least squares regression," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 3, pp. 806-817, Mar. 2012.
16. N. Pilkington, H. Zen, and M. J. F. Gales, "Gaussian process experts for voice conversion," in Proc. Interspeech, 2011, pp. 2761-2764.
17. S. Desai, A. Black,B.Yegnanarayana, andK. Prahallad, "Spectralmapping using artificial neural networks for voice conversion," IEEE Trans.Audio, Speech, Lang. Process., vol. 18, no. 5, pp. 954-964, Jul. 2010.
18. T. Nakashika, T. Takashima, R. Takiguchi, and Y. Ariki, "Voice conversion in high-order eigen space using deep belief nets," in Proc. Interspeech, 2013, pp. 369-372.
19. Z. Wu, E. S. Chng, and H. Li, "Conditional restricted Boltzmann machine for voice conversion," in Proc. IEEE China Summit Int. Conf.Signal Inf. Process. (ChinaSIP), 2013, pp. 104-108.
20. L.-H. Chen, Z.-H. Ling, Y. Song, and L.-R. Dai, "Joint spectral distribution modeling using restricted Boltzmann machines for voice conversion," in Proc. Interspeech, 2013, pp. 3052-3056.
21. L.-J. Liu, L.-H. Chen, Z.-H. Ling, and L.-R. Dai, "Using bidirectional associative memories for joint spectral envelope modeling in voice conversion," in Proc. ICASSP, 2014, pp. 7884-7888.
22. Z.-H. Ling, L. Deng, and D. Yu, "Modeling spectral envelopes using restricted Boltzmann machines for statistical parametric speech synthesis," in Proc. ICASSP, 2013, pp. 7825-7829.
23. G. Hinton, "Training products of experts by minimizing contrastive divergence," Neural Comput., vol. 12, no. 14, pp. 1711-1800, 2002.
24. Z.-H. Ling, L. Deng, and D. Yu, "Modeling spectral envelopes using restricted Boltzmann machines and deep belief networks for statistical parametric speech synthesis," IEEE Trans. Audio, Speech, Lang.Process., vol. 21, no. 10, pp. 2129-2139, Oct. 2013.
25. Y. Tang and R. Salakhutdinov, "Learning stochastic feedforward neural networks," in Advances in Neural Information Processing Systems 26. Cambridge, MA, USA: MIT Press, 2013, pp. 530-538.
26. S. Kang, X. Qian, and H. Meng, "Multi-distribution deep belief network for speech synthesis," in Proc. ICASSP, 2013, pp. 8012-8016.
27. P. Smolensky, "Information processing in dynamical systems: Foundations of harmony theory," in Parallel distributed processing: explorations in the microstructure of cognition, D. E. Rumelhart and J. L.McClelland, Eds. Cambridge, MA, USA: MIT Press, 1986, vol. 1, ch. 6, pp. 194-281.
28. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
29. R. Salakhutdinov, "Learning deep generative models," Ph.D. dissertation, Univ. of Toronto, Toronto, ON, Canada, 2009.
30. B. Kosko, "Bidirectional associative memories," IEEE Trans. Systems, Man, Cybern., vol. 18, no. 1, pp. 49-60, Jan. 1988.
31. H. Oh and S. C. Kothari, "A pseudo-relaxation learning algorithm for bidirectional associativememory," in Proc. Int. Joint Conf. Neural Networks (IJCNN'92), 1992, vol. 2, pp. 208-213.
32. M. Hattori, M. Hagiwara, and M. Nakagawa, "Quick learning for bidirectional associative memory," IEICE Trans. Inf. Syst., vol. 77, no. 4, pp. 385-392, 1994.
33. H. Zen, K. Tokuda, and T. Kitamura, "Reformulating the HMM as a trajectory model by imposing explicit relationships between static and dynamic feature vector sequences," Comput. Speech Lang., vol. 21, no.1, pp. 153-173, 2007.
34. T. Stafylakis, P. Kenny, M. Senoussaoui, and P. Dumouchel, "PLDA using Gaussian restricted Boltzmann machines with application to speaker verification," in Proc. Interspeech, 2012.
35. Y. Bengio, "Learning deep architectures for AI," Foundat. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, Jan. 2009.
36. G. E. Hinton, "A practical guide to training restricted Boltzmann machines," in Neural Networks: Tricks of the Trade. NewYork,NY, USA: Springer, 2012, vol. 7700, pp. 599-619.
37. G. E. Hinton, J. L. McClelland, and D. E. Rumelhart, "Distributed representations," in Parallel distributed processing: explorations in the microstructure of cognition, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA, USA: MIT Press, 1986, vol. 1, ch. 6, pp.77-109.
38. R. Salakhutdinov and G. E. Hinton, "Deep Boltzmann machines," in Proc. Int. Conf. Artif. Intell. Statist., 2009, pp. 448-455.
39. H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigné, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Commun., vol.27, no. 3, pp. 187-208, 1999.
40. G. E. Hinton, "Products of experts," in Proc. 9th Int. Conf. Artif. Neural Netw. (ICANN '99) (Conf. Publ. No. 470), 1999, vol. 1, pp. 1-6, IET.
41. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R.Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors," arXiv preprint arXiv:1207.0580 2012.
42. T. Toda and K. Tokuda, "A speech parameter generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol. 90, no. 5, pp. 816-824, 2007.
43. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pretrained deep neural networks for large-vocabulary speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012.