Deep bidirectional LSTM modeling of timbre and prosody for emotional voice conversion

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 08-12-September-2016, Issue , 2016, Pages 2453-2457

  Ming, Huaiping ^a   Huang, Dongyan ^a   Xie, Lei ^b   Wu, Jie ^b   Dong, Minghui ^a   Li, Haizhou ^a  

^a INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

^b NORTHWESTERN POLYTECHNICAL UNIVERSITY   (China)

Author keywords

Long short term memory; Prosody; Recurrent neural networks; Voice conversion

Indexed keywords

BRAIN; RECURRENT NEURAL NETWORKS; SPEECH COMMUNICATION; WAVELET TRANSFORMS;

CONTINUOUS WAVELET TRANSFORMS; CONVERSION METHODS; EMOTIONAL VOICES; ENERGY CONTOURS; LONG SHORT TERM MEMORY; PROSODY; PROSODY FEATURES; VOICE CONVERSION;

SPEECH PROCESSING;

EID: 84994251909     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: 10.21437/Interspeech.2016-1053     Document Type: Conference Paper

References (30)

1. J. A. Russell, J. A. Bachorowski and J. M. Fernández-Dols, (2003). Facial and vocal expressions of emotion. Annual review of psychology, 54(1), 329-349.
2. R.W. Picard and R. Picard, (1997). Affective computing (Vol. 252). Cambridge: MIT press.
3. D. L. Schacter, (2011). Psychology Second Edition, 41 Madison Avenue, New York, NY 10010.
4. K. R. Scherer, (2003). Vocal communication of emotion: A review of research paradigms. Speech communication, 40(1), 227-256.
5. D. Hirst, and A. Di Cristo, (1998). Intonation systems: a survey of twenty languages. Cambridge University Press.
6. J. Tao, Y. Kang and A. Li, (2006). Prosody conversion from neutral speech to emotional speech. IEEE Transactions on Audio, Speech, and Language Processing, 14(4), 1145-1154.
7. Z. Inanoglu and S. Young, (2007, August). A system for transform- ing the emotion in speech: combining data-driven conversion tech- niques for prosody and voice quality. In INTERSPEECH (pp. 490- 493).
8. R. Aihara, R. Takashima, T. Takiguchi and Y. Ariki, (2012). GMM- based emotional voice conversion using spectrum and prosody fea- tures. In American Journal of Signal Processing, 2(5), 134-138.
9. R. Aihara, R. Ueda, T. Takiguchi and Y. Ariki, (2014, December). Exemplar-based emotional voice conversion using non-negative matrix factorization. In Asia-Pacific Signal and Information Pro- cessing Association, 2014 Annual Summit and Conference (APSI- PA) (pp. 1-7). IEEE.
10. H. Ming, D. Huang, L. Xie, S. Zhang, M. Dong and H. Li, (2015). Fundamental frequency modeling using wavelets for emo- tional voice conversion. In 6th Affective Computing and Intelligent Interaction (ACII)Workshop on Affective Social Multimedia Com- puting.
11. K. Yu, (2012, October). Review of F0 modelling and generation in HMM based speech synthesis. In IEEE 11th International Con- ference on Signal Processing (ICSP), (Vol. 1, pp. 599-604).
12. C. H. Wu, C. C. Hsia, C. H. Lee and M. C. Lin, (2010). Hier- archical prosody conversion using regression-based clustering for emotional speech synthesis. IEEE Transactions on Audio, Speech, and Language Processing, 18(6), 1394-1405.
13. A. Wennerstrom, (2001). The music of everyday speech: Prosody and discourse analysis. Oxford University Press.
14. Y. Xu, (2011). Speech prosody: A methodological review. Journal of Speech Sciences, 1(1), 85-115.
15. J. J. O. H. Kawasaki-Fukumori, (1997). Alternatives to the sonor- ity hierarchy for explaining segmental sequential constraints. In Language and its ecology: Essays in memory of Einar Haugen, 100, 343.
16. M. S. Ribeiro and R. A.Clark (2015, April). A multi-level rep- resentation of f0 using the continuous wavelet transform and the discrete cosine transform. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (pp. 4909- 4913).
17. J. Latorre and M. Akamine, (2008, September). Multilevel parametric-base F0 model for speech synthesis. In INTERSPEECH (pp. 2274-2277).
18. Y. Qian, Z. Wu, B. Gao and F. K. Soong, F. K. (2011). Improved prosody generation by maximizing joint probability of state and longer units. IEEE Transactions on Audio, Speech, and Language Processing, 19(6), 1702-1710.
19. N. Obin, A. Lacheret and X. Rodet, (2011, August). Stylization and trajectory modelling of short and long term speech prosody variations. In INTERSPEECH.
20. M. Vainio, A. Suni and D. Aalto, (2013). Continuous wavelet transform for analysis of speech prosody. In Tools and Resources for the Analysys of Speech Prosody, an INTERSPEECH 2013 satellite event.
21. A. S. Suni, D. Aalto, T. Raitio, P. Alku and M. Vainio, (2013). Wavelets for intonation modeling in HMM speech synthesis. In 8th ISCA Workshop on Speech Synthesis, Proceedings, Barcelona.
22. H. Ming, D. Huang, L. Xie, S. Zhang, M. Dong and H. Li, (2016). Exemplar-based Sparse Representation of Timbre and Prosody for Voice Conversion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
23. F. A. Gers, and J. Schmidhuber, (2001). LSTM recurrent network- s learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks, 12(6), 1333-1340.
24. H. Sak, A. W. Senior, and F. Beaufays, (2014, September). Long short-term memory recurrent neural network architectures for large vocabulary speech recognition. In INTERSPEECH (pp. 338-342).
25. Y. Fan, Y. Qian, F. L., Xie and F. K. Soong, (2014, September). TTS synthesis with bidirectional LSTM based recurrent neural net- works. In INTERSPEECH (pp. 1964-1968).
26. L. Sun, S. Kang, K. Li and H. Meng, (2015, April). Voice con- version using deep Bidirectional Long Short-Term Memory based Recurrent Neural Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (pp. 4869- 4873).
27. H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino and H. Banno, (2008, March). TANDEM-STRAIGHT: A temporally stable power spectral representation for periodic signals and appli- cations to interference-free spectrum, F0, and aperiodicity estima- tion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (pp. 3933-3936).
28. J. Kominek and A. W. Black, (2004). The CMU Arctic speech databases. In Fifth ISCA Workshop on Speech Synthesis.
29. S. Liu, D. Y. Huang,W. Lin, M. Dong, H. Li and E. P. Ong, (2014, December). Emotional facial expression transfer based on tempo- ral restricted Boltzmann machines. In Asia-Pacific Signal and In- formation Processing Association (APSIPA).
30. F. Eyben, J. Bergmann, and F. Weninger. CURRENNT CUDA- enabled Machine Learning Library For Recurrent Neural Network- s. https://sourceforge.net/projects/currennt/