Deep Learning for Acoustic Modeling in Parametric Speech Generation: A systematic review of existing techniques and future trends

IEEE Signal Processing Magazine

Volumn 32, Issue 3, 2015, Pages 35-52

  Ling, Zhen Hua ^a   Kang, Shi Yin ^b   Zen, Heiga ^c   Senior, Andrew ^d   Schuster, Mike ^e   Qian, Xiao Jun ^f   Meng, Helen ^g   Deng, Li ^h  

^a UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

^c NAGOYA INSTITUTE OF TECHNOLOGY   (Japan)

^d UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^e UNIVERSITY OF DUISBURG ESSEN   (Germany)

^f FUDAN UNIVERSITY   (China)

^g Department of Systems Engineering and Engineering Management ^*  (China)

^h Deep Learning Technology Center   (China)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; HIDDEN MARKOV MODELS; MARKOV PROCESSES; SPEECH; TRELLIS CODES;

ACOUSTIC FEATURES; AUTOMATIC SPEECH RECOGNITION; DEEP NEURAL NETWORKS; GAUSSIAN MIXTURE MODEL (GMMS); HIDDEN MARKOV MODELS (HMMS); HIERARCHICAL PROCESS; NON-LINEAR RELATIONSHIPS; PARAMETRIC APPROACH;

SPEECH RECOGNITION;

EID: 85032750981     PISSN: 10535888     EISSN: None     Source Type: Journal    
DOI: 10.1109/MSP.2014.2359987     Document Type: Review

References (86)

1. K. Tokuda, Y. Nankaku, T. Toda, H. Zen, H. Yamagishi, and K. Oura, "Speech synthesis based on hidden Markov models," Proc. IEEE, vol. 101, no. 5, pp. 1234-1252, 2013.
2. T. Toda, A. Black, and K. Tokuda, "Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory," IEEE Trans. Audio Speech Lang. Process., vol. 15, no. 8, pp. 2222-2235, 2007.
3. H. Zen, T. Toda, M. Nakamura, and K. Tokuda, "Details of Nitech HMM-based speech synthesis system for the Blizzard Challenge 2005," IEICE Trans. Inf. Syst., vol. E90-D, no. 1, pp. 325-333, 2007.
4. Z.-H. Ling, Y.-J. Wu, Y.-P. Wang, L. Qin, and R.-H. Wang, "USTC system for Blizzard Challenge 2006: An improved HMM-based speech synthesis method," in Proc. Blizzard Challenge Workshop, 2006.
5. H. Zen, K. Tokuda, and A. Black, "Statistical parametric speech synthesis,"Speech Commun., vol. 51, no. 11, pp. 1039-1064, 2009.
6. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,"in Proc. Eurospeech, 1999, pp. 2347-2350.
7. K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, "Speech parameter-generation algorithms for HMM-based speech synthesis," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2000, vol. 3, pp. 1315-1318.
8. H. Zen, K. Tokuda, and T. Kitamura, "Reformulating the HMM as a trajectory model by imposing explicit relationships between static and dynamic feature vector sequences," Comput. Speech Lang., vol. 21, no. 1, pp. 153-173, 2006.
9. H. Zen, M. Gales, Y. Nankaku, and K. Tokuda, "Product of experts for statistical parametric speech synthesis," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 3, pp. 794-805, 2012.
10. T. Koriyama, T. Nose, and T. Kobayashi, "Statistical parametric speech synthesis based on Gaussian process regression," IEEE J. Select. Topics Signal Processing, vol. 8, no. 2, pp. 173-183, 2014.
11. Y.-J. Wu and R.-H. Wang, "Minimum generation error training for HMM-based speech synthesis," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2006, pp. 89-92.
12. Y.-J. Wu and K. Tokuda, "Minimum generation error training with direct log spectral distortion on LSPs for HMM-based speech synthesis," in Proc. Interspeech, 2008, pp. 577-580.
13. T. Toda and K. Tokuda, "A speech parameter-generation algorithm considering global variance for HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 816-824, 2007.
14. T. Tiomkin, D. Malah, and S. Shechtman, "Statistical text-to-speech synthesis based on segment-wise representation with a norm constraint," IEEE Trans. Audio Speech Lang. Processing, vol. 18, no. 5, pp. 1077-1082, 2010.
15. Z.-H. Ling and L.-R. Dai, "Minimum Kullback-Leibler divergence parametergeneration for HMM-based speech synthesis," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 5, pp. 1492-1502, 2012.
16. G. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural Computat., vol. 18, no. 7, pp. 1527-1554, 2006.
17. G. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
18. R. Salakhutdinov, "Learning deep generative models," Ph.D. thesis, Univ. of Toronto, 2009.
19. P. Smolensky, "Information processing in dynamical systems: Foundations of harmony theory," in Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, vol. 1, ch. 6, pp. 194-281.
20. L. Deng, M. Seltzer, D. Yu, A. Acero, A. Mohamed, and G. Hinton, "Binary coding of speech spectrograms using a deep auto-encoder," in Proc. Interspeech, 2010, pp. 1692-1695.
21. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," J. Mach. Learn. Res., vol. 11, pp. 3371-3408, Dec. 2010.
22. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition," IEEE Signal Processing Mag., vol. 29, no. 6, pp. 82-97, 2012.
23. Z.-H. Ling, L. Deng, and D. Yu, "Modeling spectral envelopes using restricted Boltzmann machines for statistical parametric speech synthesis," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2013, pp. 7825-7829.
24. Z.-H. Ling, L. Deng, and D. Yu, "Modeling spectral envelopes using restricted Boltzmann machines and deep belief networks for statistical parametric speech synthesis,"IEEE Trans. Audio Speech Lang. Processing, vol. 21, no. 10, pp. 2129-2139, 2013.
25. S.-Y. Kang, X.-J. Qian, and H. Meng, "Multi-distribution deep belief network for speech synthesis," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2013, pp. 8012-8016.
26. H. Zen, A. Senior, and M. Schuster, "Statistical parametric speech synthesis using deep neural networks," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2013, pp. 7962-7966.
27. L.-H. Chen, Z.-H. Ling, Y. Song, and L.-R. Dai, "Joint spectral distribution modeling using restricted Boltzmann machines for voice conversion," in Proc. Interspeech, 2013, pp. 3052-3056.
28. T. Nakashika, R. Takashima, T. Takiguchi, and Y. Ariki, "Voice conversion in high-order eigen space using deep belief nets," in Proc. Interspeech, 2013, pp. 369-372.
29. Z.-Z Wu, E.S. Chng, and H.-Z. Li, "Conditional restricted Boltzmann machine for voice conversion," in Proc. ChinaSIP, 2013, pp. 104-108.
30. X. Lu, Y. Tsao, S. Matsuda, and C. Hori, "Speech enhancement based on deep denoising autoencoder," in Proc. Interspeech, 2013, pp. 436-440.
31. B.-Y. Xia and C.-C. Bao, "Speech enhancement with weighted denoising autoencoder,"in Proc. Interspeech, 2013, pp. 3444-3448.
32. Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, "An experimental study on speech enhancement based on deep neural networks," IEEE Signal Processing Lett., vol. 21, no. 1, pp. 65-68, 2014.
33. 0 contour prediction with a deep belief network-Gaussian process hybrid model," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2013, pp. 6885-6889.
34. H. Lu, S. King, and O. Watts, "Combining a vector space representation of linguistic context with a deep neural network for text-to-speech synthesis," in Proc. ISCA SSW8, 2013, pp. 261-265.
35. S.-Y. Kang and H. Meng, "Statistical parametric speech synthesis using weighted multi-distribution deep belief network," in Proc. Interspeech, 2014, pp. 1959-1963.
36. Y. Qian, Y.-C. Fan, W.-P. Hu, and F. K. Soong, "On the training aspects of deep neural networks (DNN) for parametric TTS synthesis," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 3857-3861.
37. H. Zen and A. Senior, "Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 3872-3876.
38. T. Nakashika, T. Takiguchi, and Y. Ariki, "Voice conversion in time-invariant speaker-independent space," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 7939-7943.
39. L.-H. Chen, Z.-H. Ling, L.-J. Liu, and L.-R. Dai, "Voice conversion using deep neural networks with layer-wise generative training," in Proc. IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 12, 2014, pp. 1859-1872.
40. Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, "Dynamic noise aware training for speech enhancement based on deep neural networks," in Proc. Interspeech (to be published).
41. K. Tokuda, T. Kobayashi, and S. Imai, "Speech parameter-generation from HMM using dynamic features," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 1995, pp. 660-663.
42. J. Yamagishi, T. Nose, H. Zen, Z.-H. Ling, T. Toda, K. Tokuda, S. King, and S. Renals, "Robust speaker-adaptive HMM-based text-to-speech synthesis," IEEE Trans. Audio Speech Lang. Processing, vol. 17, no. 6, pp. 1208-1230, 2009.
43. J. Yamagishi and T. Kobayashi, "Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training," IEICE Trans. Inf. Syst., vol. E90-D, no. 2, pp. 533-543, 2007.
44. J. Yamagishi, K. Onishi, T. Masuko, and T. Kobayashi, "Acoustic modeling of speaking styles and emotional expressions in HMM-based speech synthesis," IEICE Trans. Inf. Syst., vol. E88-D, no. 3, pp. 503-509, 2005.
45. M. Tachibana, J. Yamagishi, T. Masuko, and T. Kobayashi, "Speech synthesis with various emotional expressions and speaking styles by style interpolation and morphing," IEICE Trans. Inf. Syst., vol. E88-D, no. 11, pp. 2484-2491, 2005.
46. T. Nose, J. Yamagishi, and T. Kobayashi, "A style control technique for HMM-based expressive speech synthesis," IEICE Trans. Inf. Syst., vol. E90-D, no. 9, pp. 1406-1413, 2007.
47. L. Saheer, J. Dines, and P. N. Garner, "Vocal tract length normalization for statistical parametric speech synthesis," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 7, pp. 2134-2148, 2012.
48. H. Zen, N. Braunschweiler, S. Buchholz, M.J.F. Gales, K. Knill, S. Krstulovic, and J. Latorre, "Statistical parametric speech synthesis based on speaker and language factorization," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 6, pp. 1713-1724, 2012.
49. Z.-H. Ling, K. Richmond, and J. Yamagishi, "Articulatory control of HMM-based parametric speech synthesis using feature-space-switched multiple regression," IEEE Trans. Audio Speech Lang. Processing, vol. 21, no. 1, pp. 207-219, 2013.
50. K. Tokuda, H. Zen, and A. Black, "An HMM-based speech synthesis system applied to English," in Proc. IEEE Speech Synthetic Workshop, 2002, CD-ROM Proc.
51. K. Yu, H. Zen, F. Mairesse, and S. Young, "Context adaptive training with factorized decision trees for HMM-based statistical parametric speech synthesis,"Speech Commun., vol. 53, no. 6, pp. 914-923, 2011.
52. J. Odell, "The use of context in large vocabulary speech recognition," Ph.D. thesis, Cambridge Univ., 1995.
53. H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "A hidden semi-Markov model-based speech synthesis system," IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 825-834, 2007.
54. T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Duration modeling in HMM-based speech synthesis system," in Proc. ICSLP, 1998, vol. 2, pp. 29-32.
55. M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara, "Voice conversion through vector quantization," J. Acoust. Soc. Jpn. (E), vol. 11, no. 2, pp. 71-76, 1990.
56. Y. Stylianou, O. Cappe, and E. Moulines, "Continuous probabilistic transform for voice conversion," IEEE Trans. Audio Speech Lang. Processing, vol. 6, no. 2, pp. 131-142, 1998.
57. D. Erro, A. Moreno, and A. Bonafonte, "Voice conversion based on weighted frequency warping," IEEE Trans. Audio Speech Lang. Processing, vol. 18, no. 5, pp. 922-931, 2010.
58. S. Desai, A. W. Black, B. Yegnanarayana, and K. Prahallad, "Spectral mapping using artificial neural networks for voice conversion," IEEE Trans. Audio Speech Lang. Processing, vol. 18, no. 5, pp. 954-964, 2010.
59. E. Helander, H. Silen, T. Virtanen, and M. Gabbouj, "Voice conversion using dynamic kernel partial least squares regression," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 3, pp. 806C817, 2012.
60. D. Saito, S. Watanabe, A. Nakamura, and N. Minematsu, "Statistical voice conversion based on noisy channel model," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 6, pp. 1784-1794, 2012.
61. K. Park and H. Kim, "Narrowband to wideband conversion of speech using GMM based transformation," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2000, pp. 1843-1846.
62. A. Mouchtaris, J. Van der Spiegel, P. Mueller, and P. Tsakalides, "A spectral conversion approach to single-channel speech enhancement," IEEE Trans. Audio Speech Lang. Processing, vol. 15, no. 4, pp. 1180-1193, 2007.
63. T. Toda, M. Nakagiri, and K. Shikano, "Statistical voice conversion techniques for body-conducted unvoiced speech enhancement," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 9, pp. 2505-2517, 2012.
64. T. Toda, A. Black, and K. Tokuda, "Statistical mapping between articulatory movements and acoustic spectrum using a Gaussian mixture model," Speech Commun., vol. 50, pp. 215-227, 2008.
65. H. Zen. (2013). Deep learning in speech synthesis. Keynote speech given at ISCA SSW8. [Online]. Available: http://research.google.com/pubs/archive/41539.pdf
66. L. Deng and D. O'Shaughnessy, Speech Processing: A Dynamic and Optimization-Oriented Approach. New York: Marcel Dekker, 2003.
67. J. Sun and L. Deng, "An overlapping-feature based phonological model incorporating linguistic constraints: Applications to speech recognition," J. Acoust. Soc. Am., vol. 111, pp. 1086-1101, 2002.
68. L. Deng, G. Ramsay, and D. Sun, "Production models as a structural basis for automatic speech recognition," Speech Commun., vol. 33, nos. 2-3, pp. 93-111, Aug. 1997.
69. L. Deng, "Switching dynamic system models for speech articulation and acoustics," in Mathematical Foundations of Speech and Language Processing. New York: Springer-Verlag, 2003, pp. 115-134.
70. D. Yu, L. Deng, and G. Dahl, "Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition," in Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010.
71. G. Dahl, D. Yu, L. Deng, and A. Acero, "Large vocabulary continuous speech recognition with context-dependent DBN-HMMs," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 4688-4691.
72. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition," IEEE Trans. Speech Audio Processing, vol. 20, no. 1, pp. 30-42, 2012.
73. A. Mohamed, G. Dahl, and G. Hinton, "Acoustic modeling using deep belief networks," IEEE Trans. Audio Speech Lang. Processing, vol. 20, no. 1, pp. 14-22, 2012.
74. T.N. Sainath, B. Kingsbury, H. Soltau, and B. Ramabhadran, "Optimization techniques to improve training speed of deep neural networks for large speech tasks," IEEE Trans. Audio Speech Lang. Processing, vol. 21, no. 11, pp. 2267-2276, 2013.
75. X.-L. Zhang and Ji Wu, "Deep belief networks based voice activity detection,"IEEE Trans. Audio Speech Lang. Processing, vol. 21, no. 4, pp. 697-710, 2013.
76. B. Uria, S. Renals, and K. Richmond, "A deep neural network for acoustic-articulatory speech inversion," in Proc. NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
77. G. Hinton, "Training products of experts by minimizing contrastive divergence,"Neural Computat., vol. 14, no. 8, pp. 1711-1800, 2002.
78. th Int. Conf. Artificial Neural Networks, 1999, pp. 1-6.
79. G. Taylor, G. Hinton, and S. Roweis, "Modeling human motion using binary latent variables," in Proc. Advances in Neural Information Processing Systems, 2007, pp. 1345-1352.
80. R. Neal, "Connectionist learning of belief networks," Artificial Intell., vol. 56, no. 1, pp. 71-113, 1992.
81. D. Rumelhart, G. Hinton, and R. Williams, "Learning representations by backpropagating errors," Nature, vol. 323, no. 6088, pp. 533-536, 1986.
82. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, "Gradient flow in recurrent nets: The difficulty of learning long-term dependencies," in A Field Guide to Dynamical Recurrent Neural Networks, S. Kremer and J. Kolen, Eds. Piscataway, NJ: IEEE Press, 2001, pp. 237-244.
83. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layer-wise training of deep networks," in Proc. Advances in Neural Information Processing Systems, 2007, pp. 153-160.
84. H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigne, "Restructuring speech representations using pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Commun., vol. 27, nos. 3-4, pp. 187-207, 1999.
85. T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai, "An adaptive algorithm for mel-cepstral analysis of speech," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 1992, vol. 1, pp. 137-140.
86. T. Yoshimura, "Simultaneous modeling of phonetic and prosodic parameters, and characteristic conversion for HMM-based text-to-speech systems," Ph.D. thesis, Nagoya Inst. of Tech., 2002.