Direct Speech Reconstruction from Articulatory Sensor Data by Machine Learning

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 25, Issue 12, 2017, Pages 2362-2374

  Gonzalez, Jose A ^a   Cheah, Lam A ^b   Gomez, Angel M ^c   Green, Phil D ^a   Gilbert, James M ^b   Ell, Stephen R ^c   Moore, Roger K ^a   Holdsworth, Ed ^d  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b UNIVERSITY OF HULL   (United Kingdom)

^c UNIVERSITY OF GRANADA   (Spain)

^d Practical Control Limited   (United Kingdom)

Author keywords

articulatory to acoustic mapping; permanent magnet articulography; Silent speech interfaces; speech rehabilitation; speech synthesis

Indexed keywords

ARTIFICIAL INTELLIGENCE; DEEP LEARNING; DEEP NEURAL NETWORKS; LEARNING SYSTEMS; MAGNETIC LEVITATION VEHICLES; MAGNETS; PERMANENT MAGNETS; RECURRENT NEURAL NETWORKS; SPEECH SYNTHESIS;

ACOUSTIC MAPPING; GAUSSIAN MIXTURE MODEL; MACHINE LEARNING TECHNIQUES; PERMANENT MAGNETICS; RECURRENT NEURAL NETWORK (RNNS); SILENT SPEECH INTERFACES; SPEECH REHABILITATION; SUBJECTIVE LISTENING TEST;

SPEECH;

EID: 85040448315     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2017.2757263     Document Type: Article

References (52)

1. B. Denby, T. Schultz, K. Honda, T. Hueber, J. Gilbert, and J. Brumberg, "Silent speech interfaces," Speech Commun., vol. 52, no. 4, pp. 270-287, Apr. 2010.
2. J. S. Brumberg, A. Nieto-Castanon, P. R. Kennedy, and F. H. Guenther, "Brain-computer interfaces for speech communication," Speech Commun., vol. 52, no. 4, pp. 367-379, Apr. 2010.
3. C. Herff et al., "Brain-to-text: Decoding spoken phrases from phone representations in the brain," Frontiers Neurosci., vol. 9, p. 217, Jun. 2015.
4. T. Schultz and M.Wand, "Modeling coarticulation in EMG-based continuous speech recognition," Speech Commun., vol. 52, no. 4, pp. 341-353, Apr. 2010.
5. M. Wand, M. Janke, and T. Schultz, "Tackling speaking mode varieties in EMG-based speech recognition," IEEE Trans. BioMed. Eng., vol. 61, no. 10, pp. 2515-2526, Oct. 2014.
6. L. Diener, C. Herff,M. Janke, and T. Schultz, "An initial investigation into the real-time conversion of facial surface EMG signals to audible speech," in 38th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2016, pp. 888-891.
7. M. J. Fagan, S. R. Ell, J. M. Gilbert, E. Sarrazin, and P. M. Chapman, "Development of a (silent) speech recognition system for patients following laryngectomy," Med. Eng. Phys., vol. 30, no. 4, pp. 419-425, 2008.
8. T. Toda, A. W. Black, and K. Tokuda, "Statistical mapping between articulatory movements and acoustic spectrum using a Gaussian mixture model," Speech Commun., vol. 50, no. 3, pp. 215-227, Mar. 2008.
9. T. Hueber, E.-L. Benaroya, G. Chollet, B. Denby, G. Dreyfus, and M. Stone, "Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips," Speech Commun., vol. 52, no. 4, pp. 288-300, 2010.
10. J. A. Gonzalez et al., "Evaluation of a silent speech interface based on magnetic sensing and deep learning for a phonetically rich vocabulary," in Proc. Interspeech, 2017, pp. 3986-3990.
11. T. M. Jones, M. De, B. Foran, K. Harrington, and S. Mortimore, "Laryngeal cancer: United Kingdom national multidisciplinary guidelines," J. Laryngology Otology, vol. 130, no. Suppl 2, pp. S75-S82, 2016.
12. J. M. Gilbert et al., "Isolated word recognition of silent speech using magnetic implants and sensors," Med. Eng. Phys., vol. 32, no. 10, pp. 1189-1197, 2010.
13. R. Hofe et al., "Small-vocabulary speech recognition using a silent speech interface based on magnetic sensing," Speech Commun., vol. 55, no. 1, pp. 22-32, 2013.
14. A. Toutios and S. Narayanan, "Articulatory synthesis of french connected speech from EMA data," in Proc. Interspeech, 2013, pp. 2738-2742.
15. M. Speed, D. Murphy, and D. Howard, "Modeling the vocal tract transfer function using a 3D digital waveguide mesh," IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 22, no. 2, pp. 453-464, Feb. 2014.
16. J. L. Kelly and C. C. Lochbaum, "Speech synthesis," in Proc. Stockholm Speech Commun. Seminar, 1962, pp. 1-4.
17. J. A. Gonzalez et al., "A silent speech system based on permanent magnet articulography and direct synthesis," Comput. Speech Lang., vol. 39, pp. 67-87, 2016.
18. T. Hueber and G. Bailly, "Statistical conversion of silent articulation into audible speech using full-covariance HMM," Comput. Speech Lang., vol. 36, pp. 274-293, 2016.
19. S. Aryal and R. Gutierrez-Osuna, "Data driven articulatory synthesis with deep neural networks," Comput. Speech Lang., vol. 36, pp. 260-273, 2016.
20. F. Bocquelet, T. Hueber, L. Girin, P. Badin, and B. Yvert, "Robust articulatory speech synthesis using deep neural networks for BCI applications," in Proc. Interspeech, 2014, pp. 2288-2292.
21. F. Bocquelet, T. Hueber, L. Girin, C. Savariaux, and B. Yvert, "Real-time control of an articulatory-based speech synthesizer for brain computer interfaces," PLoS Comput. Biol., vol. 12, no. 11, 2016, Art. no. e1005119.
22. A. J. Yates, "Delayed auditory feedback," Psychol. Bull., vol. 60, no. 3, pp. 213-232, 1963.
23. A. Stuart, J. Kalinowski, M. P. Rastatter, and K. Lynch, "Effect of delayed auditory feedback on normal speakers at two speech rates," J. Acoust. Soc. Amer., vol. 111, no. 5, pp. 2237-2241, 2002.
24. J. M. Gilbert et al., "Restoring speech following total removal of the larynx by a learned transformation from sensor data to acoustics," J. Acoust. Soc. Amer., vol. 141, no. 3, pp. EL307-EL313, 2017.
25. L. A. Cheah et al., "A user-centric design of permanent magnetic articulography based assistive speech technology," in Proc. BioSignals, 2015, pp. 109-116.
26. T. Toda, A. W. Black, and K. Tokuda, "Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov. 2007.
27. K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, "Speech parameter generation algorithms for HMM-based speech synthesis," in Proc. Int. Conf. Acoust., Speech, Signal Process., 2000, pp. 1315-1318.
28. K. Cho et al., "Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proc. Conf. Empirical Methods Natural Lang. Process., 2014, pp. 1724-1734.
29. J. Kominek and A. W. Black, "The CMU Arctic speech databases," in Proc. 5th ISCA Workshop Speech Synthesis, 2004, pp. 223-224.
30. T. Toda, M. Nakagiri, and K. Shikano, "Statistical voice conversion techniques for body-conducted unvoiced speech enhancement," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 9, pp. 2505-2517, Nov. 2012.
31. B. S. Atal, J. J. Chang, M. V. Mathews, and J. W. Tukey, "Inversion of articulatory-to-acoustic transformation in the vocal tract by a computersorting technique," J. Acoust. Soc. Amer., vol. 63, no. 5, pp. 1535-1555, 1978.
32. C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY, USA: Springer-Verlag, 2006.
33. H. Zen, K. Tokuda, and A. W. Black, "Statistical parametric speech synthesis," Speech Commun., vol. 51, no. 11, pp. 1039-1064, Nov. 2009.
34. G. Hinton et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
35. H. Zen, A. Senior, and M. Schuster, "Statistical parametric speech synthesis using deep neural networks," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. 2013, pp. 7962-7966.
36. Z. Wu, C. Valentini-Botinhao, O. Watts, and S. King, "Deep neural networks employing multi-task learning and stacked bottleneck features for speech synthesis," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 4460-4464.
37. L.-H. Chen, Z.-H. Ling, L.-J. Liu, and L.-R. Dai, "Voice conversion using deep neural networks with layer-wise generative training," IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 22, no. 12, pp. 1859-1872, Dec. 2014.
38. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, May 2015.
39. T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, "Lowdelay voice conversion based on maximum likelihood estimation of spectral parameter trajectory," in Proc. Interspeech, 2008, pp. 1076-1079.
40. S. De Jong, "SIMPLS: An alternative approach to partial least squares regression," Chemometrics Intell. Lab. Syst., vol. 18, no. 3, pp. 251-263, 1993.
41. P. J. Werbos, "Generalization of backpropagation with application to a recurrent gas market model," Neural Netw., vol. 1, no. 4, pp. 339-356, 1988.
42. A. J. Robinson and F. Fallside, "The utility driven dynamic error propagation network," Eng. Dept., Univ. Cambridge, Cambridge, U.K., Tech. Rep. CUED/F-INFENG/TR.1, 1987.
43. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
44. P. Liu, Q. Yu, Z. Wu, S. Kang, H. Meng, and L. Cai, "A deep recurrent approach for acoustic-to-articulatory inversion," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 4450-4454.
45. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural networks," IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673-2681, Nov. 1997.
46. H. Kawahara, I. Masuda-Katsuse, and A. De Cheveigne, "Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Commun., vol. 27, no. 3, pp. 187-207, Apr. 1999.
47. T. Fukada, K. Tokuda, T. Kobayashi, and S. Imai, "An adaptive algorithm for Mel-cepstral analysis of speech," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 1992, pp. 137-140.
48. D. Kingma and J. Ba, "Adam: A met hod for stochastic optimization," in Proc. Int. Conf. Learn. Representations, 2015.
49. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929-1958, 2014.
50. R. Kubichek, "Mel-cepstral distance measure for objective speech quality assessment," in Proc. IEEE Pac. Rim Conf. Commun., Comput. Signal Process., 1993, pp. 125-128.
51. J. A. Gonzalez et al., "Analysis of phonetic similarity in a silent speech interface based on permanent magnetic articulography," in Proc. Interspeech, 2014, pp. 1018-1022.
52. "Method for the subjective assessment of intermediate quality levels of coding systems," ITU-R BS.1534-1 Recommendation, 2015.