Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces

PLoS Computational Biology

Volumn 12, Issue 11, 2016, Pages

  Bocquelet, Florent ^a,b,c   Hueber, Thomas ^b,c   Girin, Laurent ^b,d   Savariaux, Christophe ^b,c   Yvert, Blaise ^a,b  

^a INSERM   (France)

^b UNIV GRENOBLE ALPES   (France)

^c CNRS   (France)

^d INRIA RHÔNE ALPES   (France)

Author keywords

[No Author keywords available]

Indexed keywords

AUDIO ACOUSTICS; DEEP NEURAL NETWORKS; INTERFACES (COMPUTER); LINGUISTICS; REAL TIME CONTROL; SPEECH COMMUNICATION; SPEECH SYNTHESIS;

ACOUSTIC MAPPING; BRAIN-COMPUTER INTERFACE APPLICATIONS; CONTROL PARAMETERS; ELECTROMAGNETIC ARTICULOGRAPHY; NATURAL SPEECH; OFFLINE MODES; REAL- TIME; REAL-TIME CONTROL; SPEECH SIGNALS; SPEECH SYNTHESIZER;

BRAIN COMPUTER INTERFACE;

ACCURACY; ACOUSTICS; ADULT; ARTICLE; BRAIN COMPUTER INTERFACE; CALIBRATION; CONSONANT; DEEP NEURAL NETWORK; FEMALE; HUMAN; HUMAN EXPERIMENT; JAW; LIP; MACHINE LEARNING; MALE; SOFT PALATE; SPEECH ARTICULATION; TONGUE; VOWEL; ARTIFICIAL NEURAL NETWORK; BIOFEEDBACK; COMMUNICATION AID; COMPUTER SYSTEM; DEVICES; PHONETICS; PROCEDURES; SOUND DETECTION; SPEECH; SPEECH ANALYSIS; SPEECH INTELLIGIBILITY;

BIOFEEDBACK, PSYCHOLOGY; BRAIN-COMPUTER INTERFACES; COMMUNICATION AIDS FOR DISABLED; COMPUTER SYSTEMS; HUMANS; NEURAL NETWORKS (COMPUTER); PHONETICS; SOUND SPECTROGRAPHY; SPEECH ACOUSTICS; SPEECH INTELLIGIBILITY; SPEECH PRODUCTION MEASUREMENT;

EID: 84999828343     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1005119     Document Type: Article

References (64)

1. Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kubler A, et al. A spelling device for the paralysed. Nature. 1999;398: 297–8. doi: 10.1038/1858110192330
2. Chapin JK, Moxon KA, Markowitz RS, Nicolelis MA, Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat Neurosci. 1999;2: 664–70. doi: 10.1038/1022310404201
3. Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature. 2000;408: 361–365. doi: 10.1038/3504258211099043
4. Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP, Brain-machine interface: Instant neural control of a movement signal. Nature. 2002;416: 141–142. doi: 10.1038/416141a11894084
5. Hochberg L, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature. 2006;442: 164–171. doi: 10.1038/nature0497016838014
6. Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB, Cortical control of a prosthetic arm for self-feeding. Nature. Nature Publishing Group; 2008;453: 1098–1101. doi: 10.1038/nature0699618509337
7. Hochberg L, Bacher D, Jarosiewicz B, Masse N, Simeral J, Vogel J, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature. Nature Publishing Group; 2012;485: 372–5. doi: 10.1038/nature1107622596161
8. Gilja V, Nuyujukian P, Chestek C a, Cunningham JP, Yu BM, Fan JM, et al. A high-performance neural prosthesis enabled by control algorithm design. Nat Neurosci. 2012;15: 1752–7. doi: 10.1038/nn.326523160043
9. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet. Elsevier Ltd; 2013;381: 557–64. doi: 10.1016/S0140-6736(12)61816-923253623
10. Donchin E, Spencer KM, Wijesinghe R, The mental prosthesis: Assessing the speed of a P300-based brain- computer interface. IEEE Trans Rehabil Eng. 2000;8: 174–179. 10896179
11. Brumberg JS, Guenther FH, Development of speech prostheses: current status and recent advances. Expert Rev Med Devices. 2010;7: 667–79. doi: 10.1586/erd.10.3420822389
12. Guenther FH, Brumberg JS, Wright EJ, Nieto-Castanon A, Tourville J a, Panko M, et al. A wireless brain-machine interface for real-time speech synthesis. PLoS One. 2009;4: e8218. doi: 10.1371/journal.pone.000821820011034
13. Pasley BN, Knight RT, Decoding speech for understanding and treating aphasia. 1st ed.Progress in brain research. Elsevier B.V.; 2013.
14. Chan AM, Dykstra AR, Jayaram V, Leonard MK, Travis KE, Gygi B, et al. Speech-Specific Tuning of Neurons in Human Superior Temporal Gyrus. Cereb cortex. 2013;10: 2679–93.
15. Kellis S, Miller K, Thomson K, Brown R, House P, Greger B, Decoding spoken words using local field potentials recorded from the cortical surface. J Neural Eng. 2010;7: 056007. doi: 10.1088/1741-2560/7/5/05600720811093
16. Pei X, Barbour DL, Leuthardt EC, Schalk G, Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J Neural Eng. 2011;8: 046028. doi: 10.1088/1741-2560/8/4/04602821750369
17. Tankus A, Fried I, Shoham S, Structured neuronal encoding and decoding of human speech features. Nat Commun. Nature Publishing Group; 2012;3: 1015. doi: 10.1038/ncomms199522910361
18. Martin S, Brunner P, Holdgraf C, Heinze H-J, Crone NE, Rieger J, et al. Decoding spectrotemporal features of overt and covert speech from the human cortex. Front Neuroeng. 2014;7: 14. doi: 10.3389/fneng.2014.0001424904404
19. Mugler EM, Patton JL, Flint RD, Wright Z a, Schuele SU, Rosenow J, et al. Direct classification of all American English phonemes using signals from functional speech motor cortex. J Neural Eng. 2014;11: 035015. doi: 10.1088/1741-2560/11/3/03501524836588
20. Herff C, Heger D, de Pesters A, Telaar D, Brunner P, Schalk G, et al. Brain-to-text: decoding spoken phrases from phone representations in the brain. Front Neurosci. 2015;9: 1–11.
21. Bouchard KE, Mesgarani N, Johnson K, Chang EF, Functional organization of human sensorimotor cortex for speech articulation. Nature. Nature Publishing Group; 2013;495: 327–32. doi: 10.1038/nature1191123426266
22. Hickok G, Poeppel D, The cortical organization of speech processing. Nat Rev Neurosci. 2007;8: 393–402. doi: 10.1038/nrn211317431404
23. Hickok G, Houde J, Rong F, Sensorimotor integration in speech processing: computational basis and neural organization. Neuron. Elsevier Inc.; 2011;69: 407–22. doi: 10.1016/j.neuron.2011.01.01921315253
24. Penfield W, Boldrey E, Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain. 1937;60: 389–443.
25. Pulvermüller F, Huss M, Kherif F, Moscoso del Prado Martin F, Hauk O, Shtyrov Y. Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci U S A. 2006;103: 7865–70.
26. Guenther FH, Cortical interactions underlying the production of speech sounds. J Commun Disord. 2006;39: 350–365. doi: 10.1016/j.jcomdis.2006.06.01316887139
27. Grabski K, Lamalle L, Vilain C, Schwartz J-L, Vallée N, Tropres I, et al. Functional MRI assessment of orofacial articulators: neural correlates of lip, jaw, larynx, and tongue movements. Hum Brain Mapp. 2012;33: 2306–21. doi: 10.1002/hbm.2136321826760
28. Tate MC, Herbet G, Moritz-Gasser S, Tate JE, Duffau H, Probabilistic map of critical functional regions of the human cerebral cortex: Broca’s area revisited. Brain. 2014;137: 2773–2782. doi: 10.1093/brain/awu16824970097
29. Cheung C, Hamiton LS, Johnson K, Chang EF, The auditory representation of speech sounds in human motor cortex. Elife. 2016;5: 1–19.
30. Maeda S, A digital simulation method of the vocal-tract system. Speech Commun. 1982; 199–229.
31. Sondhi M, Schroeter J, A hybrid time-frequency domain articulatory speech synthesizer. IEEE Trans Acoust. 1987;35.
32. Kello CT, Plaut DC, A neural network model of the articulatory-acoustic forward mapping trained on recordings of articulatory parameters. J Acoust Soc Am. 2004;116: 2354. 15532666
33. Birkholz P, Jackel D, Kroger KJ, Construction And Control Of A Three-Dimensional Vocal Tract Model. 2006 IEEE Int Conf Acoust Speech Signal Process Proc. 2006;1: 873–876.
34. Toda T, Black AW, Tokuda K, Statistical mapping between articulatory movements and acoustic spectrum using a Gaussian mixture model. Speech Commun. 2008;50: 215–227.
35. Bocquelet F, Hueber T, Girin L, Badin P, Yvert B. Robust Articulatory Speech Synthesis using Deep Neural Networks for BCI Applications. Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech). 2014. pp. 2288–2292.
36. Assaneo MF, Trevisan M a, Mindlin GB, Discrete motor coordinates for vowel production. PLoS One. 2013;8.
37. Preuß S, Neuschaefer-Rube C, Birkholz P, Real-time control of a 2D animation model of the vocal tract using optopalatography. Proc Annu Conf Int Speech Commun Assoc INTERSPEECH. 2013; 997–1001.
38. Story BH, Phrase-level speech simulation with an airway modulation model of speech production. Comput Speech Lang. Elsevier Ltd; 2013;27: 989–1010. doi: 10.1016/j.csl.2012.10.00523503742
39. Toutios a, Ouni S, Laprie Y, Estimating the control parameters of an articulatory model from electromagnetic articulograph data. J Acoust Soc Am. 2011;129: 3245–3257. doi: 10.1121/1.356971421568426
40. Beautemps D, Badin P, Bailly G, Linear degrees of freedom in speech production: Analysis of cineradio- and labio-film data and articulatory-acoustic modeling. J Acoust Soc Am. 2001;109: 2165–2180. 11386568
41. Ifft PJ, Shokur S, Li Z, Lebedev M a, International LS. Brain-Machine Interface Enables Bimanual Arm Movements in Monkeys. Sci Transl Med. 2013;5. e
42. Wodlinger B, Downey JE, Tyler-Kabara EC, Schwartz a B, Boninger ML, Collinger JL, Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J Neural Eng. IOP Publishing; 2015;12: 016011. doi: 10.1088/1741-2560/12/1/01601125514320
43. Richmond K, Hoole P, King S, Forum I. Announcing the Electromagnetic Articulography (Day 1) Subset of the mngu0 Articulatory Corpus. Proc Interspeech.: 1–4.
44. Uría B. A Deep Belief Network for the Acoustic-Articulatory Inversion Mapping Problem [Internet]. 2011.
45. Tokuda K, Oura K, Tamamori A, Sako S, Zen H, Nose T, et al. Speech Signal Processing Toolkit (SPTK). In: http://sp-tk.sourceforge.net/. 2014.
46. Imai S, Sumita K, Furuichi C,. Mel Log Spectrum Approximation (MLSA) Filter for Speech Synthesis. Electron Commun Japan. 1983;66-A: 10–18.
47. Grimaldi M, Fivela BG. New technologies for simultaneous acquisition of speech articulatory data: 3D articulograph, ultrasound and electroglottograph. Proc …. 2008;
48. Markel JD, Gray AH, Linear Prediction of Speech [Internet]. Berlin, Heidelberg: Springer Berlin Heidelberg; 1976.
49. Bechet F, LIA_PHON—Un systeme complet de phonetisation de textes. Trait Autom des Langues. 2001;42.
50. Larochelle H, Bengio Y, Lamblin P, Exploring Strategies for Training Deep Neural Networks. J Mach Learn Res. 2009;1: 1–40.
51. Aryal S, Gutierrez-Osuna R, Data driven articulatory synthesis with deep neural networks. Comput Speech Lang. Elsevier Ltd; 2015; 1–14.
52. Hinton G, A Practical Guide to Training Restricted Boltzmann Machines. Comput Sci. 2010;7700: 599–619.
53. Hinton GE, Salakhutdinov RR, Reducing the dimensionality of data with neural networks. Science. 2006;313: 504–7. doi: 10.1126/science.112764716873662
54. Rumelhart DE, Hinton GE, Williams RJ, Learning representations by back-propagating errors. Nature. 1986;323: 533–536.
55. Rasmussen CE. Function minimization using conjugate gradients. 1996;0: 1–7.
56. Hothorn T, Bretz F, Westfall P, Simultaneous Inference in General Parametric Models. Biometrical J. 2008;50: 346–363.
57. Hueber T, Badin P, Savariaux C. Differences in articulatory strategies between silent, whispered and normal speech? a pilot study using electromagnetic articulography. Proc ISSP, to …. 2010; 0–1.
58. Janke M, Wand M, Schultz T, Impact of lack of acoustic feedback in EMG-based silent speech recognition. Proc Interspeech. 2010; 2686–2689.
59. Wang J, Samal A, Green JR. Across-speaker Articulatory Normalization for Speaker-independent Silent Speech Recognition Callier Center for Communication Disorders Department of Computer Science & Engineering Department of Communication Sciences & Disorders. 2014; 1179–1183.
60. Denby B, Schultz T, Honda K, Hueber T, Gilbert JM, Brumberg JS, Silent speech interfaces. Speech Commun. Elsevier B.V.; 2010;52: 270–287.
61. Wand M, Schulte C, Janke M, Schultz T. Array-based Electromyographic Silent Speech Interface. 6th International Conference on Bio-inspired Systems and Signal Processing. 2013.
62. Cler MJ, Nieto-Castanon a, Guenther FH, Stepp CE. Surface electromyographic control of speech synthesis. Eng Med Biol Soc (EMBC), 2014 36th Annu Int Conf IEEE. 2014; 5848–5851.
63. Hueber T, Bailly G, Statistical conversion of silent articulation into audible speech using full-covariance HMM. Comput Speech Lang. 2016;36: 274–293.
64. Hueber T., Benaroya E.L., Chollet G., Denby B., Dreyfus G., Stone M.,, (2010) "Development of a Silent Speech Interface Driven by Ultrasound and Optical Images of the Tongue and Lips", Speech Communication, 52(4), pp. 288–300.