Brain-to-text: Decoding spoken phrases from phone representations in the brain

Frontiers in Neuroscience

Volumn 9, Issue MAY, 2015, Pages

  Herff, Christian ^a   Heger, Dominic ^a   de Pesters, Adriana ^b,d   Telaar, Dominic ^a   Brunner, Peter ^b,c   Schalk, Gerwin ^b,c,d   Schultz, Tanja ^a  

^a KARLSRUHE INSTITUTE OF TECHNOLOGY   (Germany)

^b WADSWORTH CENTER   (United States)

^c ALBANY MEDICAL COLLEGE   (United States)

^d STATE UNIVERSITY OF NEW YORK   (United States)

Author keywords

Automatic speech recognition; Brain computer interface; Broadband gamma; ECoG; Electrocorticography; Pattern recognition; Speech decoding; Speech production

Indexed keywords

ARTICLE; AUTOMATIC SPEECH RECOGNITION; BRAIN ELECTROPHYSIOLOGY; BRAIN TO TEXT; CLINICAL ARTICLE; CONTROLLED STUDY; DIAGONAL BAND OF BROCA; ELECTROCORTICOGRAPHY; HUMAN; HUMAN COMPUTER INTERACTION; SENSORIMOTOR CORTEX; SUPERIOR TEMPORAL GYRUS;

EID: 84931262369     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/fnins.2015.00217     Document Type: Article

References (51)

1. Blakely, T., Miller, K. J., Rao, R. P., Holmes, M. D., and Ojemann, J. G. (2008), Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids, in Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE (IEEE), 4964-4967
2. Bouchard, K. and Chang, E. (2014), Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography, in Engineering in Medicine and Biology Society, 2014. EMBS 2014. 36th Annual International Conference of the IEEE (IEEE)
3. Bouchard, K. E., Mesgarani, N., Johnson, K., and Chang, E. F. (2013), Functional organization of human sensorimotor cortex for speech articulation, Nature, 495, 7441, 327-332
4. Brumberg, J. S.,Wright, E. J., Andreasen, D. S., Guenther, F. H., and Kennedy, P. R. (2011), Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex, Frontiers in neuroscience, 5
5. Canolty, R. T., Soltani, M., Dalal, S. S., Edwards, E., Dronkers, N. F., Nagarajan, S. S., et al. (2007), Spatiotemporal dynamics of word processing in the human brain, Frontiers in neuroscience, 1, 14
6. Chang, E. F., Rieger, J. W., Johnson, K., Berger, M. S., Barbaro, N. M., and Knight, R. T. (2010), Categorical speech representation in human superior temporal gyrus, Nature neuroscience, 13, 11, 1428-1432
7. Crane, W., Gilbert, S., John, McConnell, W., Tenniel, S., John, Weir, H., and Zwecker, J. B. (1867), Mother Gooses Nursery Rhymes. A Collection of Alphabets, Rhymes, Tales and Jingles (London: George Routledge and Sons)
8. Crone, N., Hao, L., Hart, J., Boatman, D., Lesser, R., Irizarry, R., et al. (2001a), Electrocorticographic gamma activity during word production in spoken and sign language, Neurology, 57, 11, 2045-2053
9. Crone, N. E., Boatman, D., Gordon, B., and Hao, L. (2001b), Induced electrocorticographic gamma activity during auditory perception, Clinical Neurophysiology, 112, 4, 565-582
10. Deng, S., Srinivasan, R., Lappas, T., and D'Zmura, M. (2010), Eeg classification of imagined syllable rhythm using hilbert spectrum methods, Journal of neural engineering, 7, 4, 046006
11. Farwell, L. A. and Donchin, E. (1988), Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials, Electroencephalography and clinical Neurophysiology, 70, 6, 510-523
12. Formisano, E., De Martino, F., Bonte, M., and Goebel, R. (2008)," who" is saying" what"? brain-based decoding of human voice and speech, Science, 322, 5903, 970-973
13. Fukuda, M., Rothermel, R., Juhász, C., Nishida, M., Sood, S., and Asano, E. (2010), Cortical gamma oscillations modulated by listening and overt repetition of phonemes, Neuroimage, 49, 3, 2735-2745
14. Gales, M. and Young, S. (2008), The application of hidden markov models in speech recognition, Foundations and Trends in Signal Processing, 1, 3, 195-304
15. Gales, M. J. (1998), Maximum likelihood linear transformations for hmm-based speech recognition, Computer speech & language, 12, 2, 75-98
16. Gasser, T., Bächer, P., and Möcks, J. (1982), Transformations towards the normal distribution of broad band spectral parameters of the eeg, Electroencephalography and clinical neurophysiology, 53, 1, 119-124
17. Guenther, F. H., Brumberg, J. S., Wright, E. J., Nieto-Castanon, A., Tourville, J. A., Panko, M., et al. (2009), A wireless brain-machine interface for real-time speech synthesis, PloS one, 4, 12, e8218
18. Haeb-Umbach, R. and Ney, H. (1992), Linear discriminant analysis for improved large vocabulary continuous speech recognition, in Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, volume 1, 13-16 vol.1, doi:10.1109/ICASSP.1992.225984
19. Huang, X., Acero, A., and Hon, H.-W. (2001), Spoken Language Processing: A Guide to Theory, Algorithm and System Development (Prentice Hall PTR)
20. Jelinek, F. (1997), Statistical methods for speech recognition (MIT press)
21. Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., and Greger, B. (2010), Decoding spoken words using local field potentials recorded from the cortical surface, Journal of neural engineering, 7, 5, 056007
22. Kennedy, J. F. (1989), Inaugural Addresses of the Presidents of the United States. (Washington, DC) (Available online at: www.bartleby.com/124/)
23. Kubanek, J., Brunner, P., Gunduz, A., Poeppel, D., and Schalk, G. (2013), The tracking of speech envelope in the human cortex, PloS one, 8, 1, e53398
24. Kubanek, J. and Schalk, G. (2014), Neuralact: A tool to visualize electrocortical (ecog) activity on a three-dimensional model of the cortex, Neuroinformatics, 1-8
25. Lee, K.-F. (1990), Context-dependent phonetic hidden markov models for speaker-independent continuous speech recognition, Acoustics, Speech and Signal Processing, IEEE Transactions on, 38, 4, 599-609
26. Leuthardt, E. C., Gaona, C., Sharma, M., Szrama, N., Roland, J., Freudenberg, Z., et al. (2011a), Using the electrocorticographic speech network to control a brain-computer interface in humans, Journal of neural engineering, 8, 3, 036004
27. Leuthardt, E. C., Pei, X.-M., Breshears, J., Gaona, C., Sharma, M., Freudenberg, Z., et al. (2011), Temporal evolution of gamma activity in human cortex during an overt and covert word repetition task., Frontiers in human neuroscience, 6, 99-99
28. Lotte, F., Brumberg, J. S., Brunner, P., Gunduz, A., Ritaccio, A. L., Guan, C., et al. (2015), Electrocorticographic representations of segmental features in continuous speech, Frontiers in Human Neuroscience, 9, 97, doi:10.3389/fnhum.2015.00097
29. Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., et al. (2014), Decoding spectrotemporal features of overt and covert speech from the human cortex, Frontiers in Neuroengineering, 7, 14, doi:10.3389/fneng.2014.00014
30. McFarland, D. J., Miner, L. A., Vaughan, T. M., and Wolpaw, J. R. (2000), Mu and beta rhythm topographies during motor imagery and actual movements, Brain topography, 12, 3, 177-186
31. Mesgarani, N., Cheung, C., Johnson, K., and Chang, E. F. (2014), Phonetic feature encoding in human superior temporal gyrus, Science, 1245994
32. Mesgarani, N., Cheung, C., Johnson, K., and Chang, E. F. (2014), Phonetic feature encoding in human superior temporal gyrus, Science, 1245994
33. Miller, K. J., Leuthardt, E. C., Schalk, G., Rao, R. P., Anderson, N. R., Moran, D. W., et al. (2007), Spectral changes in cortical surface potentials during motor movement, The Journal of neuroscience, 27, 9, 2424-2432
34. Mugler, E., Goldrick, M., and Slutzky, M. (2014a), Cortical encoding of phonemic context during word production, in Engineering in Medicine and Biology Society, 2014. EMBS 2014. 36th Annual International Conference of the IEEE (IEEE)
35. Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., et al. (2014b), Direct classification of all american english phonemes using signals from functional speech motor cortex, Journal of Neural Engineering, 11, 3, 035015
36. Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., et al. (2012), Reconstructing speech from human auditory cortex, PLoS biology, 10, 1, e1001251
37. Pei, X., Barbour, D. L., Leuthardt, E. C., and Schalk, G. (2011a), Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans, Journal of neural engineering, 8, 4, 046028
38. Pei, X., Leuthardt, E. C., Gaona, C. M., Brunner, P., Wolpaw, J. R., and Schalk, G. (2011b), Spatiotemporal dynamics of electrocorticographic high gamma activity during overt and covert word repetition, Neuroimage, 54, 4, 2960-2972
39. Potes, C., Gunduz, A., Brunner, P., and Schalk, G. (2012), Dynamics of electrocorticographic (ecog) activity in human temporal and frontal cortical areas during music listening, NeuroImage, 61, 4, 841 - 848, doi:http://dx.doi.org/10.1016/j.neuroimage.2012.04.022
40. Pulvermüller, F., Huss, M., Kherif, F., del Prado Martin, F. M., Hauk, O., and Shtyrov, Y. (2006), Motor cortex maps articulatory features of speech sounds, Proceedings of the National Academy of Sciences, 103, 20, 7865-7870
41. Rabiner, L. (1989), A tutorial on hidden markov models and selected applications in speech recognition, Proceedings of the IEEE, 77, 2, 257-286
42. Roy, E. and Basler, P. (1955), The gettysburg address, in The Collected Works of Abraham Lincoln (New Brunswick, NJ: Rutgers University Press)
43. Sahin, N. T., Pinker, S., Cash, S. S., Schomer, D., and Halgren, E. (2009), Sequential processing of lexical, grammatical, and phonological information within brocas area, Science, 326, 5951, 445-449
44. Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., and Wolpaw, J. R. (2004), Bci2000: a general-purpose brain-computer interface (bci) system, Biomedical Engineering, IEEE Transactions on, 51, 6, 1034-1043
45. Schultz, T. and Kirchhoff, K. (2006), Multilingual Speech Processing (Elsevier, Academic Press)
46. Stolcke, A. (2002), Srilman extensible language modeling toolkit, in Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002
47. Sutter, E. E. (1992), The brain response interface: communication through visually-induced electrical brain responses, Journal of Microcomputer Applications, 15, 1, 31-45
48. Talairach, J. and Tournoux, P. (1988), Co-planar stereotaxic atlas of the human brain. 3-Dimensional proportional system: an approach to cerebral imaging (Thieme)
49. Telaar, D., Wand, M., Gehrig, D., Putze, F., Amma, C., Heger, D., et al. (2014), BioKIT-real time decoder for biosignal processing, in The 15th Annual Conference of the International Speech Communication Association (Interspeech 2014)
50. Towle, V. L., Yoon, H.-A., Castelle, M., Edgar, J. C., Biassou, N. M., Frim, D. M., et al. (2008), Ecog gamma activity during a language task: differentiating expressive and receptive speech areas, Brain, 131, 8, 2013-2027
51. unknown (2009), "Traitor among us" and "Split Feelings" (available on https://www. fanfiction.net/)
52. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002), Brain- computer interfaces for communication and control, Clinical neurophysiology, 113, 6, 767-791