Decoding spectrotemporal features of overt and covert speech from the human cortex

Frontiers in Neuroengineering

Volumn 7, Issue MAY, 2014, Pages

  Martin, Stéphanie ^a,b   Brunner, Peter ^c,d   Holdgraf, Chris ^a   Heinze, Hans Jochen ^e   Crone, Nathan E ^f   Rieger, Jochem ^a,g   Schalk, Gerwin ^c,d   Knight, Robert T ^a   Pasley, Brian N ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b EPFL   (Switzerland)

^c WADSWORTH CENTER   (United States)

^d ALBANY MEDICAL COLLEGE   (United States)

^e OTTO VON GUERICKE UNIVERSITY   (Germany)

^f JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

^g UNIVERSITY OF OLDENBURG   (Germany)

Author keywords

Covert speech; Decoding model; Electrocorticography; Pattern recognition; Speech production

Indexed keywords

ACCURACY; ADULT; ARTICLE; AUDITORY CORTEX; CLINICAL ARTICLE; CONTROLLED STUDY; ELECTROCORTICOGRAPHY; FEMALE; HUMAN; IMAGERY; MALE; MIDDLE AGED; NEUROANATOMY; POSTCENTRAL GYRUS; PRIMARY MOTOR CORTEX; SOUND DETECTION; SPEECH ANALYSIS; SPEECH IMAGERY; SPEECH PERCEPTION; SUPERIOR TEMPORAL GYRUS; TASK PERFORMANCE;

EID: 84901639833     PISSN: 16626443     EISSN: None     Source Type: Journal    
DOI: 10.3389/fneng.2014.00014     Document Type: Article

References (82)

1. Aleman, A. (2004). The functional neuroanatomy of metrical stress evaluation of perceived and imagined spoken words. Cereb. Cortex 15, 221-228. doi: 10.1093/cercor/bhh124
2. Aziz-Zadeh, L., Cattaneo, L., Rochat, M., and Rizzolatti, G. (2005). Covert speech arrest induced by rTMS over both motor and nonmotor left hemisphere frontal sites. J. Cogn. Neurosci. 17, 928-938. doi: 10.1162/0898929054021157
3. Basho, S., Palmer, E. D., Rubio, M. A., Wulfeck, B., and Müller, R.-A. (2007). Effects of generation mode in fMRI adaptations of semantic fluency: paced production and overt speech. Neuropsychologia 45, 1697-1706. doi: 10.1016/j.neuropsychologia.2007.01.007
4. Bialek, W., Rieke, F., de Ruyter van Steveninck, R., and Warland, D. (1991). Reading a neural code. Science 252, 1854-1857. doi: 10.1126/science.2063199
5. Billingsley-Marshall, R., Clear, T., Mencl, W. E., Simos, P. G., Swank, P. R., Men, D., et al. (2007). A comparison of functional MRI and magnetoencephalography for receptive language mapping. J. Neurosci. Methods 161, 306-313. doi: 10.1016/j.jneumeth.2006.10.020
6. Birk, D. (2013). Cocor: Comparing Correlations. Available online at: http://r.birkdiedenhofen.de/pckg/cocor/
7. Boonstra, T. W., Houweling, S., and Muskulus, M. (2009). Does asynchronous neuronal activity average out on a macroscopic scale? J. Neurosci. 29, 8871-8874. doi: 10.1523/JNEUROSCI.2020-09.2009
8. Brigham, K., and Kumar, B. V. K. (2010). "Imagined speech classification with EEG signals for silent communication: a preliminary investigation into synthetic telepathy, " in IEEE. 1-4. Available online at:http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5515807
9. Canolty, R. T. (2007). Spatiotemporal dynamics of word processing in the human brain. Front. Neurosci. 1, 185-196. doi: 10.3389/neuro.01.1.1.014.2007
10. Carmena, J. M., Lebedev, M. A., Crist, R. E., O'Doherty, J. E., Santucci, D. M., Dimitrov, D. F., et al. (2003). Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1:e2. doi: 10.1371/journal.pbio.0000042
11. Chi, T., Gao, Y., Guyton, M. C., Ru, P., and Shamma, S. (1999). Spectro-temporal modulation transfer functions and speech intelligibility. J. Acoust. Soc. Am. 106, 2719-2732. doi: 10.1121/1.428100
12. Chi, T., Ru, P., and Shamma, S. A. (2005). Multiresolution spectrotemporal analysis of complex sounds. J. Acoust. Soc. Am. 118, 887. doi: 10.1121/1.1945807
13. Cichy, R. M., Heinzle, J., and Haynes, J.-D. (2012). Imagery and perception share cortical representations of content and location. Cereb. Cortex 22, 372-380. doi: 10.1093/cercor/bhr106
14. Creutzfeldt, O., Ojemann, G., and Lettich, E. (1989). Neuronal activity in the human lateral temporal lobe. II. responses to the subjects own voice. Exp. Brain Res. 77, 476-489. doi: 10.1007/BF00249601
15. Crone, N. E., Boatman, D., Gordon, B., and Hao, L. (2001). Induced electrocorticographic gamma activity during auditory perception. Brazier award-winning article, 2001. Clin. Neurophysiol. 112, 565-582. doi: 10.1016/S1388-2457(00)00545-9
16. Duffau, H., Capelle, L., Denvil, D., Gatignol, P., Sichez, N., Lopes, M., et al. (2003). The role of dominant premotor cortex in language: a study using intraoperative functional mapping in awake patients. Neuroimage 20, 1903-1914. doi: 10.1016/S1053-8119(03)00203-9
17. Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. Society for Industrial and Applied Mathematics. Available online at: http://epubs.siam.org/doi/book/10.1137/1.9781611970319
18. Elliott, T. M., and Theunissen, F. E. (2009). The modulation transfer function for speech intelligibility. PLoS Comput. Biol. 5:e1000302. doi: 10.1371/journal.pcbi.1000302
19. Feinberg, T. E., Gonzalez Rothi, L. J., and Heilman, K. M. (1986). "Inner speech" in conduction aphasia. Arch. Neurol. 43, 591-593. doi: 10.1001/archneur.1986.00520060053017
20. Fiez, J. A., and Petersen, S. E. (1998). Neuroimaging studies of word reading. Proc. Natl. Acad. Sci. 95, 914-921. doi: 10.1073/pnas.95.3.914
21. Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika 10, 507. doi: 10.2307/2331838
22. Flinker, A., Chang, E. F., Kirsch, H. E., Barbaro, N. M., Crone, N. E., and Knight, R. T. (2010). Single-trial speech suppression of auditory cortex activity in humans. J. Neurosci. 30, 16643-16650. doi: 10.1523/JNEUROSCI.1809-10.2010
23. Geva, C., Correia, M., and Warburton, E. A. (2011a). Diffusion tensor imaging in the study of language and aphasia. Aphasiology 25, 543-558. doi: 10.1080/02687038.2010.534803
24. Geva, S., Bennett, S., Warburton, E., and Patterson, K. (2011b). Discrepancy between inner and overt speech: implications for post-stroke aphasia and normal language processing. Aphasiology 25, 323-243. doi: 10.1080/02687038.2010.511236
25. Geva, S., Jones, P. S., Crinion, J. T., Baron, J. C., and Warburton, E. (2011c). The neural correlates of inner speech defined by voxel-based lesion-symptom mapping. Brain 134, 3071-3082. doi: 10.1093/brain/awr232
26. Giorgino, T. (2009). Computing and visualizing dynamic time warping alignments in R: the Dtw package. J. Stat. Softw. 31, 1-24. Available online at: www.jstatsoft.org/v31/i07/
27. Guenther, F. H., Brumberg, J. S., Wright, E. J., Nieto-Castanon, A., Tourville, J. A., Panko, M., et al. (2009). A wireless brain-machine interface for real-time speech synthesis. PLoS ONE 4:e8218. doi: 10.1371/journal.pone.0008218
28. Guenther, F. H., Ghosh, S. S., and Tourville, J. A. (2006). Neural modeling and imaging of the cortical interactions underlying syllable production. Brain Lang. 96, 280-301. doi: 10.1016/j.bandl.2005.06.001
29. Gunduz, A., Brunner, P., Daitch, A., Leuthardt, E. C., Ritaccio, A. L., Pesaran, B., et al. (2012). Decoding covert spatial attention using electrocorticographic (ECoG) signals in humans. Neuroimage 60, 2285-2293. doi: 10.1016/j.neuroimage.2012.02.017
30. Halpern, A. R. (1988). Mental scanning in auditory imagery for songs. J. Exp. Psychol. Learn. Mem. Cogn. 14, 434-443. doi: 10.1037//0278-7393.14.3.434
31. Halpern, A. R. (1989). Memory for the absolute pitch of familiar songs. Mem. Cogn. 17, 572-581. doi: 10.3758/BF03197080
32. Halpern, A. R., Zatorre, R. J., Bouffard, M., and Johnson, J. A. (2004). Behavioral and neural correlates of perceived and imagined musical timbre. Neuropsychologia 42, 1281-1292. doi: 10.1016/j.neuropsychologia.2003.12.017
33. Heim, S., Opitz, B., and Friederici, A. D. (2002). Broca's area in the human brain is involved in the selection of grammatical gender for language production: evidence from event-related functional magnetic resonance imaging. Neurosci. Lett. 328, 101-104. doi: 10.1016/S0304-3940(02)00494-9
34. Hickok, G. (2001). Functional anatomy of speech perception and speech production: psycholinguistic implications.J. Psycholinguist. Res. 30, 225-235. doi: 10.1023/A:1010486816667
35. Hickok, G., and Poeppel, D. (2007). The cortical organization of speech processing. Nat. Rev. Neurosci. 8, 393-402. doi: 10.1038/nrn2113
36. Hinke, R. M., Hu, X., Stillman, A. E., Kim, S. G., Merkle, H., Salmi, R., et al. (1993). Functional magnetic resonance imaging of broca's area during internal speech. Neuroreport 4, 675-678. doi: 10.1097/00001756-199306000-00018
37. Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372-375. doi: 10.1038/nature11076
38. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164-171. doi: 10.1038/nature04970
39. Hotelling, H. (1940). The selection of variates for use in prediction with some comments on the general problem of nuisance parameters. Ann. Math. Stat. 11, 271-283. doi: 10.1214/aoms/1177731867
40. Huang, J., Carr, T. H., and Cao, Y. (2002). Comparing cortical activations for silent and overt speech using event-related fMRI. Hum. Brain Mapp. 15, 39-53. doi: 10.1002/hbm.1060
41. Hubbard, T. L. (2013). "Auditory aspects of auditory imagery, " in Multisensory Imagery, eds S. Lacey and R. Lawson (New York, NY: Springer New York), 51-76. Available online at:http://www.springerlink.com/index/10.1007/978-1-4614-5879-1_4
42. Intons-Peterson, M. J. (1980). The role of loudness in auditory imagery. Mem. Cogn. 8, 385-393. doi: 10.3758/BF03211134
43. Jeannerod, M. (2003). "Action monitoring and forward control of movements, " in The Handbook of Brain Theory and Neural Networks, 2nd Edn, ed M. Arbib (Cambridge, MA: MIT Press), 83-85.
44. Kaneoke, Y., Donishi, T., Iwatani, J., Ukai, S., Shinosaki, K., and Terada, M. (2012). Variance and autocorrelation of the spontaneous slow brain activity. PLoS ONE 7:e38131. doi: 10.1371/journal.pone.0038131
45. Kay, K. N., and Gallant, J. L. (2009). I can see what you see. Nat. Neurosci. 12:245. doi: 10.1038/nn0309-245
46. Kay, K. N., Naselaris, T., Prenger, R. J., and Gallant, J. L. (2008). Identifying natural images from human brain activity. Nature 452, 352-355. doi: 10.1038/nature06713
47. Kennedy, J. F. (1961). Inaugural Addresses of the Presidents of the United States. (Washington, DC: U.S. G.P.O., 1989). Available online at: www.bartleby.com/124/
48. Kosslyn, S. M. (2005). Mental images and the brain. Cogn. Neuropsychol. 22, 333-347. doi: 10.1080/02643290442000130
49. Kosslyn, S. M., Ganis, G., and Thompson, W. L. (2001). Neural foundations of imagery. Nat. Rev. Neurosci. 2, 635-642. doi: 10.1038/35090055
50. Kosslyn, S. M., and Thompson, W. L. (2000). "Shared mechanisms in visual imagery and visual perception: insights from cognitive neuroscience, " in The New Cognitive Neurosciences, 2nd Edn, ed M. S. Gazzaniga (Cambridge, MA: MIT Press).
51. Lachaux, J. P., Axmacher, N., Mormann, F., Halgren, E., and Crone, N. E. (2012). High-frequency neural activity and human cognition: past, present and possible future of intracranial EEG research. Prog. Neurobiol. 98, 279-301. doi: 10.1016/j.pneurobio.2012.06.008
52. Lancaster, J. L., Woldorff, M. G., Parsons, L. M., Liotti, M., Freitas, C. S., Rainey, L., et al. (2000). Automated talairach atlas labels for functional brain mapping. Hum. Brain Mapp. 10, 120-131. doi: 10.1002/1097-0193(200007)10:3%3C120::AID-HBM30%3E3.0.CO;2-8
53. Leuthardt, E. C., Schalk, G., Wolpaw, J. R., Ojemann, J. G., and Moran, D. W. (2004). A brain-computer interface using electrocorticographic signals in humans. J. Neural Eng. 1, 63-71. doi: 10.1088/1741-2560/1/2/001
54. Llorens, A., Trébuchon, A., Liégeois-Chauvel, C., and Alario, F.-X. (2011). Intra-cranial recordings of brain activity during language production. Front. Psychol. 2:375. doi: 10.3389/fpsyg.2011.00375
55. McGuire, P. K., Silbersweig, D. A., Murray, R. M., David, A. S., Frackowiak, R. S., and Frith, C. D. (1996). Functional anatomy of inner speech and auditory verbal imagery. Psychol. Med. 26, 29-38. doi: 10.1017/S0033291700033699
56. Mesgarani, N., David, S. V., Fritz, J. B., and Shamma, S. A. (2009). Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. J. Neurophysiol. 102, 3329-3339. doi: 10.1152/jn.91128.2008
57. Miller, K. J., Leuthardt, E. C., Schalk, G., Rao, R. P. N., Anderson, N. R., Moran, D. W., et al. (2007). Spectral changes in cortical surface potentials during motor movement. J. Neurosci. 27, 2424-2432. doi: 10.1523/JNEUROSCI.3886-06.2007
58. Mother Goose's Nursery Rhymes. (1867). A Collection of Alphabets, Rhymes, Tales and Jingles (1877). London: George Routledge and Sons.
59. Naselaris, T., Kay, K. N., Nishimoto, S., and Gallant, J. L. (2011). Encoding and decoding in fMRI. Neuroimage 56, 400-410. doi: 10.1016/j.neuroimage.2010.07.073
60. Palmer, E. D., Rosen, H. J., Ojemann, J. G., Buckner, R. L., Kelley, W. M., and Petersen, S. E. (2001). An event-related fMRI study of overt and covert word stem completion. Neuroimage 14, 182-193. doi: 10.1006/nimg.2001.0779
61. Partovi, S., Konrad, F., Karimi, S., Rengier, F., Lyo, J. K., Zipp, L., et al. (2012). Effects of covert and overt paradigms in clinical language fMRI. Acad. Radiol. 19, 518-525. doi: 10.1016/j.acra.2011.12.017
62. Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., et al. (2012). Reconstructing speech from human auditory cortex. PLoS Biol. 10:e1001251. doi: 10.1371/journal.pbio.1001251
63. Pei, X., Leuthardt, E. C., Gaona, C. M., Brunner, P., Wolpaw, J. R., and Schalk, G. (2011). Spatiotemporal dynamics of electrocorticographic high gamma activity during overt and covert word repetition. Neuroimage 54, 2960-2972. doi: 10.1016/j.neuroimage.2010.10.029
64. Pistohl, T., Ball, T., Schulze-Bonhage, A., Aertsen, A., and Mehring, C. (2008). Prediction of arm movement trajectories from ECoG-recordings in humans. J. Neurosci. Methods 167, 105-114. doi: 10.1016/j.jneumeth.2007.10.001
65. Pitt, M. A., and Crowder, R. G. (1992). The role of spectral and dynamic cues in imagery for musical timbre. J. Exp. Psychol. Hum Percept. Perform. 18, 728-738. doi: 10.1037//0096-1523.18.3.728
66. Price, C. J. (2011). A generative model of speech production in broca's and wernicke's areas. Front. Psychol. 2:237. doi: 10.3389/fpsyg.2011.00237
67. Price, C. J. (2012). A review and synthesis of the first 20years of PET and fMRI studies of heard speech, spoken language and reading. Neuroimage 62, 816-847. doi: 10.1016/j.neuroimage.2012.04.062
68. Rabiner, L. R. (1993). Fundamentals of Speech Recognition. Prentice Hall Signal Processing Series. Englewood Cliffs, NJ: PTR Prentice Hall.
69. Rosen, H. J., Ojemann, J. G., Ollinger, J. M., and Petersen, S. E. (2000). Comparison of brain activation during word retrieval done silently and aloud using fMRI. Brain Cogn. 42, 201-217. doi: 10.1006/brcg.1999.1100
70. Roth, M., Decety, J., Raybaudi, M., Massarelli, R., Delon-Martin, C., Segebarth, C., et al. (1996). Possible involvement of primary motor cortex in mentally simulated movement: a functional magnetic resonance imaging study. Neuroreport 7, 1280-1284. doi: 10.1097/00001756-199605170-00012
71. Roy, E., and Basler, P. (1955). "The gettysburg address, " in The Collected Works of Abraham Lincoln, ed A. Lincoln (New Brunswick, NJ: Rutgers UP), 611-612.
72. Sakoe, H., and Chiba, S. (1978). Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. 26, 43-49. doi: 10.1109/TASSP.1978.1163055
73. Schalk, G. (2010). A Practical Guide to Brain-Computer Interfacing with BCI2000: General-Purpose Software for Brain-Computer Interface Research, Data Acquisition, Stimulus Presentation, and Brain Monitoring. London, New York: Springer. doi: 10.1007/978-1-84996-092-2
74. Schalk, G., Kubánek, J., Miller, K. J., Anderson, N. R., Leuthardt, E. C., Ojemann, J. G., et al. (2007). Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J. Neural Eng. 4, 264-275. doi: 10.1088/1741-2560/4/3/012
75. Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., and Wolpaw, J. R. (2004). BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51, 1034-1043. doi: 10.1109/TBME.2004.827072
76. Shamma, S. (2003). Physiological foundations of temporal integration in the perception of speech. J. Phon. 31, 495-501. doi: 10.1016/j.wocn.2003.09.001
77. Shuster, L. I., and Lemieux, S. K. (2005). An fMRI investigation of covertly and overtly produced mono-and multisyllabic words. Brain Lang. 93, 20-31. doi: 10.1016/j.bandl.2004.07.007
78. Stevenson, R. J., and Case, T. I. (2005). Olfactory imagery: a review. Psychon. Bull. Rev. 12, 244-264. doi: 10.3758/BF03196369
79. Tian, X., and Poeppel, D. (2012). Mental imagery of speech: linking motor and perceptual systems through internal simulation and estimation. Front. Hum. Neurosci. 6:314. doi: 10.3389/fnhum.2012.00314
80. Towle, V. L., Yoon, H.-A., Castelle, M., Edgar, J. C., Biassou, N. M., Frim, D. M., et al. (2008). ECoG gamma activity during a language task: differentiating expressive and receptive speech areas. Brain 131, 2013-2027. doi: 10.1093/brain/awn147
81. Warland, D. K., Reinagel, P., and Meister, M. (1997). Decoding visual information from a population of retinal ganglion cells. J. Neurophysiol. 78, 2336-2350.
82. Yetkin, F. Z., Hammeke, T. A., Swanson, S. J., Morris, G. L., Mueller, W. M., McAuliffe, T. L., et al. (1995). A comparison of functional mr activation patterns during silent and audible language tasks. Am. J. Neuroradiol. 16, 1087-1092.