Visual phonetic processing localized using speech and nonspeech face gestures in video and point-light displays

Human Brain Mapping

Volumn 32, Issue 10, 2011, Pages 1660-1676

  Bernstein, Lynne E ^a,b   Jiang, Jintao ^a   Pantazis, Dimitrios ^b,c   Lu, Zhong Lin ^b   Joshi, Anand ^c  

^a HOUSE EAR INSTITUTE   (United States)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c NONE   (United States)

Author keywords

Frontal lobe; Functional magnetic resonance imaging; Gestures; Lipreading; Parietal lobe; Phonetics; Speech perception; Speechreading; Temporal lobe; Visual perception

Indexed keywords

ADULT; ARTICLE; ASSOCIATION; CONTROLLED STUDY; FACIAL EXPRESSION; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUSIFORM FACE AREA; GESTURE; HUMAN; HUMAN EXPERIMENT; LEFT HEMISPHERE; MALE; MIDDLE TEMPORAL GYRUS; NORMAL HUMAN; OCCIPITAL CORTEX; PHONETICS; PRIORITY JOURNAL; RIGHT HEMISPHERE; SPEECH; SUPERIOR TEMPORAL SULCUS; TEMPORAL GYRUS; TEMPORAL VISUAL SPEECH AREA; VIDEORECORDING;

ACOUSTIC STIMULATION; ADULT; ANALYSIS OF VARIANCE; BRAIN; BRAIN MAPPING; FACE; FEMALE; FUNCTIONAL LATERALITY; GESTURES; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; MOTION PERCEPTION; OXYGEN; PATTERN RECOGNITION, VISUAL; PHONETICS; PHOTIC STIMULATION; REACTION TIME; SPEECH; YOUNG ADULT;

EID: 79551542217     PISSN: 10659471     EISSN: 10970193     Source Type: Journal    
DOI: 10.1002/hbm.21139     Document Type: Article

References (94)

1. Amedi A, Malach R, Hendler T, Peled S, Zohary E (2001): Visuo-haptic object-related activation in the ventral visual pathway. Nat Neurosci 4: 324-330.
2. Andersson U, Lidestam B (2005): Bottom-up driven speechreading in a speechreading expert: The case of AA (JK023). Ear Hear 26: 214-224.
3. Auer ET Jr, Bernstein LE (2007): Enhanced visual speech perception in individuals with early onset hearing impairment. J Speech Lang Hear Res 50: 1157-1165.
4. Auer ET Jr, Bernstein LE, Sungkarat W, Singh M (2007): Vibrotactile activation of the auditory cortices in deaf versus hearing adults Neuroreport 18: 645-648.
5. Beauchamp MS (2005a): See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15: 145-153.
6. Beauchamp MS (2005b): Statistical criteria in fMRI studies of multisensory integration. Neuroinformatics 3: 93-113.
7. Beauchamp MS, Lee KE, Haxby JV, Martin A (2003): fMRI responses to video and point-light displays of moving humans and manipulable objects. J Cogn Neurosci 15: 991-1001.
8. Benson RR, Richardson M, Whalen DH, Lai S (2006): Phonetic processing areas revealed by sinewave speech and acoustically similar non-speech. Neuroimage 31: 342-353.
9. Bernstein LE, Auer ET Jr, Moore JK (2004): Audiovisual speech binding: Convergence or association? In: Calvert GA, Spence C, Stein BE, editors. Handbook of Multisensory Processes. Cambridge, MA: MIT. pp 203-223.
10. Bernstein LE, Auer ET Jr, Moore JK, Ponton CW, Don M, Singh M (2002): Visual speech perception without primary auditory cortex activation. Neuroreport 13: 311-315.
11. Bernstein LE, Demorest ME, Tucker PE (2000): Speech perception without hearing. Percept Psychophys 62: 233-252.
12. Bernstein LE, Lu ZL, Jiang J (2008): Quantified acoustic-optical speech signal incongruity identifies cortical sites of audiovisual speech processing. Brain Res 1242: 172-184.
13. Binder JR, Frost JA, Hammeke TA, Bellgowan PS, Springer JA, Kaufman JN, Possing ET (2000): Human temporal lobe activation by speech and nonspeech sounds. Cereb Cortex 10: 512-528.
14. Bonda E, Petrides M, Ostry D, Evans A (1996): Specific involvement of human parietal systems and the amygdala in the perception of biological motion. J Neurosci 16: 3737-3744.
15. Bunzeck N, Wuestenberg T, Lutz K, Heinze HJ, Jancke L (2005): Scanning silence: Mental imagery of complex sounds. Neuroimage 26: 1119-1127.
16. Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997): Activation of auditory cortex during silent lipreading. Science 276: 593-596.
17. Calvert GA, Campbell R (2003): Reading speech from still and moving faces: The neural substrates of visible speech. J Cogn Neurosci 15: 57-70.
18. Campbell R, Landis T, Regard M (1986): Face recognition and lipreading. A neurological dissociation. Brain 109: 509-521.
19. Campbell R, MacSweeney M, Surguladze S, Calvert G, McGuire P, Suckling J, Brammer MJ, David AS (2001): Cortical substrates for the perception of face actions: An fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). Cogn Brain Res 12: 233-243.
20. Campbell R, Zihl J, Massaro DW, Munhall K, Cohen MM (1997): Speechreading in the akinetopsic patient, LM. Brain 120: 1793-1803.
21. Capek CM, MacSweeney M, Woll B, Waters D, McGuire PK, David AS, Brammer MJ, Campbell R (2008): Cortical circuits for silent speech reading in deaf and hearing people. Neuropsychologia 46: 1233-1241.
22. Cappe C, Rouiller EM, Barone P (2009): Multisensory anatomical pathways. Hear Res 258: 28-36.
23. Chawla D, Buechel C, Edwards R, Howseman A, Josephs O, Ashburner J, Friston KJ (1999): Speed-dependent responses in V5: A replication study. Neuroimage 9: 508-515.
24. Chomsky C (1986): Analytic study of the Tadoma method: Language abilities of three deaf-blind subjects. J Speech Hear Res 29: 332-347.
25. Damasio H (2005): Human Brain Anatomy in Computerized Images. Oxford: Oxford University. 560 p.
26. Downing PE, Jiang Y, Shuman M, Kanwisher N (2001): A cortical area selective for visual processing of the human body. Science 293: 2470-2473.
27. Driver J, Noesselt T (2008): Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57: 11-23.
28. Eggermont JJ (2001): Between sound and perception: Reviewing the search for a neural code. Hear Res 157: 1-42.
29. Fischl B, Sereno MI, Dale AM (1999a): Cortical surface-based analysis. II: Inflation, flattening, and a surface-based coordinate system. Neuroimage 9: 195-207.
30. Fischl B, Sereno MI, Tootell RB, Dale AM (1999b): High-resolution intersubject averaging and a coordinate system for the cortical surface. Hum Brain Mapp 8: 272-284.
31. Fridriksson J, Moss J, Davis B, Baylis GC, Bonilha L, Rorden C (2008): Motor speech perception modulates the cortical language areas. Neuroimage 41: 605-613.
32. Genovese CR, Lazar NA, Nichols T (2002): Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15: 870-878.
33. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005): Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci 25: 5004-5012.
34. Ghazanfar AA, Schroeder CE (2006): Is neocortex essentially multisensory? Trends Cogn Sci 10: 278-285.
35. Grill-Spector K, Kourtzi Z, Kanwisher N (2001): The lateral occipital complex and its role in object recognition. Vision Res 41: 1409-1422.
36. Grossman E, Donnelly M, Price R, Pickens D, Morgan V, Neighbor G, Blake R (2000): Brain areas involved in perception of biological motion. J Cogn Neurosci 12: 711-720.
37. Grossman ED, Blake R (2002): Brain areas active during visual perception of biological motion. Neuron 35: 1167-1175.
38. Hall DA, Fussell C, Summerfield AQ (2005): Reading fluent speech from talking faces: Typical brain networks and individual differences. J Cogn Neurosci 17: 939-953.
39. Hasson U, Skipper JI, Nusbaum HC, Small SL (2007): Abstract coding of audiovisual speech: Beyond sensory representation. Neuron 56: 1116-1126.
40. Jenkins JJ, Strange W (1999): Perception of dynamic information for vowels in syllable onsets and offsets. Percept Psychophys 61: 1200-1210.
41. Jiang J, Auer ET Jr, Alwan A, Keating PA, Bernstein LE (2007): Similarity structure in visual speech perception and optical phonetics. Percept Psychophys 69: 1070-1083.
42. Johansson G (1973): Visual perception of biological motion and a model for its analysis. Percept Psychophys 14, 201-211.
43. Jones EG, Powell TP (1970): An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain 93: 793-820.
44. Jordan TR, McCotter MV, Thomas SM (2000): Visual and audiovisual speech perception with color and gray-scale facial images. Percept Psychophys 62: 1394-1404.
45. Joshi AA, Shattuck DW, Thompson PM, Leahy RM (2007): Surface-constrained volumetric brain registration using harmonic mappings. IEEE Trans Med Imaging 26: 1657-1669.
46. Kaas JH, Hackett TA (2000): Subdivisions of auditory cortex and processing streams in primates. Proc Natl Acad Sci USA 97: 11793-11799.
47. Kanwisher N, McDermott J, Chun MM (1997): The fusiform face area: A module in human extrastriate cortex specialized for face perception. J Neurosci 17: 4302-4311.
48. Kayser C, Petkov CI, Logothetis NK (2008): Visual modulation of neurons in auditory cortex. Cereb Cortex 18: 1560-1574.
49. Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI (2009): Circular analysis in systems neuroscience: The dangers of double dipping. Nat Neurosci 12: 535-540.
50. Liebenthal E, Binder JR, Spitzer SM, Possing ET, Medler DA (2005): Neural substrates of phonemic perception. Cereb Cortex 15: 1621-1631.
51. Macmillan NA, Creelman CD (2005): Detection Theory: A User's Guide. Mahwah, NJ: Earlbaum. 512 p.
52. MacSweeney M, Calvert GA, Campbell R, McGuire PK, David AS, Williams SC, Woll B, Brammer MJ (2002): Speechreading circuits in people born deaf. Neuropsychologia 40: 801-807.
53. Malach R, Reppas JB, Benson RR, Kwong KK, Jiang H, Kennedy WA, Ledden PJ, Brady TJ, Rosen BR, Tootell RB (1995): Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proc Natl Acad Sci USA 92: 8135-8139.
54. Meredith MA (2002): On the neuronal basis for multisensory convergence: A brief overview. Cogn Brain Res 14: 31-40.
55. Mesulam MM (1998): From sensation to cognition. Brain 121: 1013-1052.
56. Mohammed T, Campbell R, MacSweeney M, Milne E, Hansen P, Coleman M (2005): Speechreading skill and visual movement sensitivity are related in deaf speechreaders. Perception 34: 205-216.
57. Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005): Valid conjunction inference with the minimum statistic. Neuroimage 25: 653-660.
58. Obleser J, Zimmermann J, Van Meter J, Rauschecker JP (2007): Multiple stages of auditory speech perception reflected in event-related fMRI. Cereb Cortex 17: 2251-2257.
59. Oram MW, Perrett DI (1994): Responses of anterior superior temporal polysensory (STPa) neurons to "biological motion" stimuli. J Cogn Neurosci 6: 99-116.
60. Pantazis D, Joshi A, Jiang J, Shattuck DW, Bernstein LE, Damasio H, Leahy RM (2010): Comparison of landmark-based and automatic methods for cortical surface registration. Neuroimage 49: 2479-2493.
61. Paulesu E, Perani D, Blasi V, Silani G, Borghese NA, De Giovanni U, Sensolo S, Fazio F (2003): A functional-anatomical model for lipreading. J Neurophysiol 90: 2005-2013.
62. Peelen MV, Wiggett AJ, Downing PE (2006): Patterns of fMRI activity dissociate overlapping functional brain areas that respond to biological motion. Neuron 49: 815-822.
63. Pekkola J, Ojanen V, Autti T, Jaaskelainen IP, Möttönen R, Sams M (2006): Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Hum Brain Mapp 27: 471-477.
64. Pekkola J, Ojanen V, Autti T, Jaaskelainen IP, Möttönen R, Tarkianen A, Sams M (2005): Primary auditory cortex activation by visual speech: An fMRI study at 3T. Neuroreport 16: 125-128.
65. Pelphrey KA, Viola RJ, McCarthy G (2004): When strangers pass: Processing of mutual and averted social gaze in the superior temporal sulcus. Psychol Sci 15: 598-603.
66. Pelphrey KA, Morris JP, Michelich CR, Allison T, McCarthy G (2005): Functional anatomy of biological motion perception in posterior temporal cortex: An fMRI study of eye, mouth and hand movements. Cereb Cortex 15: 1866-1876.
67. Perrett DI, Smith PA, Mistlin AJ, Chitty AJ, Head AS, Potter DD, Broennimann R, Milner AD, Jeeves MA (1985): Visual analysis of body movements by neurones in the temporal cortex of the macaque monkey: A preliminary report. Behav Brain Res 16: 153-170.
68. Perrett DI, Mistlin AJ, Chitty AJ, Smith PA, Potter DD, Broennimann R, Harries M (1988): Specialized face processing and hemispheric asymmetry in man and monkey: Evidence from single unit and reaction time studies. Behav Brain Res 29: 245-258.
69. Poldrack RA, Wagner AD, Prull MW, Desmond JE, Glover GH, Gabrieli JD (1999): Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex. Neuroimage 10: 15-35.
70. Ponton CW, Bernstein LE, Auer ET Jr. (2009): Mismatch negativity with visual-only and audiovisual speech. Brain Topogr 21: 207-215.
71. Puce A, Perrett D (2003): Electrophysiology and brain imaging of biological motion. Philos Trans R Soc Lond B Biol Sci 358: 435-445.
72. Puce A, Allison T, Bentin S, Gore JC, McCarthy G (1998): Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci 18: 2188-2199.
73. Puce A, Syngeniotis A, Thompson JC, Abbott DF, Wheaton KJ, Castiello U (2003): The human temporal lobe integrates facial form and motion: Evidence from fMRI and ERP studies. Neuroimage 19: 861-869.
74. Reed CM (1995): Tadoma: An overview of research. In: Plant G, Spens K-E, editors. Profound Deafness and Speech Communication. London: Whurr.
75. Rizzolatti G, Arbib MA (1998): Language within our grasp. Trends Neurosci 21: 188-194.
76. Rizzolatti G, Craighero L (2004): The mirror-neuron system. Annu Rev Neurosci 27: 169-192.
77. Rosenblum LD (2008): Speech perception as a multimodal phenomenon. Curr Dir Psychol Sci 17: 405-409.
78. Rosenblum LD, Johnson JA, Saldana HM (1996): Point-light facial displays enhance comprehension of speech in noise. J Speech Hear Res 39: 1159-1170.
79. Rosenblum LD, Saldana HM (1996): An audiovisual test of kinematic primitives for visual speech perception. J Exp Psychol Hum Percept Perform 22: 318-331.
80. Rosenblum LD, Saldana HM (1998): Time-varying information for visual speech perception. In: Campbell R, Dodd B, Burnham D, editors. Hearing by Eye II: Advances in the Psychology of Speech Reading and Auditory-Visual Speech. East Sussex, Hove, UK: Psychology Press. pp 61-81.
81. Sadato N, Okada T, Honda M, Matsuki K-I, Yoshida M, Kashikura K-I, Takei W, Sato T, Kochiyama T, Yonekura Y (2005): Cross-modal integration and plastic changes revealed by lip movement, random-dot motion and sign languages in the hearing and deaf. Cereb Cortex 15: 1113-1122.
82. Sams M, Aulanko R, Hamalainen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991): Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127: 141-145.
83. Santi A, Servos P, Vatikiotis-Bateson E, Kuratate T, Munhall K (2003): Perceiving biological motion: Dissociating visible speech from walking. J Cogn Neurosci 15: 800-809.
84. Schroeder CE, Foxe JJ (2002): The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Cogn Brain Res 14: 187-198.
85. Scott SK, Blank CC, Rosen S, Wise RJ (2000): Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123: 2400-2406.
86. Scott SK, Rosen S, Lang H, Wise RJ (2006): Neural correlates of intelligibility in speech investigated with noise vocoded speech-A positron emission tomography study. J Acoust Soc Am 120: 1075-1083.
87. Shattuck DW, Leahy RM (2002): BrainSuite: An automated cortical surface identification tool. Med Image Anal 6: 129-142.
88. Skipper JI, Nusbaum HC, Small SL (2005): Listening to talking faces: Motor cortical activation during speech perception. Neuroimage 25: 76-89.
89. SPSS. 2006. SPSS 15.0 for Windows: LEAD Technologies, Inc.
90. Thompson JC, Hardee JE, Panayiotou A, Crewther D, Puce A (2007): Common and distinct brain activation to viewing dynamic sequences of face and hand movements. Neuroimage 37: 966-973.
91. Uppenkamp S, Johnsrude IS, Norris D, Marslen-Wilson W, Patterson RD (2006): Locating the initial stages of speech-sound processing in human temporal cortex. Neuroimage 31: 1284-1296.
92. Vigneau M, Beaucousin V, Herve PY, Duffau H, Crivello F, Houde O, Mazoyer B, Tzourio-Mazoyer N (2006): Meta-analyzing left hemisphere language areas: Phonology, semantics, and sentence processing. Neuroimage 30: 1414-1432.
93. Vul E, Harris C, Winkielman P, Pashler H (2009): Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect Psychol Sci 4: 274-290.
94. Waters D, Campbell R, Capek CM, Woll B, David AS, McGuire PK, Brammer MJ, MacSweeney M (2007): Fingerspelling, signed language, text and picture processing in deaf native signers: The role of the mid-fusiform gyrus. Neuroimage 35: 1287-1302.