The time course of auditory-visual processing of speech and body actions: Evidence for the simultaneous activation of an extended neural network for semantic processing

Neuropsychologia

Volumn 51, Issue 9, 2013, Pages 1716-1725

  Meyer, Georg F ^a   Harrison, Neil R ^b   Wuerger, Sophie M ^a  

^a UNIVERSITY OF LIVERPOOL   (United Kingdom)

^b LIVERPOOL HOPE UNIVERSITY   (United Kingdom)

Author keywords

Biological motion; EEG; Multisensory; Semantic processing; Speech

Indexed keywords

ACTION POTENTIAL; ADULT; ARTICLE; AUDITORY STIMULATION; BEHAVIOR; BODY MOVEMENT; BRAIN ELECTROPHYSIOLOGY; EVENT RELATED POTENTIAL; FEMALE; FRONTAL CORTEX; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEARING; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; IMAGE QUALITY; INFERIOR FRONTAL GYRUS; LOW RESOLUTION BRAIN ELECTROMAGNETIC TOMOGRAPHY; MALE; NERVE CELL NETWORK; NORMAL HUMAN; PARIETAL CORTEX; PERCEPTION; SEMANTICS; SENSORY CORTEX; SPEECH; TEMPORAL CORTEX; VISION;

BIOLOGICAL MOTION; EEG; MULTISENSORY; SEMANTIC PROCESSING; SPEECH;

ACOUSTIC STIMULATION; ADULT; BRAIN; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; FEMALE; HUMANS; MALE; MOTION PERCEPTION; NERVE NET; PHONETICS; PHOTIC STIMULATION; SEMANTICS; SPEECH PERCEPTION; YOUNG ADULT;

EID: 84880383773     PISSN: 00283932     EISSN: 18733514     Source Type: Journal    
DOI: 10.1016/j.neuropsychologia.2013.05.014     Document Type: Article

References (64)

1. Bach P., Gunter T.C., Knoblich G., Prinz W., Friederici A.D. N400-like negativities in action perception reflect the activation of two components of an action representation. Social Neuroscience 2009, 4(3):212-232.
2. Beer A.L., Plank T., Meyer G., Greenlee M.W. Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing. Frontiers in Integrative Neuroscience 2013, 7:5. 10.3389/fnint.2013.00005.
3. Bentin S., Deouell L.Y. Structural encoding and identification in face processing: ERP evidence for separate mechanisms. Cognitive Neuropsychology 2000, 17(1-3):35-55.
4. Besle J, Bertrand O., Giard M.H. Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hearing Research 2009, 258(1-2):143-151.
5. Besle J., Fort A., Delpuech C., Giard M.H. Bimodal speech: early suppressive visual effects in human auditory cortex. European Journal of Neuroscience 2004, 20(8):2225-2234. 10.1111/j.1460-9568.2004.03670.x.
6. Besle J., Fort A., Giard M.H. Interest and validity of the additive model in electrophysiological studies of multisensory interactions. Cognitive Processing 2004, 5(3):189-192.
7. Brooks, A., van der Zwan, R., Billard, A., Petreska, B., Clarke, S., & Blanke, O. (2007). Auditory motion affects visual biological motion processing. Neuropsychologia, 45, 523-530.
8. Calvo-Merino B., Glaser D.E., Grezes J., Passingham R.E., Haggard P. Action observation and acquired motor skills: an fMRI study with expert dancers. Cerebral Cortex 2005, 15:1243.
9. Cummings A.,?, eponien? R., Koyama A., Saygin A., Townsend J., Dick F. Auditory semantic networks for words and natural sounds. Brain Research 2006, 1115(1):92-107.
10. De Munck J.C., Van Dijk B.W., Spekreijse H. An analytic method to determine the effect of source modeling errors on the apparent location and direction of biological sources. Journal of Applied Physics 1988, 63(3):944-956.
11. De Munck J.C., Van Dijk B.W., Spekreijse H. Mathematical dipoles are adequate to describe realistic generators of human brain activity. Biomedical Engineering, IEEE Transactions on 1988, 35(11):960-966.
12. Dekeyser M., Verfaillie K., Vanrie J Creating stimuli for the study of biological-motion perception. Behavior Research Methods 2002, 34(3):375-382.
13. Doehrmann O., Naumer M.J. Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Research 2008, 1242:136-150. 10.1016/j.brainres.2008.03.071.
14. Giard M.H., Besle J. Methodological considerations: electrophysiology of multisensory interactions in humans. Multisensory Object Perception in the Primate Brain 2010, 55-70.
15. Giard M.H., Peronnet F. Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. Journal of Cognitive Neuroscience 1999, 11(5):473-490. 10.1162/089892999563544.
16. Guthrie D., Buchwald J.S. Significance testing of difference potentials. Psychophysiology 1991, 28:240-244.
17. Harrison N.R., Wuerger S.M., Meyer G.F. Reaction time facilitation for horizontally moving auditory-visual stimuli. Journal of Vision 2010, 10:14.
18. Hein G., Doehrmann O., Müller N.G., Kaiser J., Muckli L., Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. The Journal of Neuroscience 2007, 27(30):7881-7887. 10.1523/jneurosci.1740-07.2007.
19. Hickok G., Poeppel D. The cortical organization of speech processing. Nature Reviews Neuroscience 2007, 8(5):393.
20. Hillyard S.A., Vogel E.K., Luck S.J. Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. Philosophical Transactions of the Royal Society of London, Series B 1998, 353:1257-1270.
21. Junghöfer M., Elbert T., Tucker D.M., Rockstroh B. Statistical control of artifacts in dense array EEG/MEG studies. Psychophysiology 2000, 37(4):523-532.
22. Kelly S.D., Kravitz C., Hopkins M. Neural correlates of bimodal speech and gesture comprehension. Brain and Language 2004, 89(1):253-260. 10.1016/s0093-934x(03)00335-3.
23. Klucharev V., Möttönen R., Sams M. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 2003, 18(1):65-75.
24. Koelsch S., Gunter T., Schroger E., Friederici A. Processing tonal modulations: an ERP study. Journal of Cognitive Neuroscience 2003, 15(8):1149-1159.
25. Kutas M., Federmeier K.D. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Annual Review of Psychology 2011, 62:621-647.
26. Kutas M., Hillyard S.A. Reading senseless sentences: brain potentials reflect semantic incongruity. Science 1980, 207(4427):203-205.
27. Kutas M., Hillyard S.A. Event-related brain potentials to grammatical errors and semantic anomalies. Memory & Cognition 1983, 11(5):539-550.
28. Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H., & Wallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 158, 405-414.
29. Lewis J.W. Audio-visual perception of everyday natural objects-hemodynamic studies in humans in multisensory object perception in the primate brain 2010, 155-190. Springer, New York. M.J. Naumer, J.J. Kaiser (Eds.).
30. Liberman A.M., Cooper F.S., Shankweiler D.P., Studdert-Kennedy M. Perception of the speech code. Psychological Review 1967, 74(6):431.
31. Meredith M., Nemitz J., Stein B. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. The Journal of Neuroscience 1987, 7(10):3215-3229.
32. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus neurons. Journal of Neurophysiology, 75(5), 1843-1857.
33. Meyer G.F., Greenlee M., Wuerger S. Interactions between auditory and visual semantic stimulus classes: evidence for common processing networks for speech and body actions. Journal of Cognitive Neuroscience 2011, 23(9):2291-2308. 10.1162/jocn.2010.21593.
34. Meyer G., Wuerger S. Cross-modal integration of auditory and visual motion signals. NeuroReport 2001, 12:2557-2600.
35. Meyer G., Wuerger S., Roehrbein F., Zetzsche C. Low-level integration of auditory and visual motion signals requires spatial co-localisation. Experimental Brain Research 2005, 166:538-547.
36. Molholm S., Ritter W., Javitt D.C., Foxe J.J. Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cerebral Cortex 2004, 14(4):452-465.
37. Molholm S., Ritter W., Murray M.M., Javitt D.C., Schroeder C.E., Foxe J.J. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cognitive Brain Research 2002, 14(1):115-128.
38. Noppeney U. The sensory-motor theory of semantics: evidence from functional imaging. Language and Cognition 2009, 1(2):249-276.
39. Noppeney U., Ostwald D., Werner S. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. The Journal of Neuroscience 2010, 30(21):7434-7446.
40. Pascual-Marqui R.D. Review of methods for solving the EEG inverse problem. International Journal of Bioelectromagnetism 1999, 1(1):75-86.
41. Pascual-Marqui R.D., Esslen M., Kochi K., Lehmann D. Functional imaging with low resolution brain electromagnetic tomography (LORETA): review, new comparisons, and new validation. Japanese Journal of Clinical Neurophysiology 2002, 30:81-94.
42. Pascual-Marqui R.D., Michel C.M., Lehmann D. Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain. International Journal of Psychophysiology 1994, 18(1):49-65.
43. Perrin F., Pernier J., Bertrand O., Echallier J.F. Spherical splines for scalp potential and current density mapping. Electroencephalography and Clinical Neurophysiology 1989, 72(2):184-187.
44. Phillips J.A., Humphreys G.W., Noppeney U., Price C.J. The neural substrates of action retrieval: an examination of semantic and visual routes to action. Visual Cognition 2002, 9(4-5):662-685.
45. Proctor B., Meyer G. Electrophysiological correlates of facial configuration and audio-visual congruency: evidence that face processing is a visual rather than a multisensory task. Experimental Brain Research 2011, 213(2):203-211. 10.1007/s00221-011-2724-x.
46. Puce A, Perrett D. Electrophysiology and brain imaging of biological motion. Philosophical Transactions of the Royal Society of London, Series B 2003, 358:435-445.
47. Santi A., Servos P., Vatikiotis-Bateson E., Kuratate T., Munhall K. Perceiving biological motion: dissociating visible speech from walking. Journal of Cognitive Neuroscience 2003, 15:800.
48. Scherg M. Fundamentals of dipole source potential analysis. Auditory evoked magnetic fields and electric potentials 1990, 6-40. Karger, Basel. F. Grandori, M. Hoke, G. Romani (Eds.).
49. Skipper J.I., van Wassenhove V., Nusbaum H.C., Small S.L. Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cerebral Cortex 2007, 17(10):2387-2399. 10.1093/cercor/bhl147.
50. Steinbeis N., Koelsch S. Comparing the processing of music and language meaning using EEG and FMRI provides evidence for similar and distinct neural representations. PLoS One 2008, 3(5):e2226.
51. Stekelenburg J.J., Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience 2007, 19(12):1964-1973.
52. Soto-Faraco, S., Navarra, J., & Alsius, A. (2004). Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition, 92, B13-B23.
53. Talairach P., Tournoux J. A stereotactic coplanar atlas of the human brain 1988, Thieme, Stuttgart.
54. Teder-Sälejärvi W.A., McDonald J.J., Di Russo F., Hillyard S.A. An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research 2002, 14(1):106-114. 10.1016/s0926-6410(02)00065-4.
55. Teder-Sälejärvi W.A., Russo F.D., McDonald J.J., Hillyard S.A. Effects of spatial congruity on audio-visual multimodal integration. Journal of Cognitive Neuroscience 2005, 17(9):1396-1409. 10.1162/0898929054985383.
56. Troje N.F., Westhoff C., Lavrov M. Person identification from biological motion: effects of structural and kinematic cues. Perception & Psychophysics 2005, 67:667.
57. van van Atteveldt, N., Formisano, E., Goebel, R., & Blomert, L. (2004). Integration of letters and speech sounds in the human brain. Neuron, 43, 271-282.
58. van Wassenhove V., Grant K.W., Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America 2005, 102(4):1181-1186. 10.1073/pnas.0408949102.
59. Vasey M.W., Thayer J.F. The continuing problem of false positives in repeated measures ANOVA in psychophysiology: a multivariate solution. Psychophysiology 1987, 24:479-486.
60. Vroomen J., Stekelenburg J.J. Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. Journal of Cognitive Neuroscience 2009, 22:1583-1596.
61. Werner S., Noppeney U. Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cerebral Cortex 2010, 20(8):1829-1842.
62. Wuerger S.M., Crocker-Buque A., Meyer G.F. Evidence for auditory visual processing specific to biological motion. Seeing and Perceiving 2012, 25(1):15-28.
63. Wuerger S.M., Hofbauer M., Meyer G. The integration of auditory and visual motion signals at threshold. Perception & Psychophysics 2003, 65(8):1188-1196.
64. Wuerger S.M., Parkes L., Lewis P.A., Crocker-Buque A., Rutschmann R., Meyer G.F. Premotor cortex is sensitive to auditory-visual congruence for biological motion. Journal of Cognitive Neuroscience 2012, 24(3):575-587. 10.1162/jocn_a_00173.