Cerebral correlates and statistical criteria of cross-modal face and voice integration

Seeing and Perceiving

Volumn 24, Issue 4, 2011, Pages 351-367

  Love, Scott A ^a   Pollick, Frank E ^a   Latinus, Marianne ^a,b  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

^b UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

Audiovisual; Connectivity; Fmri; Multisensory; Speech; Super additive

Indexed keywords

AUDIOVISUAL; CONNECTIVITY; FMRI; MULTISENSORY; SUPER-ADDITIVE;

MAGNETIC RESONANCE IMAGING; SIGNAL TO NOISE RATIO; SPEECH;

INTEGRATION;

ADULT; ARTICLE; AUDITORY STIMULATION; BRAIN CORTEX; FEMALE; HEARING; HUMAN; MALE; METHODOLOGY; NUCLEAR MAGNETIC RESONANCE IMAGING; PATTERN RECOGNITION; PHOTOSTIMULATION; PHYSIOLOGY; PSYCHOPHYSICS; SIGNAL NOISE RATIO; VOICE;

ACOUSTIC STIMULATION; ADULT; AUDITORY PERCEPTION; CEREBRAL CORTEX; FEMALE; FORM PERCEPTION; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; PATTERN RECOGNITION, VISUAL; PHOTIC STIMULATION; PSYCHOPHYSICS; SIGNAL-TO-NOISE RATIO; VOICE; YOUNG ADULT;

EID: 80053401716     PISSN: 18784755     EISSN: 18784763     Source Type: Journal    
DOI: 10.1163/187847511X584452     Document Type: Article

References (56)

1. Baart, M. and Vroomen, J. (2010). Do you see what you are hearing? Cross-modal effects of speech sounds on lipreading, Neuroscience Letters 471, 100-103.
2. Beauchamp, M. S. (2005). Statistical criteria in fMRI studies of multisensory integration, Neuroinformatics 3, 93-114.
3. Beauchamp, M. S., Lee, K. E., Argall, B. D. and Martin, A. (2004). Integration of auditory and visual information about objects in superior temporal sulcus, Neuron 41, 809-823. (Pubitemid 38326842)
4. Belin, P., Fecteau, S. and Bédard, C. (2004). Thinking the voice: neural correlates of voice perception, Trends in Cognitive Sciences 8, 129-135. (Pubitemid 38299323)
5. Bernstein, L. E., Auer, E. T., Moore, J. K., Ponton, C. W., Don, M. and Singh, M. (2002). Visual speech perception without primary auditory cortex activation, Neuroreport 13, 311-315. (Pubitemid 34183573)
6. Besle, J., Fort, A., Delpuech, C. and Giard, M.-H. (2004). Bimodal speech: early suppressive visual effects in human auditory cortex, Eur. J. Neurosci. 20, 2225-2234. (Pubitemid 39331143)
7. Brainard, D. H. (1997). The psychophysics toolbox, Spatial Vis. 10, 433-436. (Pubitemid 127582520)
8. Brefczynski-Lewis, J., Lowitszch, S., Parsons,M., Lemieux, S. and Puce, A. (2009). Audiovisual nonverbal dynamic faces elicit converging fMRI and ERP responses, Brain Topography 21, 193-206.
9. Calvert, G. A. (1997). Activation of auditory cortex during silent lipreading, Science 276, 593-596. (Pubitemid 27193683)
10. Calvert, G. A. (2001). Crossmodal processing in the human brain: insights from functional neuroimaging studies, Cerebral Cortex 11, 1110-1123. (Pubitemid 33108238)
11. Calvert, G. A. and Thesen, T. (2004). Multisensory integration: methodological approaches and emerging principles in the human brain, J. Physiol. Paris 98, 191-205. (Pubitemid 39345371)
12. Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D. and David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding, NeuroReport 10, 2619-2623. (Pubitemid 29418147)
13. Calvert, G. A., Campbell, R. and Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex, Current Biology 10, 649-657. (Pubitemid 30354700)
14. Calvert, G. A., Hansen, P. C., Iversen, S. D. and Brammer, M. J. (2001). Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect, NeuroImage 14, 427-438. (Pubitemid 32709557)
15. Campanella, S. and Belin, P. (2007). Integrating face and voice in person perception, Trends in Cognitive Sciences 11, 535-543. (Pubitemid 350181565)
16. Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G. A., McGuire, P., Suckling, J., Brammer, M. J. and David, A. S. (2001). Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning), Cognitive Brain Res. 12, 233-243. (Pubitemid 32925069)
17. Friston, K. J., Williams, S., Howard, R., Frackowiak, R. S. and Turner, R. (1996). Movement-related effects in fMRI time-series, Magn. Reson. Med. 35, 346-355. (Pubitemid 26072106)
18. Friston, K. J., Buechel, C., Fink, G. R., Morris, J., Rolls, E. and Dolan, R. J. (1997). Psychophysiological and modulatory interactions in neuroimaging, NeuroImage 6, 218-229. (Pubitemid 27488453)
19. Friston, K. J., Holmes, A. P., Price, C. J., Büchel, C. and Worsley, K. J. (1999). Multisubject fMRI studies and conjunction analyses, NeuroImage 10, 385-396. (Pubitemid 29472824)
20. Gitelman, D. (2003). Modeling regional and psychophysiologic interactions in fMRI: the importance of hemodynamic deconvolution, NeuroImage 19, 200-207. (Pubitemid 36627392)
21. Goebel, R. and van Atteveldt, N. (2009). Multisensory functional magnetic resonance imaging: a future perspective, Exper. Brain Res. 198, 153-164.
22. Grant, K. W., Walden, B. E. and Seitz, P. F. (1998). Auditory-visual speech recognition by hearingimpaired subjects: consonant recognition, sentence recognition, and auditory-visual integration, J. Acoust. Soc. Amer. 103, 2677-2690.
23. Hickok, G. and Poeppel, D. (2000). Towards a functional neuroanatomy of speech perception, Trends in Cognitive Sciences 4, 131-138. (Pubitemid 30182529)
24. Hole, G. and Bourne, V. (2010). Face Processing: Psychological, Neuropsychological, and Applied Perspectives. Oxford University Press, Oxford.
25. Joassin, F., Maurage, P. and Campanella, S. (2011a). The neural network sustaining the crossmodal processing of human gender from faces and voices: an fMRI study, NeuroImage 54, 1654-1661.
26. Joassin, F., Pesenti, M., Maurage, P., Verreckt, E., Bruyer, R. and Campanella, S. (2011b). Crossmodal interactions between human faces and voices involved in person recognition, Cortex 47, 367-376.
27. Kreifelts, B., Ethofer, T., Grodd, W., Erb, M. and Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: an event-related fMRI study, NeuroImage 37, 1445-1456. (Pubitemid 47385290)
28. Latinus, M. and Belin, P. (2011). Human voice perception, Current Biology 21, R1-R3.
29. Latinus, M., VanRullen, R. and Taylor, M. J. (2010). Top-down and bottom-up modulation in processing bimodal face/voice stimuli, BMC Neurosci. 11, 36.
30. Laurienti, P. J., Burdette, J. H., Wallace, M. T., Yen, Y.-F., Field, A. S. and Stein, B. E. (2002). Deactivation of sensory-specific cortex by cross-modal stimuli, J. Cognitive Neurosci. 14, 420-429. (Pubitemid 34267542)
31. Laurienti, P. J., Perrault, T. J., Stanford, T. R., Wallace, M. T. and Stein, B. E. (2005). On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies, Exper. Brain Res. 166, 289-297. (Pubitemid 41502075)
32. Ma,W. J., Zhou, X., Ross, L. A., Foxe, J. J. and Parra, L. C. (2009). Lip-reading aids word recognition most in moderate noise: a Bayesian explanation using high-dimensional feature space, PloS One 4, e4638.
33. McGurk, H. and MacDonald, J. (1976). Hearing lips and seeing voices, Nature 264, 746-748.
34. Nath, A. R. and Beauchamp, M. S. (2011). Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech, J. Neurosci. 31, 1704-1714.
35. Noppeney, U., Josephs, O., Hocking, J., Price, C. J. and Friston, K. J. (2008). The effect of prior visual information on recognition of speech and sounds, Cerebral Cortex 18, 598-609. (Pubitemid 351264229)
36. Olson, I. R., Gatenby, J. C. and Gore, J. C. (2002). A comparison of bound and unbound audio-visual information processing in the human cerebral cortex, Cognitive Brain Res. 14, 129-138. (Pubitemid 34626803)
37. Pekkola, J., Ojanen, V., Autti, T., Jääskeläinen, I. P., Möttönen, R., Tarkiainen, A. and Sams, M. (2005). Primary auditory cortex activation by visual speech: an fMRI study at 3 T, Neuroreport 16, 125-128. (Pubitemid 40250383)
38. Pelli, D. (1997). The VideoToolbox software for visual psychophysics: transforming numbers into movies, Spatial Vis. 10, 437-442. (Pubitemid 127582521)
39. Perrault, T. J., Vaughan, J.W., Stein, B. E. andWallace, M. T. (2005). Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli, J. Neurophysiol. 93, 2575-2586. (Pubitemid 40570840)
40. Puce, A., Allison, T., Bentin, S., Gore, J. C. and McCarthy, G. (1998). Temporal cortex activation in humans viewing eye and mouth movements, J. Neurosci. 18, 2188-2199. (Pubitemid 28141423)
41. Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C. and Foxe, J. J. (2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments, Cerebral Cortex 17, 1147-1153. (Pubitemid 46598855)
42. Scott, S. K. and Johnsrude, I. S. (2003). The neuroanatomical and functional organization of speech perception, Trends in Neurosciences 26, 100-107. (Pubitemid 36110544)
43. Stanford, T. R., Quessy, S. and Stein, B. E. (2005). Evaluating the operations underlying multisensory integration in the cat superior colliculus, J. Neurosci. 25, 6499-6508. (Pubitemid 40995282)
44. Stanford, T. R. and Stein, B. E. (2007). Superadditivity in multisensory integration: putting the computation in context, Neuroreport 18, 787-792. (Pubitemid 46686028)
45. Stein, B. E. and Meredith, M. (1993). The Merging of the Senses. MIT Press, Cambridge, MA.
46. Stein, B. E. and Stanford, T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron, Nature Rev. Neurosci. 9, 255-266. (Pubitemid 351430829)
47. Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J. and Rowland, B. A. (2009). Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness, Exper. Brain Res. 198, 113-126.
48. Stevenson, R. A., Geoghegan, M. L. and James, T. W. (2007). Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects, Exper. Brain Res. 179, 85-95. (Pubitemid 46659222)
49. Stevenson, R. A. and James, T.W. (2009). Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition, NeuroImage 44, 1210-1023.
50. Sumby, W. H. and Pollack, I. (1954). Visual contribution to speech intelligibility in noise, J. Acoust. Soc. Amer. 26, 212-215.
51. Szycik, G. R., Tausche, P. and Münte, T. F. (2008). A novel approach to study audiovisual integration in speech perception: localizer fMRI and sparse sampling, Brain Res. 1220, 142-149. (Pubitemid 352004027)
52. Tiippana, K., Puharinen, H., Möttönen, R. and Sams, M. (2011). Sound location can influence audiovisual speech perception when spatial attention is manipulated, Seeing and Perceiving 24, 67-90.
53. Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N.,Mazoyer, B. and Joliot, M. (2002). Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain, NeuroImage 15, 273-289. (Pubitemid 34852226)
54. van Wassenhove, V. and Nagarajan, S. S. (2007). Auditory cortical plasticity in learning to discriminate modulation rate, J. Neurosci. 27, 2663-2672. (Pubitemid 46393602)
55. von Kriegstein, K., Kleinschmidt, A., Sterzer, P. and Giraud, A.-L. (2005). Interaction of face and voice areas during speaker recognition, J. Cognitive Neurosci. 17, 367-376. (Pubitemid 40364526)
56. Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J. and McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech, Cerebral Cortex 13, 1034-1043. (Pubitemid 37163202)