The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect

Experimental Brain Research

Volumn 234, Issue 5, 2016, Pages 1209-1219

  Stekelenburg, Jeroen J ^a   Keetels, Mirjam ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Audiovisual integration; Colavita effect; Event related potentials; Synesthetic congruency

Indexed keywords

ADULT; ANTERIOR CINGULATE; ARTICLE; ASSOCIATION; AUDITORY STIMULATION; AUDITORY SYSTEM PARAMETERS; CLINICAL ARTICLE; COLAVITA EFFECT; ELECTROENCEPHALOGRAM; FEMALE; HUMAN; HUMAN EXPERIMENT; INFERIOR PARIETAL LOBULE; MALE; MIDDLE TEMPORAL GYRUS; PITCH; PREMOTOR CORTEX; PRIORITY JOURNAL; VISUAL STIMULATION; VISUAL SYSTEM PARAMETERS; ANALYSIS OF VARIANCE; ATTENTION; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; EVOKED RESPONSE; HEARING; PHOTOSTIMULATION; PHYSIOLOGY; PSYCHOPHYSICS; REACTION TIME; TIME FACTOR; VISION; YOUNG ADULT;

ACOUSTIC STIMULATION; ANALYSIS OF VARIANCE; ASSOCIATION; ATTENTION; AUDITORY PERCEPTION; BRAIN MAPPING; ELECTROENCEPHALOGRAPHY; EVOKED POTENTIALS; FEMALE; HUMANS; MALE; PHOTIC STIMULATION; PSYCHOPHYSICS; REACTION TIME; TIME FACTORS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84934779975     PISSN: 00144819     EISSN: 14321106     Source Type: Journal    
DOI: 10.1007/s00221-015-4363-0     Document Type: Article

References (69)

1. Baart M, Stekelenburg JJ, Vroomen J (2014) Electrophysiological evidence for speech-specific audiovisual integration. Neuropsychologia 53:115–121. doi:10.1016/j.neuropsychologia.2013.11.011
2. Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823
3. Bertelson P (1999) Ventriloquism: A case of crossmodal perceptual grouping. In: Aschersleben G, Bachmann T, Müsseler J (eds) Cognitive contributions to the perception of spatial and temporal events. Elsevier, Amsterdam, pp 347–363
4. Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29:578–584
5. Bien N, ten Oever S, Goebel R, Sack AT (2012) The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study. NeuroImage 59:663–672. doi:10.1016/j.neuroimage.2011.06.095
6. Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA (2007) Neural basis of the ventriloquist illusion. Curr Biol 17:1697–1703
7. Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. The MIT Press, Cambridge
8. Colavita FB (1974) Human sensory dominance. Percept Psychophys 16:409–412
9. de Gelder B (2000) Recognizing emotions by ear and by eye. In: Lane RD, Nadel L (eds) Cognitive neuroscience of emotion. Oxford University Press, New York, pp 84–105
10. de Gelder B, Bertelson P (2003) Multisensory integration, perception and ecological validity. Trends Cogn Sci 7:460–467
11. de Peralta Grave, Menendez R, Gonzalez Andino SL, Lantz G, Michel CM, Landis T (2001) Noninvasive localization of electromagnetic epileptic activity. I Method descriptions and simulations. Brain Topogr 14:131–137
12. Diaconescu AO, Alain C, McIntosh AR (2011) The co-occurrence of multisensory facilitation and cross-modal conflict in the human brain. J Neurophysiol 106:2896–2909. doi:10.1152/jn.00303.2011
13. Dolan RJ, Morris JS, de Gelder B (2001) Crossmodal binding of fear in voice and face. Proc Natl Acad Sci USA 98:10006–10010
14. Dolscheid S, Hunnius S, Casasanto D, Majid A (2014) Prelinguistic infants are sensitive to space-pitch associations found across cultures. Psychol Sci 25:1256–1261. doi:10.1177/0956797614528521
15. Evans KK, Treisman A (2010) Natural cross-modal mappings between visual and auditory features. J Vis 10(6):1–12. doi:10.1167/10.1.6
16. Gallace A, Spence C (2006) Multisensory synesthetic interactions in the speeded classification of visual size. Percept Psychophys 68:1191–1203
17. Gratton G, Coles MG, Donchin E (1983) A new method for off-line removal of ocular artifact. Electroencephalogr Clin Neurophysiol 55:468–484
18. Guthrie D, Buchwald JS (1991) Significance testing of difference potentials. Psychophysiology 28:240–244
19. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887. doi:10.1523/JNEUROSCI.1740-07.2007
20. Johnson TL, Shapiro KL (1989) Attention to auditory and peripheral visual stimuli: effects of arousal and predictability. Acta Psychol 72:233–245
21. Jones JA, Callan DE (2003) Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. NeuroReport 14:1129–1133. doi:10.1097/01.wnr.0000074343.81633.2a
22. Keetels M, Vroomen J (2007) No effect of auditory-visual spatial disparity on temporal recalibration. Exp Brain Res 182:559–565
23. Klapetek A, Ngo MK, Spence C (2012) Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search? Atten Percept Psychophys 74:1154–1167. doi:10.3758/s13414-012-0317-9
24. Klucharev V, Möttönen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Res Cogn Brain Res 18:65–75
25. Koppen C, Spence C (2007a) Audiovisual asynchrony modulates the Colavita visual dominance effect. Brain Res 1186:224–232. doi:10.1016/j.brainres.2007.09.076
26. Koppen C, Spence C (2007b) Seeing the light: exploring the Colavita visual dominance effect. Exp Brain Res 180:737–754. doi:10.1007/s00221-007-0894-3
27. Koppen C, Spence C (2007c) Spatial coincidence modulates the Colavita visual dominance effect. Neurosci Lett 417:107–111. doi:10.1016/j.neulet.2006.10.069
28. Koppen C, Alsius A, Spence C (2008) Semantic congruency and the Colavita visual dominance effect. Exp Brain Res 184:533–546. doi:10.1007/s00221-007-1120-z
29. Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223. doi:10.1002/hbm.10112
30. Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158:405–414
31. Lebib R, Papo D, Douiri A, de Bode S, Gillon Dowens M, Baudonniere PM (2004) Modulations of ‘late’ event-related brain potentials in humans by dynamic audiovisual speech stimuli. Neurosci Lett 372:74–79
32. Lewald J, Guski R (2003) Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cogn Brain Res 16:468–478
33. Marks LE (1987) On cross-modal similarity: perceiving teomporal patterns by hearing, touch, and vision. Percept Psychophys 42:250–256
34. Martino G, Marks LE (2000) Cross-modal interaction between vision and touch: the role of synesthetic correspondence. Perception 29:745–754
35. Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci 7:3215–3229
36. Meyer GF, Harrison NR, Wuerger SM (2013) The time course of auditory-visual processing of speech and body actions: evidence for the simultaneous activation of an extended neural network for semantic processing. Neuropsychologia 51:1716–1725. doi:10.1016/j.neuropsychologia.2013.05.014
37. Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, Grave de Peralta R (2004) EEG source imaging. Clin Neurophysiol 115:2195–2222
38. Miller J (1982) Divided attention: evidence for coactivation with redundant signals. Cogn Psychol 14:247–279
39. Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465
40. Nava E, Pavani F (2013) Changes in sensory dominance during childhood: converging evidence from the colavita effect and the sound-induced flash illusion. Child Dev 84:604–616. doi:10.1111/j.1467-8624.2012.01856.x
41. Ngo MK, Cadieux ML, Sinnett S, Soto-Faraco S, Spence C (2011) Reversing the Colavita visual dominance effect. Exp Brain Res 214:607–618. doi:10.1007/s00221-011-2859-9
42. Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18:598–609
43. Parise C, Spence C (2008) Synesthetic congruency modulates the temporal ventriloquism effect. Neurosci Lett 442:257–261. doi:10.1016/j.neulet.2008.07.010
44. Parise CV, Spence C (2009) ‘When birds of a feather flock together’: synesthetic correspondences modulate audiovisual integration in non-synesthetes. PLoS One 4:e5664. doi:10.1371/journal.pone.0005664
45. Petrini K, Pollick FE, Dahl S et al (2011) Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56:1480–1492. doi:10.1016/j.neuroimage.2011.03.009
46. Radeau M (1994) Auditory-visual spatial interaction and modularity. Cah Psychol Cogn 13:3–51
47. Radeau M, Bertelson P (1977) Adaptation to auditory-visual discordance and ventriloquism in semi-realistic situations. Percept Psychophys 22:137–146
48. Radeau M, Bertelson P (1978) Cognitive factors and adaptation to auditory-visual discordance. Percept Psychophys 23:341–343
49. Radeau M, Bertelson P (1987) Auditory-visual interaction and the timing of inputs: Thomas (1941) revisited. Psychol Res 49:17–22
50. Raftery AE (1995) Bayesian model selection in social research. In: Marsden PV (ed) Sociological methodology. Blackwell, Cambridge, pp 111–196
51. Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625
52. Slutsky DA, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. NeuroReport 12:7–10
53. Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995. doi:10.3758/s13414-010-0073-7
54. Spence C (2013) Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule. Ann N Y Acad Sci 1296:31–49. doi:10.1111/nyas.12121
55. Spence C, Parise C (2010) Prior-entry: a review. Conscious Cogn 19:364–379. doi:10.1016/j.concog.2009.12.001
56. Spence C, Parise C, Chen YC (2012) The Colavita visual dominance effect. In: Wallace MT (ed) Murray MM. The Neural Bases of Multisensory Processes, Boca Raton
57. Stekelenburg JJ, Vroomen J (2007) Neural correlates of multisensory integration of ecologically valid audiovisual events. J Cogn Neurosci 19:1964–1973
58. Stekelenburg JJ, Vroomen J, de Gelder B (2004) Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity. Neurosci Lett 357:163–166
59. Stubblefield A, Jacobs LA, Kim Y, Goolkasian P (2013) Colavita dominance effect revisited: the effect of semantic congruity. Atten Percept Psychophys 75:1827–1839. doi:10.3758/s13414-013-0530-1
60. Szycik GR, Jansma H, Munte TF (2009) Audiovisual integration during speech comprehension: an fMRI study comparing ROI-based and whole brain analyses. Hum Brain Mapp 30:1990–1999. doi:10.1002/hbm.20640
61. Taylor KI, Moss HE, Stamatakis EA, Tyler LK (2006) Binding crossmodal object features in perirhinal cortex. Proc Natl Acad Sci USA 103:8239–8244. doi:10.1073/pnas.0509704103
62. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Human 34:1053–1065
63. Wagenmakers EJ (2007) A practical solution to the pervasive problems of p values. Psychon Bull Rev 14:779–804
64. Walker P, Bremner JG, Mason U, Spring J, Mattock K, Slater A, Johnson SP (2014) Preverbal infants are sensitive to cross-sensory correspondences: much ado about the null results of Lewkowicz and Minar (2014). Psychol Sci 25:835–836. doi:10.1177/0956797613520170
65. Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158:252–258
66. Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638–667
67. Wilkinson D, Halligan P (2004) The relevance of behavioural measures for functional-imaging studies of cognition. Nat Rev Neurosci 5:67–73. doi:10.1038/nrn1302
68. Wuerger SM, Parkes L, Lewis PA, Crocker-Buque A, Rutschmann R, Meyer GF (2012) Premotor cortex is sensitive to auditory-visual congruence for biological motion. J Cogn Neurosci 24:575–587. doi:10.1162/jocn_a_00173
69. Yuval-Greenberg S, Deouell LY (2007) What you see is not (always) what you hear: induced gamma band responses reflect cross-modal interactions in familiar object recognition. J Neurosci 27:1090–1096. doi:10.1523/JNEUROSCI.4828-06.2007