The time-course of the cross-modal semantic modulation of visual picture processing by naturalistic sounds and spokenWords

Multisensory Research

Volumn 26, Issue 4, 2013, Pages 371-386

  Chen, Yi Chuan ^a,b   Spence, Charles ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b MCMASTER UNIVERSITY   (Canada)

Author keywords

Audiovisual; Cross modal semantic congruency; Environmental sounds; Speech

Indexed keywords

AUDIOVISUAL; CROSS-MODAL; ENVIRONMENTAL SOUNDS; INHIBITORY EFFECT; NOISE CONDITIONS; SEMANTIC INTERACTIONS; SEMANTIC PRIMINGS; SEMANTIC REPRESENTATION;

AUDITION; SPEECH;

SEMANTICS;

ADULT; ARTICLE; AUDITORY STIMULATION; COMPARATIVE STUDY; FEMALE; HUMAN; MALE; METHODOLOGY; PATTERN RECOGNITION; PHOTOSTIMULATION; PHYSIOLOGY; REACTION TIME; SEMANTICS; SPEECH PERCEPTION; YOUNG ADULT; PROCEDURES;

ACOUSTIC STIMULATION; ADULT; FEMALE; HUMANS; MALE; PATTERN RECOGNITION, VISUAL; PHOTIC STIMULATION; REACTION TIME; SEMANTICS; SPEECH PERCEPTION; YOUNG ADULT;

EID: 84893235183     PISSN: 22134794     EISSN: 22134808     Source Type: Journal    
DOI: 10.1163/22134808-00002420     Document Type: Article

References (38)

1. Alais, D., Morrone, C. and Burr, D. (2006). Separate attentional resources for vision and audition, Proc. R. Soc. B 273, 1339-1345. (Pubitemid 47308753)
2. Arnell, K. M. and Larson, J. M. (2002). Cross-modality attentional blinks without preparatory task-set switching, Psychon. B. Rev. 9, 497-506.
3. Arrighi, R., Lunardi, R. and Burr, D. (2011). Vision and audition do not share attentional resources in sustained tasks, Front. Psychol. 2, 56.
4. Ballas, J. A. (1993). Common factors in the identification of an assortment of brief everyday sounds, J. Exp. Psychol. Human 19, 240-267.
5. Bates, E., D?Amico, S., Jacobsen, T., Székely, A., Andonova, E., Devescovi, A., Herron, D., Ching Lu, C., Pechmann, T., Pléh, C., Wicha, N., Federmeier, K., Gerdjikova, I., Gutierrez, G., Hung, D., Hsu, J., Iyer, G., Kohnert, K., Mehotcheva, T., Orozco-Figueroa, A., Tzeng, A. and Tzeng, O. (2003). Timed picture naming in seven languages, Psychon. Bull. Rev. 10, 344-380. (Pubitemid 37544956)
6. Chen, Y.-C. and Spence, C. (2010). When hearing the bark helps to identify the dog: Semantically congruent sounds modulate the identification of masked pictures, Cognition 114, 389-404.
7. Chen, Y.-C. and Spence, C. (2011). Cross-modal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity, J. Exp. Psychol. Human 37, 1554-1568.
8. Coltheart, M. (2000). Dual routes from print to speech and dual routes from print to meaning: some theoretical issues, in:Reading as a Perceptual Process , A. Kennedy, R. Radach, J. Pynte and D. Heller (Eds), pp. 475-490. Elsevier, Oxford, UK.
9. Cummings, A., Ceponiene, R., Koyama, A., Saygin, A. P., Townsend, J. and Dick, F. (2006). Auditory semantic networks for words and natural sounds, Brain Res. 1115, 92-107. (Pubitemid 44436091)
10. Dick, F., Bussiere, J. and Saygin, A. P. (2002). The effects of linguistic mediation on the identification of environmental sounds, Center for Research in Language Newsletter 14, 3-9.
11. Donohue, S. E., Appelbaum, L. G., Park, C. J., Roberts, K. C. and Woldorff, M. G. (submitted). Cross-modal stimulus conflict: the effects of stimulus input timing in a visual-auditory Stroop task.
12. Duncan, J., Martens, S. and Ward, R. (1997). Restricted attentional capacity within but not between sensory modalities, Nature 387, 808-810. (Pubitemid 27270906)
13. Elliott, E. M., Cowan, N. and Valle-Inclan, F. (1998). The nature of cross-modal color-word interference effects, Percept. Psychophys. 60, 761-767. (Pubitemid 128461656)
14. Glaser, W. R. and Glaser, M. O. (1989). Context effects in Stroop-like word and picture processing, J. Exp. Psychol. Gen. 118, 13-42.
15. Grill-Spector, K. and Kanwisher, N. (2005). Visual recognition: as soon as you know it is there, you know what it is, Psychol. Sci. 16, 152-160. (Pubitemid 40306134)
16. Hays,W. (1973). Statistics for the Social Sciences. Holt, Rinehart andWinston, New York, NY, USA.
17. Iordanescu, L., Guzman-Martinez, E., Grabowecky, M. and Suzuki, S. (2008). Characteristic sound facilitates visual search, Psychon. B. Rev. 15, 548-554.
18. Iordanescu, L., Grabowecky, M., Franconeri, S., Theeuwes, J. and Suzuki, S. (2010). Characteristic sounds make you look at target objects more quickly, Atten. Percept. Psychophys. 72, 1736-1741.
19. Klemen, J., Buchel, C. and Rose, M. (2009). Perceptual load interacts with stimulus processing across sensory modalities, Eur. J. Neurosci. 29, 2426-2434.
20. Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H. andWallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance, Exp. Brain Res. 158, 405-414. (Pubitemid 39371385)
21. Mack, M. L., Gauthier, I., Sadr, J. and Palmeri, T. J. (2008). Object detection and basic-level categorization: sometimes you know it is there before you know what it is, Psychon. B. Rev. 15, 28-35.
22. Marcell, M. M., Borella, D., Greene, M., Kerr, E. and Rogers, S. (2000). Confrontation naming of environmental sounds, J. Clin. Exp. Neuropsyc. 22, 830-864. (Pubitemid 32187330)
23. Marcell, M.,Malatanos,M., Leahy, C. and Comeaux, C. (2007). Identifying, rating and remembering environmental sound events, Behav. Res. Methods 39, 561-569.
24. Murray, M. M. and Spierer, L. (2009). Auditory spatio-temporal brain dynamics and their consequences for multisensory interactions in humans, Hearing Res. 258, 121-133.
25. Murray, M. M., Camen, C., Andino, S. L. G., Bovet, P. and Clarke, S. (2006). Rapid brain discrimination of sounds of objects, J. Neurosci. 26, 1293-1302. (Pubitemid 43237061)
26. Robertson, I. H., Mattingley, J. B., Rorden, C. and Driver, J. (1998). Phasic alerting of neglect patients overcomes their spatial deficit in visual awareness, Nature 395, 169-173.
27. Roelofs, A. (2005). The visual-auditory color-word Stroop asymmetry and its time course, Mem. Cognit. 33, 1325-1336. (Pubitemid 46207048)
28. Santangelo, V., Belardinelli, M. O. and Spence, C. (2007). The suppression of reflexive visual and auditory orienting when attention is otherwise engaged, J. Exp. Psychol. Human 33, 137-148.
29. Saygin, A. P., Dick, F. and Bates, E. (2005). An on-line task for contrasting auditory processing in the verbal and nonverbal domains and norms for younger and older adults, Behav. Res. Methods 37, 99-110. (Pubitemid 41560263)
30. Schneider, T. R., Engel, A. K. and Debener, S. (2008). Multisensory identification of natural objects in a two-way cross-modal priming paradigm, Exp. Psychol. 55, 121-132.
31. Shimada, H. (1990). Effect of auditory presentation of words on color naming: the intermodal Stroop effect, Percept. Motor Skill 70, 1155-1161. (Pubitemid 20286570)
32. Shore, D. I., Barnes, M. E. and Spence, C. (2006). Temporal aspects of the visuotactile congruency effect, Neurosci. Lett. 392, 96-100. (Pubitemid 41757583)
33. Smith, M. C. and Magee, L. E. (1980). Tracing the time course of picture-word processing, J. Exp. Psychol. Gen. 109, 373-392.
34. Snodgrass, J. G. and Vanderwart, M. (1980). A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity, J. Exp. Psychol.- Hum. Learn. 6, 174-215.
35. Spence, C. and Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting, Percept. Psychophys. 59, 1-22. (Pubitemid 127456363)
36. Stuart, D. M. and Carrasco, M. (1993). Semantic component of a cross-modal Stroop-like task, Am. J. Psychol. 106, 383-405.
37. Van Petten, C., Coulson, S., Rubin, S., Plante, E. and Parks, M. (1999). Time course of word identification and semantic integration in spoken language, J. Exp. Psychol. Learn. 25, 394-417.
38. Yuval-Greenberg, S. and Deouell, L. Y. (2009). The dog?s meow: asymmetrical interaction in cross-modal object recognition, Exp. Brain Res. 193, 603-614.