When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures

Cognition

Volumn 114, Issue 3, 2010, Pages 389-404

  Chen, Yi Chuan ^a   Spence, Charles ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Audiovisual; Multisensory; Object identification; Semantic congruency

Indexed keywords

ADULT; ARTICLE; AUDIOVISUAL EQUIPMENT; FEMALE; HEARING; HUMAN; HUMAN EXPERIMENT; MALE; NORMAL HUMAN; PRIORITY JOURNAL; SEMANTICS; SHORT TERM MEMORY; SOUND; TASK PERFORMANCE; WHITE NOISE;

ADULT; ANIMALS; DOGS; FEMALE; HUMANS; MALE; PERCEPTUAL MASKING; SEMANTICS; SIGNAL DETECTION, PSYCHOLOGICAL; VOCALIZATION, ANIMAL;

EID: 76349106681     PISSN: 00100277     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cognition.2009.10.012     Document Type: Article

References (138)

1. Alais D., Morrone C., and Burr D. Separate attentional resources for vision and audition. Proceedings of the Royal Society B 273 (2006) 1339-1345
2. Arnell K.M., and Larson J.M. Cross-modality attentional blinks without preparatory task-set switching. Psychonomic Bulletin & Review 9 (2002) 497-506
3. Arnold D.H., Johnston A., and Nishida S. Timing sight and sound. Vision Research 45 (2005) 1275-1284
4. Ballas J.A. Common factors in the identification of an assortment of brief everyday sounds. Journal of Experimental Psychology: Human Perception and Performance 19 (1993) 250-267
5. Barsalou L.W. Perceptual symbol systems. Behavioral and Brain Sciences 22 (1999) 577-660
6. Beauchamp M.S., Argall B.D., Bodurka J., Duyn J.H., and Martin A. Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neuroscience 7 (2004) 1190-1192
7. Beauchamp M.S., Lee K.E., Argall B.D., and Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41 (2004) 809-823
8. Belardinelli M.O., Sestieri C., Di Matteo R., Delogu F., Del Gratta C., Ferretti A., et al. Audio-visual crossmodal interactions in environmental perception: An fMRI investigation. Cognitive Processing 5 (2004) 167-174
9. Bertelson P., and Aschersleben G. Automatic visual bias of perceived auditory location. Psychonomic Bulletin & Review 5 (1998) 482-489
10. Biederman I. Perceiving real-world scenes. Science 177 (1972) 77-80
11. Biederman I., Mezzanotte R.J., and Rabinowitz J.C. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive Psychology 14 (1982) 143-177
12. Bowman H., and Wyble B. The simultaneous type, serial token model of temporal attention and working memory. Psychological Review 114 (2007) 38-70
13. Boyce S.J., and Pollastek A. Identification of objects in scenes: The role of scene background in object naming. Journal of Experimental Psychology: Learning, Memory, and Cognition 18 (1992) 531-543
14. Boyce S.J., Pollastek A., and Rayner K. Effect of background information on object identification. Journal of Experimental Psychology: Human Perception and Performance 15 (1989) 556-566
15. Brand-D'Abrescia M., and Lavie N. Task coordination between and within sensory modalities: Effects on distraction. Perception & Psychophysics 70 (2008) 508-515
16. In: Calvert G.A., Spence C., and Stein B.E. (Eds). The handbook of multisensory processes (2004), MIT Press, Cambridge, MA
17. Caramazza A., and Mahon B.Z. The organization of conceptual knowledge: The evidence from category-specific semantic deficits. Trends in Cognitive Sciences 7 (2003) 354-361
18. Chan J.S., and Newell F.N. Behavioral evidence for task-dependent "what" versus "where" processing within and across modalities. Perception & Psychophysics 70 (2008) 36-49
19. Choe C.S., Welch R.B., Gilford R.M., and Juola J.F. The "ventriloquist effect": Visual dominance or response bias?. Perception & Psychophysics 18 (1975) 55-60
20. Colavita F.B. Human sensory dominance. Perception & Psychophysics 16 (1974) 409-412
21. Dalton P., and Spence C. Attentional capture in serial audiovisual search tasks. Perception & Psychophysics 69 (2007) 422-438
22. Davenport J.L., and Potter M.C. Scene consistency in object and background perception. Psychological Science 15 (2004) 559-564
23. Dijkerman H.C., and de Haan E.H.F. Somatosensory processes subserving perception and action. Behavioral and Brain Sciences 30 (2007) 189-239
24. Doehrmann O., and Naumer M.J. Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration. Brain Research 1242 (2008) 136-150
25. Driver J., and Noesselt T. Multisensory interplay reveals crossmodal influences on "sensory specific" brain regions, neural responses, and judgments. Neuron 57 (2008) 11-23
26. Driver J., and Vuilleumier P. Perceptual awareness and its loss in unilateral neglect and extinction. Cognition 79 (2001) 39-88
27. Duncan J., Martens S., and Ward R. Restricted attentional capacity within but not between sensory modalities. Nature 387 (1997) 808-810
28. Folk C.L., Remington R.W., and Johnston J.C. Involuntary covert orienting is contingent on attentional control settings. Journal of Experimental Psychology: Human Perception and Performance 18 (1992) 1030-1044
29. Fujisaki W., Shimojo S., Kashino M., and Nishida S. Recalibration of audiovisual simultaneity. Nature Neuroscience 7 (2004) 773-778
30. Gazzaley A., Rissman J., and D'Esposito M. Functional connectivity during working memory maintenance. Cognitive, Affective, & Behavioral Neuroscience 4 (2004) 580-599
31. Giard M.H., and Peronnet F. Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience 11 (1999) 473-490
32. Glaser W.R., and Dungelhoff F.-J. The time course of picture-word interference. Journal of Experimental Psychology: Human Perception and Performance 10 (1984) 640-654
33. Glaser W.R., and Glaser M.O. Context effects in Stroop-like word and picture processing. Journal of Experimental Psychology: General 118 (1989) 13-42
34. Green D.M., and Swets J.A. The sensory threshold and psychophysical method. In: John A. (Ed). Signal detection theory and psychophysics (1966), Wiley, New York 117-148
35. Hancock P.A., Oron-Gilad T., and Szalma J.L. Elaborations of the multiple-resource theory of attention. In: Kramer A.F., Wiegmann D.A., and Kirlik A. (Eds). Attention: From theory to practice (2007), Oxford University Press, Oxford 45-56
36. Hecht D., and Reiner M. Sensory dominance in combinations of audio, visual and haptic stimuli. Experimental Brain Research 193 (2009) 307-314
37. Hein G., Doehrmann O., Muller N.G., Kaiser J., Muckli L., and Naumer M.J. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. Journal of Neuroscience 27 (2007) 7881-7887
38. Henderson J.M., and Hollingworth A. High-level scene perception. Annual Review of Psychology 50 (1999) 243-271
39. Hollingworth A., and Henderson J.M. Does consistent scene context facilitate object perception?. Journal of Experimental Psychology: General 127 (1998) 398-415
40. Intraub H. Rapid conceptual identification of sequentially presented pictures. Journal of Experimental Psychology: Human Perception and Performance 7 (1981) 604-610
41. Intraub H. Conceptual masking: The effects of subsequent visual events on memory for pictures. Journal of Experimental Psychology: Learning, Memory, and Cognition 10 (1984) 115-125
42. Intraub H. Understanding and remembering briefly glimpsed pictures: Implications for visual scanning and memory. In: Coltheart V. (Ed). Fleeting memories: Cognition of brief visual stimuli (1999), MIT Press, Cambridge, MA 47-70
43. Iordanescu L., Guzman-Martinez E., Grabowecky M., and Suzuki S. Characteristic sound facilitates visual search. Psychonomic Bulletin & Review 15 (2008) 548-554
44. Jackson C.V. Visual factors in auditory localization. Quarterly Journal of Experimental Psychology 5 (1953) 52-65
45. Kaas J.H., and Hackett T.A. 'What' and 'where' processing in auditory cortex. Nature Neuroscience 2 (1999) 1045-1047
46. Kahneman D., Treisman A., and Burkell J. The cost of visual filtering. Journal of Experimental Psychology: Human Perception and Performance 9 (1983) 510-522
47. Kahneman D., Treisman A., and Gibbs B.J. The reviewing of object files: Object-specific integration of information. Cognitive Psychology 24 (1992) 174-219
48. Kanwisher N. Repetition blindness: Type recognition without token individuation. Cognition 27 (1987) 117-143
49. Kanwisher N. Neural events and perceptual awareness. Cognition 79 (2001) 89-113
50. Kiefer M. The N400 is modulated by unconsciously perceived masked words: Further evidence for an automatic spreading activation account of N400 priming effects. Cognitive Brain Research 13 (2002) 27-39
51. Kirk R.E. Experimental design: Procedures for the behavioral sciences (1995), Brooks/Cole Publishing Company, Pacific Grove, CA
52. Klemen, J., Buchel, C., Buhler, M., Menz, M. M., & Rose, M. (in press). Auditory working memory load impairs visual ventral stream processing: Toward a unified model of attentional load. Journal of Cognitive Neuroscience.
53. Koppen C., Alsius A., and Spence C. Semantic congruency and the Colavita visual dominance effect. Experimental Brain Research 184 (2008) 533-546
54. Kubovy M., and van Valkenburg D. Auditory and visual objects. Cognition 80 (2001) 97-126
55. Laurienti P.J., Kraft R.A., Maldjian J.A., Burdette J.H., and Wallace M.T. Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research 158 (2004) 405-414
56. Lavie N. Distracted and confused? Selective attention under load. Trends in Cognitive Sciences 9 (2005) 75-82
57. Lavie N., Hirst A., de Fockert J.W., and Viding E. Load theory of selective attention and cognitive control. Journal of Experimental Psychology: General 133 (2004) 339-354
58. Loftus G.R., and Ginn M. Perceptual and conceptual masking of pictures. Journal of Experimental Psychology: Learning, Memory, and Cognition 10 (1984) 435-441
59. Macmillan N.A., and Creelman C.D. Alternative approaches: Threshold models and choice theory. In: John A. (Ed). Detection theory: A user's guide (2005), Lawrence Erlbaum Associates, Mahwah, NJ 81-111
60. Marcel A.J. Conscious and unconscious perception: Experiments on visual masking and word recognition. Cognitive Psychology 15 (1983) 197-237
61. Marcel A.J. Conscious and unconscious perception: An approach to the relations between phenomenal experience and perceptual processes. Cognitive Psychology 15 (1983) 238-300
62. McCarthy R.A., and Warrington E.K. Evidence for modality-specific meaning systems in the brain. Nature 334 (1988) 428-430
63. McDonald J.J., Teder-Sälejärvi W.A., and Hillyard S.A. Involuntary orienting to sound improves visual perception. Nature 407 (2000) 906-908
64. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
65. Miller J. Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology 14 (1982) 247-279
66. Miller J. Channel interaction and the redundant-targets effect in bimodal divided attention. Journal of Experimental Psychology: Human Perception and Performance 17 (1991) 160-169
67. Milner A.D., and Goodale M.A. The visual brain in action (1995), Oxford University Press, New York
68. Mishkin M., Ungerleider L.G., and Macko K.A. Object vision and spatial vision: Two central pathways. Trends in Neurosciences 6 (1983) 414-417
69. Molholm S., Ritter W., Javitt D.C., and Foxe J.J. Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex 14 (2004) 452-465
70. Murray M.M., Michel C.M., Grave de Peralta R., Ortigue S., Brunet D., Gonzalez Andino S., et al. Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. NeuroImage 21 (2004) 125-135
71. Murray M.M., Molholm S., Michel C.M., Heslenfeld D.J., Ritter W., Javitt D.C., et al. Grabbing your ear: Rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cerebral Cortex 15 (2005) 963-974
72. Murray E.A., and Richmond B.J. Role of perirhinal cortex in object perception, memory, and associations. Current Opinion in Neurobiology 11 (2001) 188-193
73. Navarra J., Vatakis A., Zampini M., Soto-Faraco S., Humphreys W., and Spence C. Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Research 25 (2005) 499-507
74. Nickerson R.S. Intersensory facilitation of reaction time: Energy summation or preparation enhancement?. Psychological Review 80 (1973) 489-509
75. Noesselt T., Rieger J.W., Schoenfeld M.A., Kanowski M., Hinrichs H., Heinze H.-J., et al. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. Journal of Neuroscience 27 (2007) 11431-11441
76. Norman D.A., and Bobrow D.G. On data-limited and resource-limited processes. Cognitive Psychology 7 (1975) 44-64
77. O'Conner K.J., and Potter M.C. Constrained formation of object representations. Psychological Science 13 (2002) 106-111
78. Palmer S.E. The effects of contextual scenes on the identification of objects. Memory & Cognition 3 (1975) 519-526
79. Parise C., and Spence C. 'When birds of a feather flock together': Synesthetic associations modulate multisensory integration in non-synesthetes. PloS ONE 4 (2009) e5664 10.1371/journal.pone.0005664
80. Plaut D.C. Graded modality-specific specialization in semantics: A computational account of optic aphasia. Cognitive Neuropsychology 19 (2002) 603-639
81. Potter M.C. Meaning in visual search. Science 187 (1975) 965-966
82. Potter M.C. Short-term conceptual memory for pictures. Journal of Experimental Psychology: Human Learning and Memory 2 (1976) 509-522
83. Potter M.C. Very short-term conceptual memory. Memory & Cognition 21 (1993) 156-161
84. Potter M.C. Understanding sentences and scenes: The role of conceptual short-term memory. In: Coltheart V. (Ed). Fleeting memories: Cognition of brief visual stimuli (1999), MIT Press, Cambridge, MA 13-46
85. Potter M.C., Chun M.M., Banks B.S., and Muckenhoupt M. Two attentional deficits in serial target search: The visual attentional blink and an amodal task-switch deficit. Journal of Experimental Psychology: Learning, Memory, and Cognition 24 (1998) 979-992
86. Potter M.C., and Faulconer B.A. Time to understand pictures and words. Nature 253 (1975) 437-438
87. Potter M.C., Stiefbold D.R., and Moryadas A. Word selection in reading sentences: Preceding versus following contexts. Journal of Experimental Psychology: Learning, Memory, and Cognition 24 (1998) 68-100
88. Prinzmetal W., McCool C., and Park S. Attention: Reaction time and accuracy reveal different mechanisms. Journal of Experimental Psychology: General 134 (2005) 73-92
89. Pylyshyn Z.W. Computation and cognition (1984), MIT Press, Cambridge, MA
90. Pylyshyn Z.W. The role of location indexes in spatial perception: A sketch of the FINST spatial index model. Cognition 32 (1989) 65-97
91. Pylyshyn Z.W. Visual indexes, preconceptual objects, and situated vision. Cognition 80 (2001) 127-158
92. Raab D. Statistical facilitation of simple reaction times. Transactions of the New York Academy of Science 24 (1962) 574-590
93. Radeau M., and Bertelson P. Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception & Psychophysics 22 (1977) 137-146
94. Rauschecker J.P., and Tian B. Mechanisms and streams for processing of "what" and "where" in auditory cortex. Proceedings of the National Academy of Sciences USA 97 (2000) 11800-11806
95. Reed C.L., Klatzky R.L., and Halgren H. What vs. Where in touch: An fMRI study. NeuroImage 25 (2005) 718-726
96. Rees G., Frith C., and Lavie N. Processing of irrelevant visual motion during performance of an auditory attention task. Neuropsychologia 39 (2001) 937-949
97. Robertson I.H., Mattingley J.B., Rorden C., and Driver J. Phasic alerting of neglect patients overcomes their spatial deficit in visual awareness. Nature 395 (1998) 169-172
98. Romanski L.M., Tian B., Fritz J., Mishkin M., Goldman-Rakic P.S., and Rauschecker J.P. Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nature Neuroscience 2 (1999) 1131-1136
99. Santangelo V., Belardinelli M.O., and Spence C. The suppression of reflexive visual and auditory orienting when attention is otherwise engaged. Journal of Experimental Psychology: Human Perception and Performance 33 (2007) 137-148
100. Santee J.L., and Egeth H.E. Do reaction time and accuracy measure the same aspects of letter recognition?. Journal of Experimental Psychology: Human Perception and Performance 8 (1982) 489-501
101. Schneider T.R., Debener S., Oostenveld R., and Engel A.K. Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming. NeuroImage 42 (2008) 1244-1254
102. Schneider W., Dumais S.T., and Shiffrin R.M. Automatic and control processing and attention. In: Parasuraman R., and Davies R. (Eds). Varieties of attention (1984), Academic Press, New York 1-27
103. Schneider T.R., Engel A.K., and Debener S. Multisensory identification of natural objects in a two-way crossmodal priming paradigm. Experimental Psychology 55 (2008) 121-132
104. Schroeder C.E., and Foxe J. Multisensory contributions to low-level, 'unisensory' processing. Current Opinion in Neurobiology 15 (2005) 454-458
105. Shallice T. Specialization within the semantic system. Cognitive Neuropsychology 5 (1988) 133-142
106. Shams L., Iwaki S., Chawla A., and Bhattacharya J. Early modulation of visual cortex by sound: An MEG study. Neuroscience Letters 378 (2005) 76-81
107. Shimojo S., Watanabe K., and Scheier C. The resolution of ambiguous motion: Attentional modulation and development. In: Braun J., Koch C., and Davis J.L. (Eds). Visual attention and control circuits (2001), MIT Press, Cambridge, MA 243-264
108. Sinnett S., Soto-Faraco S., and Spence C. The co-occurrence of multisensory competition and facilitation. Acta Psychologica 128 (2008) 153-161
109. Snodgrass J.G., and Vanderwart M. A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Human Learning and Memory 6 (1980) 174-215
110. Soto-Faraco S., and Alsius A. Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance 35 (2009) 580-587
111. Soto-Faraco S., and Spence C. Modality-specific auditory and visual temporal processing deficits. Quarterly Journal of Experimental Psychology A 55 (2002) 23-40
112. Spence C. Audiovisual multisensory integration. Acoustical Science & Technology 28 (2007) 61-70
113. Spence, C., & Soto-Faraco, S. (in press). Auditory perception: Interactions with vision. In C. Plack (Ed.), Handbook of auditory science. Oxford: Oxford University Press.
114. Spence C., and Driver J. Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics 59 (1997) 1-22
115. Spence, C., Parise, C., & Chen, Y.-C. (submitted for publication). Toward a cognitive neuroscience explanation of the Colavita visual dominance effect. In M. M. Murray & M. Wallace (Eds.), Frontiers in the neural bases of multisensory processes.
116. Spence C., Nicholls M.E.R., and Driver J. The cost of expecting events in the wrong sensory modality. Perception & Psychophysics 63 (2001) 330-336
117. Spence C., Ranson J., and Driver J. Cross-modal selective attention: On the difficulty of ignoring sounds at the locus of visual attention. Perception & Psychophysics 62 (2000) 410-424
118. Stanfield R.A., and Zwaan R.A. The effect of implied orientation derived from verbal context on picture recognition. Psychological Science 12 (2001) 153-156
119. Stein B.E., and Meredith M.A. The merging of the senses (1993), MIT Press, Cambridge, MA
120. Stone J.V., Hunkin N.M., Porrill J., Wood R., Keeler V., Beanland M., et al. When is now? Perception of simultaneity. Proceedings of the Royal Society B 268 (2001) 31-38
121. Suied C., Bonneel N., and Viaud-Delmon I. Integration of auditory and visual information in the recognition of realistic objects. Experimental Brain Research 194 (2009) 91-102
122. Taylor K.I., Moss H.E., Stamatakis E.A., and Tyler L.K. Binding crossmodal object features in perirhinal cortex. Proceedings of the National Academy of Sciences USA 103 (2006) 8239-8244
123. Treisman A., and Davies A. Dividing attention to ear and eye. In: Kornblulm S. (Ed). Attention and performance Vol. IV (1973), Academic Press, New York 101-117
124. Treisman A., and Riley J.G.A. Is selective attention selective perception or selective response? A further test. Journal of Experimental Psychology 79 (1969) 27-34
125. Tyler L.K., Stamatakis E.A., Bright P., Acres K., Abdallah S., Rodd J.M., et al. Processing objects at different levels of specificity. Journal of Cognitive Neuroscience 16 (2004) 351-362
126. Ungerleider L.G., and Mishkin M. Two cortical visual systems. In: Ingle D.J., Goodale M.A., and Mansfield R.J. (Eds). Analysis of visual behavior (1982), MIT Press, Cambridge, MA 549-580
127. Vatakis A., Ghazanfar A., and Spence C. Facilitation of multisensory integration by the 'unity assumption': Is speech special?. Journal of Vision 8 914 (2008) 1-11
128. Vatakis A., and Spence C. Crossmodal binding: Evaluating the "unity assumption" using audiovisual speech stimuli. Perception & Psychophysics 69 (2007) 744-756
129. Vatakis A., and Spence C. Evaluating the influence of the 'unity assumption' on the temporal perception of realistic audiovisual stimuli. Acta Psychologica 127 (2008) 12-23
130. Versace R., Labeye E., Badard G., and Rose M. The contents of long-term memory and the emergence of knowledge. European Journal of Cognitive Psychology 21 (2009) 522-560
131. Watt R.J. Studying human vision. Understanding vision (1991), London, Academic Press (pp. 191-216)
132. Welch R.B. Meaning, attention, and the 'unity assumption' in the intersensory bias of spatial and temporal perceptions. In: Ashersleben G., Bachmann T., and Musseler J. (Eds). Cognitive contributions to the perception of spatial and temporal events (1999), Elsevier Science, Amsterdam 371-387
133. Welch R.B., and Warren D.H. Immediate perceptual response to intersensory discrepancy. Psychological Bulletin 3 (1980) 638-667
134. Wickens C.D. The structure of attentional resources. In: Nickerson R.S. (Ed). Attention and performance Vol. VIII (1980), Erlbaum, Hillsdale, NJ 239-257
135. Wickens C.D. Processing resources in attention. In: Parasuraman R., and Davies D.R. (Eds). Varieties of attention (1984), Academic Press, San Diego, CA 63-102
136. Yeh S.L., and Liao H.I. On the generality of the contingent orienting hypothesis. Acta Psychologica 129 (2008) 157-165
137. Yuval-Greenberg S., and Deouell L.Y. The dog's meow: Asymmetrical interaction in cross-modal object recognition. Experimental Brain Research 193 (2009) 603-614
138. Zampini M., Guest S., Shore D.I., and Spence C. Audio-visual simultaneity judgments. Perception & Psychophysics 67 (2005) 531-544