I can see you better if I can hear you coming: Action-consistent sounds facilitate the visual detection of human gait

Journal of Vision

Volumn 10, Issue 12, 2010, Pages 1-11

  Thomas, James Philip ^a   Shiffrar, Maggie ^a  

^a RUTGERS UNIVERSITY   (United States)

Author keywords

Action perception; Biological motion; Multisensory integration; Point light walkers

Indexed keywords

ARTICLE; ASSOCIATION; GAIT; HEARING; HUMAN; METHODOLOGY; MOTOR CORTEX; MOVEMENT (PHYSIOLOGY); MOVEMENT PERCEPTION; PATTERN RECOGNITION; PHOTOSTIMULATION; PHYSIOLOGY; PSYCHOPHYSICS; YOUNG ADULT;

AUDITORY PERCEPTION; CUES; GAIT; HUMANS; MOTION PERCEPTION; MOTOR CORTEX; MOVEMENT; PATTERN RECOGNITION, VISUAL; PHOTIC STIMULATION; PSYCHOPHYSICS; YOUNG ADULT;

EID: 78449240519     PISSN: None     EISSN: 15347362     Source Type: Journal    
DOI: 10.1167/10.12.14     Document Type: Article

References (59)

1. Alais, D., & Burr, D. (2004). No direction-specific bimodal facilitation for audiovisual motion detection. Cognitive Brain Research, 19, 185-194.
2. Arrighi, R., Marini, F., & Burr, D. (2009). Meaningful auditory information enhances perception of visual biological motion. Journal of Vision, 9(4):25, 1-7, http://www.journalofvision.org/content/9/4/25, doi:10.1167/9.4.25.
3. Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W., & Perrett, D. I. (2005). Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience, 17, 377-391.
4. Beauchamp, M. S., Lee, K. E., Argall, B. D., & Martin, A. (2004). Integration of auditory and visual information about objects in superior temporal sulcus. Neuron, 41, 809-823.
5. Bertenthal, B. I. (1993). Perception of biomechanical motions by infants: Intrinsic image and knowledge-based constraints. In C. Granrud (Ed.), Carnegie symposium on cognition: Visual perception and cognition in infancy (pp. 175-214). Hillsdale, NJ: Erlbaum.
6. Bertenthal, B. I., & Pinto, J. (1994). Global processing of biological motion. Psychological Science, 5, 221-225.
7. Bidet-Caulet, A., Voisin, J., Bertrand, O., & Fonlupt, P. (2005). Listening to a walking human activates the temporal biological motion area. Neuroimage, 28, 132-139.
8. Blake, R., & Shiffrar, M. (2007). Perception of human motion. Annual Review of Psychology, 58, 47-74.
9. Bolognini, N., Frassinetti, F., Serino, A., & Làdavas, E. (2005). "Acoustical vision" of below threshold stimuli: Interaction among spatially converging audiovisual inputs. Experimental Brain Research, 160, 273-282.
10. Bonda, E., Petrides, M., Ostry, D., & Evans, A. (1996). Specific involvement of human parietal systems and the amygdala in the perception of biological motion. Journal of Neuroscience, 16, 3737-3744.
11. Brooks, A., van der Zwan, R., Billard, A., Petreska, B., & Blanke, O. (2007). Auditory motion affects visual biological motion processing. Neuropsychologia, 45, 523-530.
12. Bruce, C., Desimone, R., & Gross, C. G. (1981). Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology, 46, 369-384.
13. Calvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-657.
14. Chen, Y.-C., & Spence, C. (2010). When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures. Cognition, 114, 389-404.
15. Dittrich, W. H. (1993). Action categories and the perception of biological motion. Perception, 22, 15-22.
16. Driver, J., & Spence, C. (1998). Attention and the crossmodal construction of space. Trends in Cognitive Sciences, 2, 254-262.
17. Frassinetti, F., Bolognini, N., & Làdavas, E. (2002). Enhancement of visual perception by crossmodal visuo-auditory interaction. Experimental Brain Research, 147, 332-323.
18. Gazzola, V., Aziz-Zadeh, L., & Keysers, C. (2006). Empathy and the somatotopic auditory mirror system in humans. Current Biology, 16, 1824-1829.
19. Grossman, E. D., & Blake, R. (2001). Brain activity evoked by inverted and imagined biological motion. Vision Research, 40, 1475-1482.
20. Grossman, E. D., & Blake, R. (2002). Brain areas active during visual perception of biological motion. Neuron, 35, 1167-1175.
21. Hein, G., Doerhrmann, O., Muller, N. G., Kaiser, J., Muckli, L., & Naumer, M. J. (2007). Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. Journal of Neuroscience, 27, 7881-7887.
22. Iordanescu, L., Guzman-Martinez, E., Grabowecky, M., & Suzuki, S. (2008). Characteristic sounds facilitate visual search. Psychonomic Bulletin & Review, 15, 548-554.
23. Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception & Psychophysics, 14, 201-211.
24. Kaiser, M. D., Delmolino, L., Tanaka, J. W., & Shiffrar, M. (2010). Comparison of visual sensitivity to human and object motion in Autism Spectrum Disorder. Autism Research, 3, 191-195.
25. Keysers, C., Kohler, E., Umilta, M. A., Nanetti, L., Fogassi, L., & Gallese, V. (2003). Audiovisual mirror neurons and action recognition. Experimental Brain Research, 153, 628-636.
26. Kohler, E., Keysers, C., Umiltà, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). Hearing sounds, understanding actions: Action representation in mirror neurons. Science, 297, 846-848.
27. Lahav, A., Saltzman, E., & Schlaug, G. (2007). Action representation of sound: Audiomotor recognition network while listening to newly acquired actions. Journal of Neuroscience, 27, 308-314.
28. Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H., & Wallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 158, 405-414.
29. McMillan, N., & Creelman, C. (1991). Detection theory: A user's guide. Cambridge, UK: Cambridge University Press.
30. Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452-465.
31. Pavlova, M., & Sokolov, A. (2000). Orientation specificity in biological motion perception. Perception & Psychophysics, 6, 889-899.
32. Pavlova, M., & Sokolov, A. (2003). Prior knowledge about display inversion in biological motion perception. Perception, 32, 937-946.
33. Pelphrey, K. A., Mitchell, T. V., McKeown, M. J., Goldstein, J., Allison, T., & McCarthy, G. (2003). Brain activity evoked by the perception of human walking: Controlling for meaningful coherent motion. Journal of Neuroscience, 23, 6819-6825.
34. Petrini, K., Dahl, S., Rocchesso, D., Waadeland, C. H., Avanzini, F., Puce, A., et al. (2009). Multisensory integration of drumming actions. Musical experience affects perceived audiovisual asynchrony. Experimental Brain Research, 198, 339-352.
35. Petrini, K., Holt, S. P., & Pollick, F. (2010). Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. Journal of Vision, 10(5):2, 1-14, http://www.journalofvision.org/content/10/5/2, doi:10.1167/10.5.2.
36. Petrini, K., Russell, M., & Pollick, F. (2009). When knowing can replace seeing in audiovisual integration of actions. Cognition, 111, 432-439.
37. Pinto, J., & Shiffrar, M. (2009). The visual perception of human and animal motion in point-light displays. Social Neuroscience, 4, 332-246.
38. Puce, A., & Perrett, D. (2003). Electrophysiology and brain imaging of biological motion. Philosophical Transactions of the Royal Society of London, 358, 435-445.
39. Pyles, J. A., Garcia, J. O., Hoffman, D. D., & Grossman, E. D. (2007). Visual perception and neural correlates of novel "biological motion". Vision Research, 47, 2278-2797.
40. Reed, C. L., Stone, V. E., Bozova, S., & Tanaka, J. (2003). The body-inversion effect. Psychological Research, 14, 302-308.
41. Saarela, M. V., & Hari, R. (2008). Listening to humans walking together activates the social brain circuitry. Social Neuroscience, 3, 401-409.
42. Saygin, A. P. (2007). Superior temporal and premotor brain areas necessary for biological motion perception. Brain, 130, 2452-2461.
43. Saygin, A. P., Driver, J., & de Sa, V. R. (2008). In the footsteps of biological motion and multisensory perception: Judgments of audiovisual temporal relations are enhanced for upright walkers. Psychological Science, 19, 469-475.
44. Saygin, A. P., Wilson, S. M., Hagler, D. J., Bates, E., & Sereno, M. I. (2004). Point-light biological motion perception activates human premotor cortex. Journal of Neuroscience, 24, 6181-6188.
45. Schutz, M., & Kubovy, M. (2009). Causality and cross-modal integration. Journal of Experimental Psychology: Human Perception and Performance, 35, 1791-1810.
46. Seltzer, B., & Pandya, D. N. (1978). Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey. Brain Research, 149, 1-24.
47. Shimojo, S., & Shams, L. (2001). Sensory modalities are not separate modalities: Plasticity and interactions. Current Opinion in Neurobiology, 11, 505-509.
48. Spence, C. (2007). Audiovisual multisensory integration. Acoustical Science and Technology, 28, 61-70.
49. Sumi, S. (1984). Upside-down presentation of the Johansson moving light-spot pattern. Perception, 13, 283-286.
50. Taylor, K. I., Moss, H. E., Stamatakis, E. A., & Tyler, L. K. (2006). Binding crossmodal object features in perirhinal cortex. Proceedings of the National Academy of Sciences, 21, 8239-8244.
51. Taylor, K. I., Stamatakis, E. A., & Tyler, L. K. (2009). Crossmodal integration of object features: Voxel-based correlations in brain-damaged patients. Brain, 132, 671-683.
52. van der Zwan, R., MacHatch, C., Kozlowski, D., Troje, N. F., Blanke, O., & Brooks, O. (2009). Gender bending: Auditory cues affect visual judgments of gender in biological motion displays. Experimental Brain Research, 198, 373-382.
53. Vatakis, A., & Spence, C. (2007a). Crossmodal binding: Evaluating the "unity assumption" using audiovisual speech stimuli. Perception & Psychophysics, 69, 744-756.
54. Vatakis, A., & Spence, C. (2007b). How special is the human face? Evidence from an audiovisual temporal order judgment task. Neuroreport, 18, 1807-1811.
55. Vatakis, A., & Spence, C. (2008a). Evaluating the influence of the "unity assumption" on the temporal perception of realistic audiovisual stimuli. Acta Psychologica, 127, 12-23.
56. Vatakis, A., & Spence, C. (2008b). Investigating the effects of inversion on configural processing with an audiovisual temporal-order judgment task. Perception, 37, 143-160.
57. Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26, 1583-1590.
58. Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638-667.
59. Yin, R. K. (1969). Looking at upside-down faces. Journal of Experimental Psychology, 81, 141-145.