Causality and Cross-Modal Integration

Journal of Experimental Psychology: Human Perception and Performance

Volumn 35, Issue 6, 2009, Pages 1791-1810

  Schutz, Michael ^a   Kubovy, Michael ^a  

^a UNIVERSITY OF VIRGINIA   (United States)

Author keywords

auditory; causality; optimal integration; sensory integration; visual

Indexed keywords

ARTICLE; AUDITORY STIMULATION; GESTURE; HEARING; HUMAN; ILLUSION; PHOTOSTIMULATION; PHYSIOLOGY; PSYCHOLOGICAL ASPECT; PSYCHOMOTOR PERFORMANCE; SOUND; TIME; UNCERTAINTY; VISION;

ACOUSTIC STIMULATION; AUDITORY PERCEPTION; GESTURES; HUMANS; ILLUSIONS; PHOTIC STIMULATION; PSYCHOMOTOR PERFORMANCE; SOUND; TIME FACTORS; UNCERTAINTY; VISUAL PERCEPTION;

EID: 73649141367     PISSN: 00961523     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0016455     Document Type: Article

References (88)

1. Alais D., and Burr D. The ventriloquist effect results from near-optimal bimodal integration. Current Biology 14 (2004) 257-262
2. Alais D., and Carlile S. Synchronizing to real events: Subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proceedings of the National Academy of Sciences, USA 102 (2005) 2244-2247
3. Arieh Y., and Marks L.E. Cross-modal interaction between vision and hearing: A speed-accuracy analysis. Perception and Psychophysics 70 (2008) 412-421
4. Armontrout J., Schutz M., and Kubovy M. Visual determinants of a cross-modal illusion. Attention, Perception, and Psychophysics 71 (2009) 1618-1627
5. Arrighi R., Alais D., and Burr D. Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. Journal of Vision 6 (2006) 260-268
6. Baayen R.H. Analyzing linguistic data: A practical introduction to statistics (2008), Cambridge University Press, Cambridge, England
7. Baayen, R. H., Davidson, D. J., & Bates, D. M. (in press). Mixed-effects modeling with crossed random effects for subjects and items. Retrieved June 5, 2008, from http://dx.doi.org/10.1016/j.jml.2007.12.005
8. Bates D., and Sarkar D. lme4: Linear mixed-effects models using S4 classes (R package version 0.9975-13) (2007)
9. Battaglia P.W., Jacobs R.A., and Aslin R.N. Bayesian integration of visual and auditory signals for spatial localization. Journal of the Optical Society of America 20 (2003) 1391-1397
10. Bedford F.L. Object identity theory and the nature of general laws. Cahiers de Psychologie Cognitive-Current Psychology of Cognition 20 (2001) 277-293
11. Bedford F.L. Towards a general law of numerical/object identity. Cahiers de Psychologie Cognitive-Current Psychology of Cognition 20 (2001) 113-175
12. Bedford F.L. Analysis of a constraint on perception, cognition, and development: One object, one place, one time. Journal of Experimental Psychology: Human Perception and Performance 30 (2004) 907-912
13. Bertelson P., and Radeau M. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Perception and Psychophysics 29 (1981) 578-584
14. Bertelson P., Vroomen J., de Gelder B., and Driver J. The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception and Psychophysics 62 (2000) 321-332
15. Bork I. Practical tuning of xylophone bars and resonators. Applied Acoustics 46 (1995) 103-127
16. Cleveland W.S. Visualizing data (1993), AT&T, Murray Hill, NJ
17. Congedo M., Lécuyer A., and Gentaz E. The influence of spatial delocation on perceptual integration of vision and touch. Presence: Teleoperators and Virtual Environments 15 (2003) 353-357
18. Crawley M.J. The r book (2007), Wiley, New York
19. de Gelder B., Pourtois G., and Weiskrantz L. Fear recognition in the voice is modulated by unconsiously recognized facial expressions but not by unconsciously recognized affective pictures. Proceedings of the National Academy of the Sciences, USA 99 (2002) 4121-4126
20. Duncan J. The locus of interference in the perception of simultaneous stimuli. Psychological Review 87 (1980) 272-300
21. Ernst M.O., and Banks M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415 (2002, January 24) 429-433
22. Falissard B. The psy package (Version 0.7) (2008, February 16). http://cran.r-project.org/web/packages/psy/psy.pdf Retrieved March 23, 2008, from http://cran.r-project.org/web/packages/psy/psy.pdf
23. Fendrich R., and Corballis P.M. The temporal cross-capture of audition and vision. Perception and Psychophysics 63 (2001) 719-725
24. Fisher G.H. Phenomenal causality in conditions of intrasensory and intersensory stimulation. American Journal of Psychology 75 (1962) 321-323
25. Fletcher H., and Munson W.A. Loudness, its definition, measurement and calculation. Journal of the Acoustical Society of America 5 (1933) 82-108
26. Gepshtein S., Burge J., Ernst M.O., and Banks M.S. The combination of vision and touch depends on spatial proximity. Journal of Vision 5 (2005) 1013-1023
27. Green K.P., Kuhl P.K., Meltzoff A.N., and Stevens E.B. Integrating speech information across talkers, gender, and sensory modality: Female faces and male voices in the McGurk effect. Perception and Psychophysics 50 (1991) 524-536
28. Guest S., and Spence C. Tactile dominance in speeded discrimination of textures. Experimental Brain Research 150 (2003) 207-208
29. Guski R., and Troje N.F. Audiovisual phenomenal causality. Perception and Psychophysics 65 (2003) 789-800
30. Helbig H.B., and Ernst M.O. Knowledge about a common source can promote visual-haptic integration. Perception 36 (2007) 1523-1533
31. Ihaka R., and Gentleman R. R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics 5 (1996) 299-314
32. Jack C.E., and Thurlow W.R. Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. Perceptual and Motor Skills 37 (1973) 967-979
33. Jackson C. Visual factors in auditory localization. Quarterly Journal of Experimental Psychology 5 (1953) 52-65
34. Körding K.P., Beierholm U., Ma W.J., Quartz S., Tenenbaum J.B., and Shams L. Causal inference in multisensory perception. PLoS ONE 2 (2007) 1-10
35. Keetels M., Stekelenburg J., and Vroomen J. Auditory grouping occurs prior to intersensory pairing: Evidence from temporal ventriloquism. Experimental Brain Research 180 (2007) 449-456
36. King A.J. Multisensory integration: Strategies for synchronization. Current Biology 15 (2005) R339-R341
37. Kopinska A., and Harris L.R. Simultaneity constancy. Perception 33 (2004) 1049-1060
38. Kreft I., and de Leeuw J. Introducing multilevel modeling (1998), Sage, London
39. Lewald J., and Guski R. Auditory-visual temporal integration as a function of distance: No compensation for sound-transmission time in human perception. Neuroscience Letters 357 (2004) 119-122
40. Loftus G.R. Analysis, interpretation, and visual presentation of experimental data. In: Wixted J. (Ed). Stevens' handbook of experimental psychology: Vol. 4. Methodology in experimental psychology. 3rd ed. (2002), Wiley, New York 339-390
41. Maxwell S.E., and Delaney H.D. Designing experiments and analyzing data: A model comparison perspective. 2nd ed. (2004), Erlbaum, Mahwah, NJ
42. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976, December 23) 746-748
43. Michotte, A. (1963). The perception of causality (T. R. Miles & E. Miles, Trans.). New York: Basic Books.
44. Miller E.A. Interaction of vision and touch in conflict and nonconflict form perception tasks. Journal of Experiment Psychology 96 (1972) 114-123
45. Munhall K.G., Gribble P., Sacco L., and Ward M. Temporal constraints on the McGurk effect. Perception and Psychophysics 58 (1996) 351-362
46. Newport R., Rabb B., and Jackson S.R. Noninformative vision improves haptic spatial perception. Current Biology 12 (2002) 1661-1664
47. Radeau M., and Bertelson P. Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception and Psychophysics 22 (1977) 137-146
48. Raudenbush S.W., and Bryk A.S. Linear mode: Applications and data analysis methods (2002), Sage, London
49. Recanzone G.H. Auditory influences on visual temporal rate perception. Journal of Neurophysiology 89 (2003) 1078-1093
50. Remez R., Fellowes J., and Pisoni D. Multimodal perceptual organization of speech: Evidence from tone analogs of spoken utterances. Speech Communication 26 (1998) 65-73
51. Roach N.W., Heron J., and McGraw P.V. Resolving multisensory conflict: A strategy for balancing the costs and benefits of audiovisual integration. Proceedings of the Royal Society B: Biological Sciences 273 (2006) 2159-2168
52. Rosenblum L.D., and Fowler C.A. Audiovisual investigation of the loudness-effort effect for speech and nonspeech events. Journal of Experimental Psychology: Human Perception and Performance 17 (1991) 976-985
53. Saldaña H.M., and Rosenblum L.D. Visual influences on auditory pluck and bow judgments. Perception & Psychophysics 54 (1993) 406-416
54. Sams M., Manninen P., and Surakka V. McGurk effect in Finnish syllables, isolated words and words in sentences: Effects of word meaning and sentence context. Speech Communication 26 (1998) 75-87
55. Saygin A.P., Driver J., and de Sa V.R. In the footsteps of biological motion and multisensory perception. Psychological Science 19 (2008) 469-475
56. Schutz M., and Lipscomb S. Hearing gestures, seeing music: Vision influences perceived tone duration. Perception 36 (2007) 888-897
57. Schwarz N., and Clore G.L. Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology 45 (1983) 513-523
58. Sekuler R., Sekuler A.B., and Lau R. Sound alters visual motion perception. Nature 385 (1997, January 23) 308
59. Shams L., Kamitani Y., and Shimojo S. Visual illusion induced by sound. Cognitive Brain Research 14 (2002) 147-152
60. Shimojo S., Scheier C., Nijhawan R., Shams L., Kamitani Y., and Watanabe K. Beyond perceptual modality: Auditory effects on visual perception. Acoustical Science and Technology 22 (2001) 61-67
61. Shipley T. Auditory flutter-driving of visual flicker. Science 145 (1964, September 18) 1328-1330
62. Shrout P.E., and Fleiss J.L. Intraclass correlation: Uses in assessing rater reliability. Psychological Bulletin 86 (1979) 420-428
63. Snijders T., and Bosker R. Multilevel analysis: An introduction to basic and advanced multilevel modeling (1999), Sage, London
64. Soto-Faraco S., and Alsius A. Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance 35 (2009) 580-587
65. Spence C. Audiovisual multisensory integration. Acoustical Science and Technology 28 (2007) 61-71
66. Spence C., Nicholls M.E.R., and Driver J. The cost of expecting events in the wrong sensory modality. Perception and Psychophysics 63 (2001) 330-336
67. Spence C., Sanabria D., and Soto-Faraco S. Intersensory Gestalten and crossmodal scene perception. In: Noguchi K. (Ed). Psychology of beauty and Kansei: New horizons of Gestalt perception (2007), Nihon University College of Humanities and Sciences, Tokyo 519-579
68. Stekelenburg J.J., and Vroomen J. Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience 19 (2007) 1964-1973
69. Stone J.V., Hunkin N.M., Porrill J., Wood R., Keeler V., Beanland M., et al. When is now? Perception of simultaneity. Proceedings of the Royal Society B: Biological Sciences 268 (2001) 31-38
70. Sugita Y., and Suzuki Y. Implicit estimation of sound-arrival time. Nature 423 (2003, June 26) 911
71. Thomas G. Experimental study of the influence of vision on sound localization. Journal of Experimental Psychology 28 (1941) 163-175
72. Thompson W.F., Russo F., and Quinto L. Audiovisual integration of emotional cues in song. Cognition and Emotion 22 (2008) 1457-1470
73. Vatakis A., and Spence C. Crossmodal binding: Evaluating the "unity assumption" using audiovisual speech stimuli. Perception and Psychophysics 69 (2007) 744-756
74. Vatakis A., and Spence C. Evaluating the influence of the "unity assumption" on the temporal perception of realistic audiovisual stimuli. Acta Psychologica 127 (2008) 12-23
75. Vines B.W., Krumhansl C.L., Wanderley M.M., and Levitin D.J. Cross-modal interactions in the perception of musical performance. Cognition 101 (2006) 80-113
76. Vroomen J. Ventriloquism and the nature of the unity decision. Commentary on Welch. In: Aschersleben G., Bachmann T., and Müsseler J. (Eds). Cognitive contributions to the perception of spatial and temporal events (1999), Elsevier, Amsterdam 389-393
77. Wada Y., Kitagawa N., and Noguchi K. Audiovisual integration in temporal perception. Psychophysiology 50 (2003) 117-124
78. Walker J.T., and Scott K.J. Auditory-visual conflicts in the perceived duration of lights, tones, and gaps. Journal of Experimental Psychology: Human Perception and Performance 7 (1981) 1327-1339
79. Walker S., Bruce V., and O'Malley C. Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect. Perception and Psychophysics 57 (1995) 1124-1133
80. Watanabe K., and Shimojo S. When sound affects vision: Effects of auditory grouping on visual motion perception. Psychological Science 12 (2001) 109-116
81. Welch R.B. The effect of experienced limb identity upon adaptation to simulated displacement of the visual field. Perception and Psychophysics 12 (1972) 453-456
82. Welch R.B. Meaning, attention, and the "unity assumption" in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G., Bachmann T., and Müsseler J. (Eds). Cognitive contributions to the perception of spatial and temporal events (1999), Elsevier, Amsterdam 371-387
83. Welch R.B., DuttonHurt L.D., and Warren D.H. Contributions of audition and vision to temporal rate perception. Perception and Psychophysics 39 (1986) 294-300
84. Welch R.B., and Warren D.H. Immediate perceptual response to intersensory discrepancy. Psychological Bulletin 88 (1980) 638-667
85. Welch R.B., and Warren D.H. Intersensory interactions. In: Boff K., Kaufman L., and Thomas J. (Eds). Handbook of perception and human performance 1 (1986), Wiley-Interscience, New York chapter 25
86. Wilkinson L. Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 54 (1999) 594-604
87. Witkin H.A., Wapner S., and Leventhal T. Sound localization with conflicting visual and auditory cues. Journal of Experimental Psychology 43 (1952) 58-67
88. Zampini M., Guest S., Shore D.I., and Spence C. Audiovisual simultaneity judgments. Perception and Psychophysics 67 (2005) 531-544