Crossmodal correspondences: Standing issues and experimental guidelines

Multisensory Research

Volumn 29, Issue 1-3, 2016, Pages 7-28

  Parise, Cesare V ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

Crossmodal correspondences; Multisensory processing; Natural scene statistics; Perceptual learning; Psychophysics; Sensory cue integration

Indexed keywords

DESIGN OF EXPERIMENTS;

CROSS-MODAL; CUE INTEGRATION; MULTISENSORY; NATURAL SCENE STATISTICS; PERCEPTUAL LEARNING; PSYCHOPHYSICS;

SENSORY PERCEPTION;

ASSOCIATION; ATTENTION; HUMAN; PERCEPTION; PSYCHOPHYSICS;

ASSOCIATION; ATTENTION; CUES; HUMANS; PERCEPTION; PSYCHOPHYSICS;

EID: 84957576562     PISSN: 22134794     EISSN: 22134808     Source Type: Journal    
DOI: 10.1163/22134808-00002502     Document Type: Review

References (73)

1. Backus, B. T. (2011). Recruitment of new visual cues for perceptual appearance, in: Sensory Cue Integration, J. Trommershäuser, K. Körding and M. Landy (Eds), pp. 101-119. Oxford University Press, New York, NY, USA.
2. Baier, B., Kleinschmidt, A. and Müller, N. G. (2006). Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information, J. Neurosci. 26, 12260-12265.
3. Bernstein, I. H. and Edelstein, B. A. (1971). Effects of some variations in auditory input upon visual choice reaction time, J. Exp. Psychol. 87, 241-247.
4. Bien, N., Ten Oever, S., Goebel, R. and Sack, A. T. (2012). The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study, Neuroimage 59, 663-672.
5. Bien, N., Ten Oever, S., Goebel, R. and Sack, A. T. (2013). Corrigendum to 'The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study', Neuroimage 72, 325.
6. Bremner, A. J., Caparos, S., Davidoff, J., De Fockert, J., Linnell, K. J. and Spence, C. (2013). "Bouba" and "Kiki" in Namibia? A remote culture make similar shape-sound matches, but different shape-taste matches to Westerners, Cognition 126, 165-172.
7. Crisinel, A. S. and Spence, C. (2012). A fruity note: crossmodal associations between odors and musical notes, Chem. Senses 37, 151-158.
8. Deroy, O. and Spence, C. (2013). Why we are not all synesthetes (not even weakly so), Psychonom. Bull. Rev. 20, 1-22.
9. Di Luca, M., Ernst, M. O. and Backus, B. (2010). Learning to use an invisible visual signal for perception, Curr. Biol. 20, 1860-1863.
10. Dolscheid, S., Hunnius, S., Casasanto, D. and Majid, A. (2014). Prelinguistic infants are sensitive to space-pitch associations found across cultures, Psychol. Sci. 25, 1256-1261.
11. Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch, J. Vis. 7, 7. doi: 10.1167/7.5.7.
12. Ernst, M. O. and Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion, Nature 415, 429-433.
13. Evans, K. K. and Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features, J. Vis. 10, 6. doi: 10.1167/10.1.6.
14. Fechner, G. T. (1860). Elemente der Psychophysik (Elements of Psychophysics). Breitkopf and Hartel, Leipzig, Germany.
15. Flanagan, J. R. and Beltzner, M. A. (2000). Independence of perceptual and sensorimotor predictions in the size-weight illusion, Nat. Neurosci. 3, 737-741.
16. Flanagan, J. R., Bittner, J. P. and Johansson, R. S. (2008). Experience can change distinct size-weight priors engaged in lifting objects and judging their weights, Curr. Biol. 18, 1742-1747.
17. Gallace, A. and Spence, C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size, Percept. Psychophys. 68, 1191-1203.
18. Gates, G. A. and Mills, J. H. (2005). Presbycusis, Lancet 366, 1111-1120.
19. Gescheider, G. A. (2013). Psychophysics: the Fundamentals. Lawrence Erlbaum, Mahwah, NJ, USA.
20. Green, D. and Swets, J. (1966). Signal Detection Theory and Psychophysics. Wiley, New York, NY, USA.
21. Guzman-Martinez, E., Ortega, L., Grabowecky, M., Mossbridge, J. and Suzuki, S. (2012). Interactive coding of visual spatial frequency and auditory amplitude-modulation rate, Curr. Biol. 22, 383-388.
22. Haijiang, Q., and Saunders, J. A., Stone, R. W. and Backus, B. T. (2006). Demonstration of cue recruitment: change in visual appearance by means of Pavlovian conditioning, Proc. Natl Acad. Sci. USA 103, 483-488.
23. Helmholtz, H. V. (1909). Handbuch der Physiologischen Optik (Handbook of Physiological Optics). Voss, Hamburg, Germany.
24. Hillis, J., and Ernst, M. O., Banks, M. and Landy, M. (2002). Combining sensory information: mandatory fusion within, but not between, senses, Science 298, 1627-1630.
25. Holway, A. H. and Boring, E. G. (1941). Determinants of apparent visual size with distance variant, Am. J. Psychol. 54, 21-37.
26. Kaliuzhna, M., Prsa, M., Gale, S., Lee, S. J. and Blanke, O. (2015). Learning to integrate contradictory multisensory self-motion cue pairings, J. Vis. 15, 10. doi: 10.1167/15.1.10.
27. Keetels, M. and Vroomen, J. (2011). No effect of synesthetic congruency on temporal ventriloquism, Atten. Percept. Psychophys. 73, 1-10.
28. Knill, D. C. and Richards, W. (1996). Perception as Bayesian Inference. Cambridge University Press, Cambridge, UK.
29. Knöferle, K. M., Woods, A., Käppler, F. and Spence, C. (2015). That sounds sweet: using crossmodal correspondences to communicate gustatory attributes, Psychol. Marketing 32, 107-120.
30. Köhler, W. (1929). Gestalt Psychology. Liveright, New York, NY, USA.
31. Köhler, W. (1947). Gestalt Psychology: an Introduction to New Concepts in Modern Psychology. Liveright, New York, NY, USA.
32. Landy, M. S., and Maloney, L. T., Johnston, E. B. and Young, M. (1995). Measurement and modeling of depth cue combination: in defense of weak fusion, Vis. Res. 35, 389-412.
33. Landy, M. S., Ho, Y.-X., Serwe, S., Trommershäuser, J. and Maloney, L. T. (2011). Cues and pseudocues in texture and shape perception, in: Sensory Cue Integration, J. Trommershäuser, K. Körding and M. Landy (Eds), pp. 263-278. Oxford University Press, New York, NY, USA.
34. Lewkowicz, D. J. and Minar, N. J. (2014). Infants are not sensitive to synesthetic cross-modality correspondences: a comment on Walker et al. (2010), Psychol. Sci. 25, 832-834.
35. Marks, L. E. (1989). On cross-modal similarity: the perceptual structure of pitch, loudness, and brightness, J. Exp. Psychol. Hum. Percept. Perform. 15, 586-602.
36. Marks, L. E. (2004). Cross-modal interactions in speeded classification, in: The Handbook of Multisensory Processes, G. A. Calvert, C. Spence and B. E. Stein (Eds), pp. 85-106. MIT Press, Cambridge, MA, USA.
37. Nielsen, A. and Rendall, D. (2011). The sound of round: evaluating the sound-symbolic role of consonants in the classic Takete-Maluma phenomenon, Can. J. Exp. Psychol. 65, 115-124.
38. Nielsen, A. K. and Rendall, D. (2013). Parsing the role of consonants versus vowels in the classic Takete-Maluma phenomenon, Can. J. Exp. Psychol. 67, 153-163.
39. Occelli, V., Esposito, G., Venuti, P., Arduino, G. M. and Zampini, M. (2013). The takete-maluma phenomenon in autism spectrum disorders, Perception 42, 233-241.
40. Orchard-Mills, E., Van der Burg, E. and Alais, D. (2013). Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies, J. Vis. 13, 6. doi: 10.1167/13.3.6.
41. Parise, C. and Pavani, F. (2011). Evidence of sound symbolism in simple vocalizations, Exp. Brain Res. 214, 373-380.
42. Parise, C. and Spence, C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect, Neurosci. Lett. 442, 257-261.
43. Parise, C. and Spence, C. (2009). When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetes, PLoS One 4, e5664. doi: 10.1371/journal.pone.0005664.
44. Parise, C. V. and Spence, C. (2012). Audiovisual crossmodal correspondences and sound symbolism: a study using the implicit association test, Exp. Brain Res. 220, 319-333.
45. Parise, C. and Spence, C. (2013). Audiovisual cross-modal correspondences in the general population, in: Oxford Handbook of Synaesthesia, J. Simner and E. M. Hubbard (Eds), pp. 790-815. Oxford University Press, Oxford, UK.
46. Parise, C. V., Knorre, K. and Ernst, M. O. (2014). Natural auditory scene statistics shapes human spatial hearing, Proc. Natl Acad. Sci. USA 111, 6104-6108.
47. Peters, M. A. K., Balzer, J. and Shams, L. (2015). Smaller = denser, and the brain knows it: natural statistics of object density shape weight expectations, PLoS One 10, e0119794. doi: 10.1371/journal.pone.0119794.
48. Plaisier, M. A. and Smeets, J. B. J. (2012). Mass is all that matters in the size-weight illusion, PLoS One 7, e42518. doi: 10.1371/journal.pone.0042518.
49. Pratt, C. C. (1930). The spatial character of high and low tones, J. Exp. Psychol. 13, 278-285.
50. Rader, C. M. and Tellegen, A. (1987). An investigation of synesthesia, J. Pers. Soc. Psychol. 52, 981-987.
51. Ramachandran, V. and Hubbard, E. (2001). Synaesthesia: a window into perception, thought and language, J. Conscious. Stud. 8, 3-34.
52. Rammsayer, T. H. and Verner, M. (2015). Larger visual stimuli are perceived to last longer from time to time: the internal clock is not affected by nontemporal visual stimulus size, J. Vis. 15, 5. doi: 10.1167/15.3.5.
53. Roffler, S. K. and Butler, R. A. (1968a). Factors that influence the localization of sound in the vertical plane, J. Acoust. Soc. Am. 43, 1255-1259.
54. Roffler, S. K. and Butler, R. A. (1968b). Localization of tonal stimuli in the vertical plane, J. Acoust. Soc. Am. 43, 1260-1266.
55. Rogers, M. E. and Butler, R. A. (1992). The linkage between stimulus frequency and covert peak areas as it relates to monaural localization, Percept. Psychophys. 52, 536-546.
56. Ross, H. E. (1969). When is a weight not illusory? Q. J. Exp. Psychol. 21, 346-355.
57. Rusconi, E., Kwan, B., Giordano, B. L., Umilta, C. and Butterworth, B. (2006). Spatial representation of pitch height: the SMARC effect, Cognition 99, 113-129.
58. Sadaghiani, S., Maier, J. X. and Noppeney, U. (2009). Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing, J. Neurosci. 29, 6490-6499.
59. Sapir, E. (1929). A study in phonetic symbolism, J. Exp. Psychol. 12, 225-239.
60. Spence, C. (2011). Crossmodal correspondences: a tutorial review, Atten. Percept. Psychophys. 73, 1-25.
61. Spence, C. and Deroy, O. (2012). Crossmodal correspondences: innate or learned? i-Perception 3, 316-318.
62. Spence, C. and Parise, C. V. (2012). The cognitive neuroscience of crossmodal correspondences, i-Perception 3, 410-412.
63. Stevens, J. C. and Marks, L. E. (1965). Cross-modality matching of brightness and loudness, Proc. Natl Acad. Sci. USA 54, 407-411.
64. Stumpf, K. (1883). Tonpsychologie (Tone Psychology). Hirzel, Leipzig, Germany.
65. Suzuki, Y. I. and Takeshima, H. (2004). Equal-loudness-level contours for pure tones, J. Acoust. Soc. Am. 116, 918-933.
66. Trommershäuser, J., Körding, K. and Landy, M. (Eds) (2011). Sensory Cue Integration. Oxford University Press, New York, NY, USA.
67. Van Dam, L. C. J. and Ernst, M. O. (2015). Mapping shape to visuomotor mapping: learning and generalisation of sensorimotor behaviour based on contextual information, PLoS Comput. Biol. 11, e1004172. doi: 10.1371/journal.pcbi.1004172.
68. Van Dam, L. C. J., Parise, C. V. and Ernst, M. O. (2014). Modeling multisensory integration, in: Sensory Integration and the Unity of Consciousness, D. Bennett and C. Hill (Eds), pp. 209-229. MIT Press, Cambridge, MA, USA.
69. Walker, R. (1987). The effects of culture, environment, age, and musical training on choices of visual metaphors for sound, Percept. Psychophys. 42, 491-502.
70. Walker, P. and Smith, S. (1984). Stroop interference based on the synaesthetic qualities of auditory pitch, Perception 13, 75-81.
71. Walker, P. and Smith, S. (1985). Stroop interference based on the multimodal correlates of haptic size and auditory pitch, Perception 14, 729-736.
72. Walker, P., Bremner, J., Mason, U., Spring, J., Mattock, K., Slater, A. and Johnson, S. (2010). Preverbal infants' sensitivity to synaesthetic cross-modality correspondences, Psychol. Sci. 21, 21-25.
73. Yates, M. J., Loetscher, T. and Nicholls, M. E. R. (2012). A generalized magnitude system for space, time, and quantity? A cautionary note, J. Vis. 12, 9. doi: 10.1167/12.7.9.