Audiovisual correspondence between musical timbre and visual shapes

Frontiers in Human Neuroscience

Volumn 8, Issue MAY, 2014, Pages

  Adeli, Mohammad ^a   Rouat, Jean ^a,b   Molotchnikoff, Stéphane ^a,b  

^a UNIVERSITÉ DE SHERBROOKE   (Canada)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Audiovisual correspondences; Color; Pitch; Sensory substitution; Shapes; Synesthesia; Timbre

Indexed keywords

ADULT; AGED; ARTICLE; AUDITORY STIMULATION; COLOR; CONTROLLED STUDY; FEMALE; HUMAN; HUMAN EXPERIMENT; LOUDNESS; MALE; MUSIC; PERCEPTION; PHYSICAL PARAMETERS; PITCH; SHAPE; SOUND; SYNESTHESIA; TIMBRE; VISUAL STIMULATION;

EID: 84901834486     PISSN: None     EISSN: 16625161     Source Type: Journal    
DOI: 10.3389/fnhum.2014.00352     Document Type: Article

References (47)

1. Abbado, A. (1988). Perceptual correspondences of abstract animation and synthetic sound. Leonardo. Suppl. Iss. 1, 3-5. doi: 10.2307/1557901
2. Ben-Artzi, E., and Marks, L. E. (1995). Visual-auditory interaction in speeded classification: role of stimulus difference. Atten. Percept. Psychophys. 57, 1151-1162. doi: 10.3758/BF03208371
3. Bernstein, I. H., and Edelstein, B. A. (1971). Effects of some variations in auditory input upon visual choice reaction time.J. Exp. Psychol. 87, 241-247. doi: 10.1037/h0030524
4. Berthaut, F., Desainte-Catherine, M., and Hachet, M. (2010). "Combining audiovisual mappings for 3D musical interaction, " inInternational Computer Music Conference (ICMC2010)(New York, NY: ICMA), 357-364.
5. Berthaut, F., Desainte-Catherine, M., and Hachet, M. (2011). Interacting with 3D reactive widgets for musical performance.J. New Music Res. 40, 253-263. doi: 10.1080/09298215.2011.602693
6. Boersma, P., and van Heuven, V. (2001). Praat, a system for doing phonetics by computer.Glot Int. 5, 341-345.
7. Caivano, J. L. (1994). Color and sound: physical and psychophysical relations. Color Res. Appli. 19, 126-133.
8. Chiou, R., and Rich, A. N. (2012). Cross-modality correspondence between pitch and spatial location modulates attentional orienting. Perception41, 339-353. doi: 10.1068/p7161
9. Collier, W. G., and Hubbard, T. L. (2001). Judgments of happiness, brightness, speed and tempo change of auditory stimuli varying in pitch and tempo. Psychomusicology17, 36-55. doi: 10.1037/h0094060
10. Cytowic, R. E. (1995). Synesthesia: phenomenology and neuropsychology. Psyche2, 2-10.
11. Cytowic, R. E., and Eagleman, D. (2009). Wednesday Is Indigo Blue: Discovering the Brain of Synesthesia. Cambridge, MA: MIT Press.
12. Eitan, Z., and Timmers, R. (2010). Beethoven's last piano sonata and those who follow crocodiles: cross-domain mappings of auditory pitch in a musical context. Cognition114, 405-422. doi: 10.1016/j.cognition.2009.10.013
13. Erickson, R. (1975). Sound Structure in Music.[Mit Noten U. Illustr.]. Berkeley: University of California Press.
14. Ester, M., Kriegel, H.-P., Sander, J., and Xu, X. (1996). "A density-based algorithm for discovering clusters in large spatial databases with noise, " inProceedings of 2nd International Conference on Knowledge Discovery and Data Mining (KDD-96), Vol. 96 (Portland, OR), 226-231.
15. Evans, K. K., and Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features.J. Vis. 10, 6.1-6.12.
16. Fernay, L., Reby, D., and Ward, J. (2012). Visualized voices: a case study of audio-visual synesthesia. Neurocase18, 50-56. doi: 10.1080/13554794.2010.547863
17. Franssen, M. (1991). The ocular harpsichord of louis-bertrand castel. Tractrix3, 15-77.
18. Gallace, A., and Spence, C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size. Attent. Percept. Psychophys. 68, 1191-1203. doi: 10.3758/BF03193720
19. Giannakis, K. (2006a). A comparative evaluation of auditory-visual mappings for sound visualisation. Organ. Sound11, 297-307. doi: 10.1017/S1355771806001531
20. Giannakis, K. (2006b). "Perceptual-based visualization of auditory information using visual texture, " inDigital Multimedia Perception and Design, eds G. Ghinea and S. Y. Chen (London: IGI Global), 152-169.
21. Giannakis, K., and Smith, M. (2001). "Imaging soundscapes: identifying cognitive associations between audatory and visual dimensions, " inMusical Imagery, eds R. I. Godöy and H. Jörgensen (Lisse, Holland: Swets and Zeitlinger), 161-179.
22. Goto, M. (2004). "Development of the RWC music database, " inProceedings of the 18th International Congress on Acoustics (ICA 2004), Vol. 1 (Kyoto), 553-556.
23. Goto, M., Hashiguchi, H., Nishimura, T., and Oka, R. (2003). "RWC music database: music genre database and musical instrument sound database, " inProceedings of the 4th International Conference on Music Information Retrieval (ISMIR), Vol. 3 (Baltimore, MD), 229-230.
24. Grill, T., and Flexer, A. (2012). "Visualization of perceptual qualities in textural sounds, " inProceedings of the International Computer Music Conference (ICMC 2012)(Ljubljana, Slovenia: The International Computer Music Association), 589-596.
25. Guzman-Martinez, E., Ortega, L., Grabowecky, M., Mossbridge, J., and Suzuki, S. (2012). Interactive coding of visual spatial frequency and auditory amplitude-modulation rate. Curr. Biol. 22, 383-388. doi: 10.1016/j.cub.2012.01.004
26. Heer, J. (2000). A Review of Synesthesia. Technical Report, University of California Berkeley.
27. Hubbard, E. (2007). Neurophysiology of synesthesia. Curr. Psychiatry Rep. 9, 193-199. doi: 10.1007/s11920-007-0018-6
28. Hubbard, T. L. (1996). Synesthesia-like mappings of lightness, pitch, and melodic interval. Am. J. Psychol. 109, 219-238. doi: 10.2307/1423274
29. Köhler, W. (1947). Gestalt Psychology: An Introduction to New Concepts in Modern Psychology. New York, NY: Liveright Pub. Corp.
30. Marks, L. E. (1974). On associations of light and sound: the mediation of brightness, pitch, and loudness. Am. J. Psychol. 87, 173-188. doi: 10.2307/1422011
31. Marks, L. E. (1987). On cross-modal similarity: auditory-visual interactions in speeded discrimination.J. Exp. Psychol. Hum. Percept. Perform. 13, 384-394. doi: 10.1037/0096-1523.13.3.384
32. Marks, L. E. (2004). "Cross-modal interactions in speeded classification, " inThe Handbook of Multisensory Processes, eds G. A. Calvert, C. Spence, and B. E. Stein (Cambridge, MA: MIT Press), 85-105.
33. Maurer, D., Pathman, T., and Mondloch, C. J. (2006). The shape of boubas: sound-shape correspondences in toddlers and adults. Dev. Sci. 9, 316-322. doi: 10.1111/j.1467-7687.2006.00495.x
34. Melara, R. D. (1989). Dimensional interaction between color and pitch.J. Exp. Psychol. Hum. Percept. Perform. 15, 69-79. doi: 10.1037/0096-1523.15.1.69
35. Melara, R. D., and O'Brien, T. P. (1987). Interaction between synesthetically corresponding dimensions.J. Exp. Psychol. Gen. 116, 323-336. doi: 10.1037/0096-3445.116.4.323
36. Mondloch, C. J., and Maurer, D. (2004). Do small white balls squeak? pitch-object correspondences in young children. Cogn. Affect. Behav. Neurosci. 4, 133-136. doi: 10.3758/CABN.4.2.133
37. Orlandatou, K. (2009). Synesthesia and Art. Master's thesis, Academy of Music and Theater of Hamburg.
38. Parise, C., and Spence, C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect. Neurosci. Lett. 442, 257-261. doi: 10.1016/j.neulet.2008.07.010
39. Parise, C. V., and Spence, C. (2009). When birds of a feather flock: together synesthetic correspondences modulate audiovisual integration in non-synesthetes. PLoS ONE4:e5664. doi: 10.1371/journal.pone.0005664
40. Parise, C. V., and Spence, C. (2012). Audiovisual crossmodal correspondences and sound symbolism: a study using the implicit association test. Exp. Brain Res. 220, 319-333. doi: 10.1007/s00221-012-3140-6
41. Patching, G. R., and Quinlan, P. T. (2002). Garner and congruence effects in the speeded classification of bimodal signals.J. Exp. Psychol. Hum. Percept. Perform. 28, 755-775. doi: 10.1037/0096-1523.28.4.755
42. Peacock, K. (1988). Instruments to perform color-music: two centuries of technological experimentation. Leonardo21, 397-406. doi: 10.2307/1578702
43. Ramachandran, V. S., and Hubbard, E. M. (2001). Synaesthesia-a window into perception, thought and language.J. Conscious. Stud. 8, 3-34.
44. Rusconi, E., Kwan, B., Giordano, B., Umilta, C., and Butterworth, B. (2006). Spatial representation of pitch height: the smarc effect. Cognition99, 113-129. doi: 10.1016/j.cognition.2005.01.004
45. Sweeny, T. D., Guzman-Martinez, E., Ortega, L., Grabowecky, M., and Suzuki, S. (2012). Sounds exaggerate visual shape. Cognition124, 194-200. doi: 10.1016/j.cognition.2012.04.009
46. Walker, P., and Smith, S. (1985). Stroop interference based on the multimodal correlates of haptic size and auditory pitch. Perception14, 729-736. doi: 10.1068/p140729
47. Walsh, V., and Kulikowski, J. (1998). Perceptual Constancy: Why Things Look as They Do. Cambridge, MA: Cambridge University Press.