Rising tones and rustling noises: Metaphors in gestural depictions of sounds

PLoS ONE

Volumn 12, Issue 7, 2017, Pages

  Lemaitre, Guillaume ^a   Scurto, Hugo ^a   Françoise, Jules ^a   Bevilacqua, Frédéric ^a   Houix, Olivier ^a   Susini, Patrick ^a  

^a Université Pierre et Marie Curie   (France)

Author keywords

[No Author keywords available]

Indexed keywords

CASE REPORT; FEMALE; FINGER; GESTURE; HUMAN; IMITATION; INFERENTIAL STATISTICS; LITERATURE; LOUDNESS; MALE; NOISE; OBSERVATIONAL STUDY; PITCH; PLANT LEAF; QUANTITATIVE STUDY; SENSATION; VOCALIZATION; WRITING; ADOLESCENT; ADULT; HEARING; MIDDLE AGED; SOUND; SOUND DETECTION; YOUNG ADULT;

ADOLESCENT; ADULT; FEMALE; GESTURES; HUMANS; MALE; METAPHOR; MIDDLE AGED; PITCH PERCEPTION; SOUND; SOUND SPECTROGRAPHY; YOUNG ADULT;

EID: 85026540671     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0181786     Document Type: Article

References (94)

1. Lemaitre G, Susini P, Rocchesso D, Lambourg C, Boussard P. Non-Verbal Imitations as a Sketching Tool for Sound Design. In: Aramaki M, Derrien O, Kronland-Martinet R, Ystad S, editors. Sound, Music, and Motion. Lecture Notes in Computer Sciences. Berlin, Heidelberg, Germany: Springer; 2014. p. 558–574.
2. Wright P. Linguistic description of auditory signals. Journal of applied psychology. 1971; 55(3):244–250. https://doi.org/10.1037/h0031025
3. Faure A. Des sons aux mots: comment parle-t-on du timbre musical? [Unpublished Doctoral Dissertation]. École de Hautes Études en Sciences Sociales. Paris, France; 2000.
4. Porcello T. Speaking of sound: language and the professionalization of sound-recording engineers. Social Studies of Science. 2004; 34(5):733–758. https://doi.org/10.1177/0306312704047328
5. Heyes C. Automatic imitation. Psychological bulletin. 2011; 137(3):463–483. https://doi.org/10.1037/a0022288 PMID: 21280938
6. Meltzoff AN, Prinz W. The imitative mind. Development, evolution and brain bases. Cambridge, UK: Cambridge University Press; 2002. 364 pages.
7. Mercado E III, Mantell JT, Pfordresher PQ. Imitating sounds: A cognitive approach to understanding vocal imitation. Comparative Cognition & Behavior Reviews. 2014; 9.
8. Chartrand TL, Bargh JA. The chameleon effect: the perception-behavior link and social interaction. Journal of Personality and Social Psychology. 1999; 76(6):893–910. https://doi.org/10.1037/00223514.76.6.893 PMID: 10402679
9. Lemaitre G, Rocchesso D. On the effectiveness of vocal imitation and verbal descriptions of sounds. Journal of the Acoustical Society of America. 2014; 135(2):862–873. https://doi.org/10.1121/1.4861245 PMID: 25234894
10. Lemaitre G, Houix O, Voisin F, Misdariis N, Susini P. Vocal Imitations of Non-Vocal Sounds. PloS one. 2016; 11(12):e0168167. https://doi.org/10.1371/journal.pone.0168167 PMID: 27992480
11. Ekman P, Friesen WV. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. semiotica. 1969; 1(1):49–98. https://doi.org/10.1515/semi.1969.1.1.49
12. Kendon A. Gesture: Visible action as utterance. Cambridge University Press; 2004.
13. McNeill D. Gesture and thought. University of Chicago Press; 2005.
14. Goldin-Meadow S. The role of gesture in communication and thinking. Trends in cognitive sciences. 1999; 3(11):419–429. https://doi.org/10.1016/S1364-6613(99)01397-2 PMID: 10529797
15. Kelly SD, Iverson JM, Terranova J, Niego J, Hopkins M, Goldsmith L. Putting language back in the body: speech and gesture on three time frames. Developmental neuropsychology. 2002; 22(1):323–349. https://doi.org/10.1207/S15326942dn2201_1 PMID: 12405508
16. Fay N, Arbib M, Garrod S. How to bootstrap a human communication system. Cognitive science. 2013; 37(7):1356–1367. https://doi.org/10.1111/cogs.12048 PMID: 23763661
17. Valenzeno L, Alibali MW, Klatzky R. Teachers’ gestures facilitate students’ learning: A lesson in symmetry. Contemporary Educational Psychology. 2003; 28(2):187–204. https://doi.org/10.1016/S0361-476X(02)00007-3
18. Krauss RM, Dushay RA, Chen Y, Rauscher F. The communicative value of conversational hand gesture. Journal of Experimental Social Psychology. 1995; 31(6):533–552. https://doi.org/10.1006/jesp.1995.1024
19. Krauss RM. Why do we gesture when we speak? Current Directions in Psychological Science. 1998; 7 (2):54–54. https://doi.org/10.1111/1467-8721.ep13175642
20. Goldin-Meadow S, Singer MA. From children’s hands to adults’ ears: gesture’s role in the learning process. Developmental psychology. 2003; 39(3):509. https://doi.org/10.1037/0012-1649.39.3.509 PMID: 12760519
21. Clark HH. Depicting as a method of communication. Psychological review. 2016; 123(3):324–347. https://doi.org/10.1037/rev0000026 PMID: 26855255
22. Lemaitre G, Jabbari A, Misdariis N, Houix O, Susini P. Vocal imitations of basic auditory features. The Journal of the Acoustical Society of America. 2016; 139(1):290–300. https://doi.org/10.1121/1.4939738 PMID: 26827025
23. Mehrabi A, Dixon S, Sandler MB. Vocal imitation of synthesised sounds varying in pitch, loudness and spectral centroid. The Journal of the Acoustical Society of America. 2017; 141(2):783–796. https://doi.org/10.1121/1.4974825 PMID: 28253682
24. Schaeffer P. Traité des objets musicaux. Paris, France: Seuil; 1966.
25. Gaver WW. How do we hear in the world? Explorations in ecological acoustics. Ecological Psychology. 1993; 5(4):285–313. https://doi.org/10.1207/s15326969eco0504_2
26. Lemaitre G, Houix O, Misdariis N, Susini P. Listener expertise and sound identification influence the categorization of environmental sounds. Journal of Experimental Psychology: Applied. 2010; 16(1):16–32. PMID: 20350041
27. Caramiaux B, Françoise J, Schnell N, Bevilacqua F. Mapping through listening. Computer Music Journal. 2014; 38(3):34–48. https://doi.org/10.1162/COMJ_a_00255
28. Vanderveer NJ. Ecological acoustics: human perception of environmental sounds [Unpublished doctoral dissertation]. Cornell University; 1979.
29. Handel S. 8: Identification of Speakers, Instruments, and Environmental Events. In: Listening: an introduction to the perception of auditory events. Cambridge, MA: The MIT Press; 1989. p. 219–274.
30. Giordano BL, McAdams S. Material identification of real impact sounds: Effect of size variation in steel, glass, wood and plexiglass plates. Journal of the Acoustical Society of America. 2006 February; 119 (2):1171–1881. https://doi.org/10.1121/1.2149839 PMID: 16521778
31. Grassi M, Pastore M, Lemaitre G. Looking at the world with your ears: how do we get the size of an object from its sound? Acta Psychologica. 2013; 143:96–104. https://doi.org/10.1016/j.actpsy.2013.02.005 PMID: 23542810
32. Thoret E, Aramaki M, Kronland-Martinet R, Velay JL, Ystad S. From sound to shape: auditory perception of drawing. Journal of Experimental Psychology: Human Perception and Performance. 2014;. PMID: 24446717
33. Lemaitre G, Heller LM. Auditory perception of material is fragile, while action is strikingly robust. Journal of the Acoustical Society of America. 2012; 131(2):1337–1348. https://doi.org/10.1121/1.3675946 PMID: 22352507
34. Lemaitre G, Pyles JR, Halpern AR, Navolio N, Lehet M, Heller LM. Who’s that knocking at my door? Neural bases of sound source identification. Cerebral cortex. In press;.
35. Lemaitre G, Heller LM, Navolio N, Zúñiga-Peñaranda N. Priming Gestures with Sounds. PloS one. 2015; 10(11):e0141791. https://doi.org/10.1371/journal.pone.0141791 PMID: 26544884
36. Lakoff G, Johnson M. Metaphors we live by. Chicago, IL: University of Chicago press; 1980. Edition 2003.
37. Spence C. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics. 2011; 73(4):971–995. https://doi.org/10.3758/s13414-010-0073-7
38. Eitan Z, Granot RY. How music moves: musical parameters and listeners’ images of motion. Music Perception: An Interdisciplinary Journal. 2006; 23(3):221–248. https://doi.org/10.1525/mp.2006.23.3.221
39. Chang S, Cho YS. Polarity correspondence effect between loudness and lateralized response set. Frontiers in psychology. 2015; 6. https://doi.org/10.3389/fpsyg.2015.00683
40. Walker R. The effects of culture, environment, age, and musical training on choices of visual metaphors for sound. Attention, Perception, & Psychophysics. 1987; 42(5):491–502. https://doi.org/10.3758/BF03209757
41. Melara RD, Marks LE. Processes underlying dimensional interactions: Correspondences between linguistic and nonlinguistic dimensions. Memory & Cognition. 1990; 18(5):477–495. https://doi.org/10.3758/BF03198481
42. Rusconi E, Kwan B, Giordano BL, Umilta C, Butterworth B. Spatial representation of pitch height: the SMARC effect. Cognition. 2006; 99(2):113–129. https://doi.org/10.1016/j.cognition.2005.01.004 PMID: 15925355
43. th International Conference on Auditory Display, Montreal, Canada. Georgia Institute of Technology; 2007. p. 319–325.
44. Lidji P, Kolinsky R, Lochy A, Morais J. Spatial associations for musical stimuli: a piano in the head? Journal of Experimental Psychology: human perception and performance. 2007; 33:1189–1207. PMID: 17924817
45. Lu X, Sun Y, Thompson WF. An investigation of spatial representation of pitch in individuals with congenital amusia. The Quarterly Journal of Experimental Psychology. 2016;p. 1–11.
46. Pitteri M, Marchetti M, Priftis K, Grassi M. Naturally together: pitch-height and brightness as coupled factors for eliciting the SMARC effect in non-musicians. Psychological research. 2017; 81(1):243–254. https://doi.org/10.1007/s00426-015-0713-6 PMID: 26424464
47. Parise CV, Knorre K, Ernst MO. Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences. 2014; 111(16):6104–6108. https://doi.org/10.1073/pnas.1322705111
48. Dolscheid S, Shayan S, Majid A, Casasanto D. The thickness of musical pitch: Psychophysical evidence for linguistic relativity. Psychological Science. 2013; 24(5):613–621. https://doi.org/10.1177/0956797612457374 PMID: 23538914
49. Walker P, Bremner JG, Mason U, Spring J, Mattock K, Slater A, et al. Preverbal Infants’ Sensitivity to Synaesthetic Cross-Modality Correspondences. Psychological Science. 2010; 21(1):21–25. https://doi.org/10.1177/0956797609354734 PMID: 20424017
50. Lewkowicz DJ, Minar NJ. Infants Are Not Sensitive to Synesthetic Cross-Modality Correspondences A Comment on Walker et al. (2010). Psychological science. 2014; 25(3):832–834. https://doi.org/10.1177/0956797613516011 PMID: 24463555
51. Nava E, Grassi M, Turati C. Audio-visual, visuo-tactile and audio-tactile correspondences in preschoolers. Multisensory Research. 2016; 29(1-3):93–111. https://doi.org/10.1163/22134808-00002493 PMID: 27311292
52. Caramiaux B, Bevilacqua F, Bianco T, Schnell N, Houix O, Susini P. The role of sound perception in gestural sound description. ACM Transactions on Applied Perception. 2014; 11(1). https://doi.org/10.1145/2536811
53. Godøy RI, Haga E, Jensenius AR. Playing “air instruments”: mimicry of sound-producing gestures by novices and experts. In: International Gesture Workshop. Springer; 2005. p. 256–267.
54. Leman M. Embodied Music Cognition and mediation technology. Cambridge, Massachusetts London, England: The MIT Press; 2008.
55. Godøy RI, Jensenius AR, Nymoen K. Chunking in Music by Coarticulation. Acta Acustica united with Acustica. 2010; 96(4).
56. Maes PJ, Leman M, Palmer C, Wanderley MM. Action-based effects on music perception. Frontiers in Psychology. 2014; 4. Article 1008. https://doi.org/10.3389/fpsyg.2013.01008 PMID: 24454299
57. nd ConGAS International Symposium on Gesture Interfaces for Multimedia Systems, Leeds, UK; 2006. p. 7 pages.
58. Leman M, Desmet F, Styns F, Noorden LV, Moelants D. Sharing Musical Expression Through Embodied Listening: A Case Study Based on Chinese Guqin Music. Music Perception. 2009; 26(3):263–278. https://doi.org/10.1525/mp.2009.26.3.263
59. Nymoen K, Caramiaux B, Kozak M, Torresen J. Analyzing sound tracings: a multimodal approach to music information retrieval. In: Proceedings of the 1st international ACM workshop on Music information retrieval with user-centered and multimodal strategies. ACM; 2011. p. 39–44.
60. Nymoen K, Godøy R, Jensenius AR, Torresen J. Analyzing Correspondence Between Sound Objects and Body Motion. ACM Transaction on Applied Perception. 2013; 10(2):9:1–9:22.
61. Caramiaux B, Bevilacqua F, Schnell N. Towards a gesture-sound cross-modal analysis. In: International Gesture Workshop. Springer; 2009. p. 158–170.
62. Caramiaux B, Bevilacqua F, Schnell N. Analyzing gesture and sound similarities with a HMM-based divergence measure. In: Proceedings of the Sound and Music Conference (SMC 2010). Barcelona, Spain; 2010.
63. Houix O, Lemaitre G, Voisin F, Bevilacqua F, Susini P. A database of vocal and gestural imitations of sounds. Manuscript submitted for publication. 2016;.
64. Houix O, Lemaitre G, Misdariis N, Susini P, Urdapilleta I. A lexical analysis of environmental sound categories. Journal of Experimental Psychology: Applied. 2012; 18(1):52–80. PMID: 22122114
65. Lemaitre G, Heller LM. Evidence for a basic level in a taxonomy of everyday action sounds. Experimental Brain Research. 2013; 226(2):253–264. https://doi.org/10.1007/s00221-013-3430-7 PMID: 23411674
66. Zwicker E, Fastl H. Psychoacoustics Facts and Models. Berlin, Heidelberg, Gemany: Springer Verlag; 1990. 463 pages.
67. Hansen H, Verhey JL, Weber R. The magnitude of tonal content. A review. Acta Acustica united with Acustica. 2011; 97(3):355–363. https://doi.org/10.3813/AAA.918416
68. Peeters G, Deruty E. Sound indexing using morphological description. IEEE Transactions on audio, speech, and language processing. 2010; 18(3):675—687. https://doi.org/10.1109/TASL.2009.2038809
69. Rasamimanana N, Bevilacqua F, Schnell N, Guédy F, Maestracci EC, Zamborlin B, et al. Modular Musical Objects Towards Embodied Control Of Digital Music. In: Proceedings of Tangible Embedded and Embodied Interaction Conference (TEI), Funchal, Portugal. New York City, NY: ACM; 2011.
70. Augoyard JF. L’entretien sur écoute réactivée. In: Grosjean M, Thibaud JP, editors. L’espace urbain en méthodes. Marseille, France: Parenthèses; 2001. p. 127–152.
71. Fleiss JL. Measuring nominal scale agreement among many raters. Psychological bulletin. 1971; 76 (5):378–382. https://doi.org/10.1037/h0031619
72. Sundberg J. The science of the singing voice. DeKalb, IL: Northern Illinois University Press; 1989.
73. Ladefoged P. Vowels and consonants: an introduction to the sounds of language. Oxford, UK: Blackwell; 2001. 215 pages.
74. de Cheveigné A, Kawahara H. YIN, a fundamental frequency estimator for speech and music. Journal of the Acoustical Society of America. 2002; 111(4):1917–1930. https://doi.org/10.1121/1.1458024 PMID: 12002874
75. Peeters G, Giordano BL, Susini P, Misdariis N, McAdams S. The timbre toolbox: Extracting audio descriptors from musical signals. Journal of the Acoustical Society of America. 2011; 130(5):2902. https://doi.org/10.1121/1.3642604 PMID: 22087919
76. Boersma P, Weenink D. Praat: doing phonetics by computer (Version 5.1.05); 2009. Computer program. Retrieved May 1, 2009, from http://www.praat.org/.
77. Mallat S. A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way. 3rd ed. Academic Press; 2008.
78. Torrence C, Compo GP. A practical guide to wavelet analysis. Bulletin of the American Meteorological society. 1998; 79(1):61–78. https://doi.org/10.1175/1520-0477(1998)079%3C0061:APGTWA%3E2.0.CO;2
79. Addison PS. Wavelet transforms and the ECG: a review. Physiological Measurement. 2005; 26(5): R155. https://doi.org/10.1088/0967-3334/26/5/R01 PMID: 16088052
80. Olejnik S, Algina J. Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychological methods. 2003; 8(4):434. https://doi.org/10.1037/1082989X.8.4.434 PMID: 14664681
81. Johanssen G, Nakra TM. Conductors’s gestures and their mapping to sound synthesis. In: Godøy RI, Leman M, editors. Musical gestures. Sound, Movement, and meaning. Routledge, New York City, NY; 2010. p. 264–298.
82. Shepard RN. Circularity in judgments of relative pitch. Journal of the Acoustical Society of America. 1964; 36(12):2346–2353. https://doi.org/10.1121/1.1919362
83. Köhler W. Gestalt psychology. New York, NY: Liveright Publishing Corporation; 1945.
84. Ramachandran VS, Hubbard EM. Synaesthesia–a window into perception, thought and language. Journal of consciousness studies. 2001; 8(12):3–34.
85. Schouwstra M, Smith K, Kirby S. From natural order to convention in silent gesture. In: The Evolution of Language: Proceedings of the 11th International Conference (EVOLANG XI). EVOLANG XI, New Orleans, United States. vol. 20; 2016. p. 24.
86. Corballis MC. From hand to mouth: the origins of language. Princeton, NJ: Princeton University Press; 2002. 272 pages.
87. Corballis MC. Language as gesture. Human Movement Science. 2009; 28(5):556–565. https://doi.org/10.1016/j.humov.2009.07.003 PMID: 19665811
88. Aboitiz F. Gestures, vocalizations, and memory in language origins. Front Evol Neurosci. 2012; 4(2). https://doi.org/10.3389/fnevo.2012.00002 PMID: 22347184
89. Pirhonen A, Brewster S, Holguin C. Gestural and audio metaphors as a means of control for mobile devices. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM; 2002. p. 291–298.
90. Wessel D, Wright M, Schott J. Intimate musical control of computers with a variety of controllers and gesture mapping metaphors. In: Proceedings of the 2002 conference on New interfaces for musical expression. National University of Singapore; 2002. p. 1–3.
91. Francese R, Passero I, Tortora G. Wiimote and Kinect: gestural user interfaces add a natural third dimension to HCI. In: Proceedings of the International Working Conference on Advanced Visual Interfaces. ACM; 2012. p. 116–123.
92. Boyer EO, Pyanet Q, Hanneton S, Bevilacqua F. Guiding Motion using a Sound Target. In: 10th International Symposium on Computer Music Multidisciplinary Research (CMMR) Sound, Music and Motion; 2013. p. 176–189.
93. Schnell N, Bevilacqua F. Engaging with Recorded Sound Materials Through Metaphorical Actions. Contemporary Music Review. 2016; 35(4-5):379–401. https://doi.org/10.1080/07494467.2016.1257293
94. Rocchesso D, Lemaitre G, Susini P, Ternström S, Boussard P. Sketching sound with voice and gesture. ACM Interactions. 2015; 22(1):38–41. https://doi.org/10.1145/2685501