On the acoustics of emotion in audio: What speech, music, and sound have in common

Frontiers in Psychology

Volumn 4, Issue MAY, 2013, Pages

  Weninger, Felix ^a   Eyben, Florian ^a   Schuller, Björn W ^a,b   Mortillaro, Marcello ^b   Scherer, Klaus R ^b  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF GENEVA   (Switzerland)

Author keywords

Audio signal processing; Emotion recognition; Feature selection; Music perception; Sound perception; Speech perception; Transfer learning

Indexed keywords

EID: 84878925980     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2013.00292     Document Type: Article

References (38)

1. Bänziger, T., Mortillaro, M., and Scherer, K. R. (2012). Introducing the Geneva multimodal expression corpus for experimental research on emotion perception. Emotion 12, 1161-1179. doi:10.1037/a0025827
2. Bone, D., Lee, C.-C., and Narayanan, S. (2012). "A robust unsupervised arousal rating framework using prosody with cross-corpora evaluation," in Proceeding of the Interspeech (Portland, OR: ISCA).
3. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W., and Weiss, B. (2005). "A database of German emotional speech," in Proceeding of the Interspeech (Lisbon: ISCA), 1517-1520.
4. Daido, R., Hahm, S., Ito, M., Makino, S., and Ito, A. (2011). "A system for evaluating singing enthusiasm for karaoke," in Proceeding of the ISMIR (Miami, FL: International Society for Music Information Retrieval), 31-36.
5. Davis, S. B., and Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Trans. Acoust. 28, 357-366. doi:10.1109/TASSP.1980.1163420
6. Drossos, K., Floros, A., and Kanellopoulos, N.-G. (2012). "Affective acoustic ecology: towards emotionally enhanced sound events," in Proceedings of the 7th Audio Mostly Conference: A Conference on Interaction with Sound (New York, NY: ACM), 109-116.
7. Fontaine, J., Scherer, K. R., Roesch, E., and Ellsworth, P. (2007). The world of emotion is not two-dimensional. Psychol. Sci. 18, 1050-1057. doi:10.1111/j.1467-9280.2007.02024.x
8. Grimm, M., and Kroschel, K. (2005). "Evaluation of natural emotions using self assessment manikins," in Proceeding of the ASRU (Cancún: IEEE), 381-385.
9. Grimm, M., Kroschel, K., Mower, E., and Narayanan, S. (2007a). Primitives-based evaluation and estimation of emotions in speech. Speech Commun. 49, 787-800. doi:10.1016/j.specom.2007.01.010
10. Grimm, M., Kroschel, K., and Narayanan, S. (2007b). "Support vector regression for automatic recognition of spontaneous emotions in speech," in Proceeding of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. IV (Honolulu, HI: IEEE), 1085-1088.
11. Grimm, M., Kroschel, K., and Narayanan, S. (2008). "The Vera am Mittag German audio-visual emotional speech database," in Proceeding of the IEEE International Conference on Multimedia and Expo (ICME) (Hannover: IEEE), 865-868.
12. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. (2009). The WEKA data mining software: an update. SIGKDD Explor. 11, 10-18. doi:10.1145/1656274.1656278
13. Hu, X., Downie, J. S., Laurier, C., Bay, M., and Ehmann, A. F. (2008). "The 2007 MIREX audio mood classification task: lessons learned," in Proceeding of the ISMIR (Philadelphia: International Society for Music Information Retrieval), 462-467.
14. Ilie, G., and Thompson, W. F. (2006). A comparison of acoustic cues in music and speech for three dimensions of affect. Music Percept. 23, 319-329. doi:10.1525/mp.2006.23.4.319
15. Juslin, P. N., and Laukka, P. (2003). Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129, 770-814. doi:10.1037/0033-2909.129.5.770
16. McKeown, G., Valstar, M., Cowie, R., Pantic, M., and Schröder, M. (2012). The SEMAINE database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3, 5-17. doi:10.1109/T-AFFC.2011.20
17. Mortillaro, M., Meuleman, B., and Scherer, K. R. (2012). Advocating a componential appraisal model to guide emotion recognition. Int. J. Synth. Emot. 3, 18-32. doi:10.4018/jse.2012010102
18. Nilsonne, A., and Sundberg, J. (1985). Differences in ability of musicians and nonmusicians to judge emotional state from the fundamental frequency of voice samples. Music Percept. 2, 507-516. doi:10.2307/40285316
19. Peeters, G. (2004). A Large Set of Audio Features for Sound Description. Technical Report. Paris: IRCAM.
20. Platt, J. C. (1999). "Fast training of support vector machines using sequential minimal optimization," in Advances in Kernel Methods: Support Vector Learning (Cambridge, MA: MIT Press), 185-208.
21. Rice, S. V., and Bailey, S. M. (2005). "A web search engine for sound effects," in Proceeding of the 119th Conference of the Audio Engineering Society (AES) (New York: Audio Engineering Society).
22. Scherer, K. R. (1991). "Emotion expression in speech and music," in Music, Language, Speech, and Brain, eds J. Sundberg, L. Nord, and R. Carlson (London: Macmillan), 146-156.
23. Scherer, K. R. (2013). "Emotion in action, interaction, music, and speech," in Language, Music, and the Brain: A Mysterious Relationship, ed. M. Arbib (Cambridge, MA: MIT Press), 107-139.
24. Scherer, K. R., Johnstone, T., and Klasmeyer, G. (2003). "Vocal expression of emotion," in Handbook of Affective Sciences, eds R. J. Davidson, K. R. Scherer, and H. H. Goldsmith (Oxford, NY: Oxford University Press), 433-456.
25. Schuller, B., Batliner, A., Steidl, S., and Seppi, D. (2011a). Recognising realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge. Speech Commun. 53, 1062-1087. doi:10.1016/j.specom.2011.01.011
26. Schuller, B., Weninger, F., and Dorfner, J. (2011b). "Multi-modal non-prototypical music mood analysis in continuous space: reliability and performances," in Proceedings 12th International Society for Music Information Retrieval Conference, ISMIR 2011 (Miami, FL: ISMIR), 759-764.
27. Schuller, B., Dorfner, J., and Rigoll, G. (2010). Determination of nonprototypical valence and arousal in popular music: features and performances. EURASIP J. Audio Speech Music Process. 2010:735854. doi:10.1186/1687-4722-2010-735854
28. Schuller, B., Hantke, S., Weninger, F., Han, W., Zhang, Z., and Narayanan, S. (2012). "Automatic recognition of emotion evoked by general sound events," in Proceedings 37th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2012 (Kyoto: IEEE), 341-344.
29. Schuller, B., Steidl, S., and Batliner, A. (2009). "The INTERSPEECH 2009 emotion challenge," in Proceeding of the Interspeech (Brighton: ISCA), 312-315.
30. Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., et al. (2013). "The INTERSPEECH 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism," in Proceedings INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association (Lyon: ISCA).
31. Smola, A., and Schölkopf, B. (2004). A tutorial on support vector regression. Stat. Comput. 14, 199-222. doi:10.1023/B:STCO.0000035301.49549.88
32. Steidl, S. (2009). Automatic Classification of Emotion-Related User States in Spontaneous Children's Speech. Berlin: Logos Verlag.
33. Sundaram, S., and Schleicher, R. (2010). "Towards evaluation of example-based audio retrieval system using affective dimensions," in Proceeding of the ICME (Singapore: IEEE), 573-577.
34. Thompson, W. F., Schellenberg, E. G., and Husain, G. (2004). Decoding speech prosody: do music lessons help? Emotion 4, 46-64. doi:10.1037/1528-3542.4.1.46
35. Ververidis, D., and Kotropoulos, C. (2006). Emotional speech recognition: resources, features, and methods. Speech Commun. 48, 1162-1181. doi:10.1016/j.specom.2006.04.003
36. Yang, Y.-H., and Chen, H.-H. (2012). Machine recognition of music emotion: a review. ACM Trans. Intell. Syst. Technol. 3, 1-30. doi:10.1145/2168752.2168754
37. Young, S. J., Evermann, G., Gales, M. J. F., Hain, T., Kershaw, D., Liu, X., et al. (2006). The HTK Book, Version 3.4.1. Cambridge: Cambridge University Engineering Department.
38. Zwicker, E., and Fastl, H. (1999). Psychoacoustics - Facts and Models. Heidelberg: Springer.