The Mirror to Our Soul? Comparisons of Spontaneous and Posed Vocal Expression of Emotion

Journal of Nonverbal Behavior

Volumn 42, Issue 1, 2018, Pages 1-40

  Juslin, Patrik N ^a   Laukka, Petri ^a,b   Bänziger, Tanja ^a  

^a UPPSALA UNIVERSITY   (Sweden)

^b STOCKHOLM UNIVERSITY   (Sweden)

Author keywords

Communication; Emotion; Expression; Nonverbal; Voice

Indexed keywords

HUMAN; HUMAN EXPERIMENT; STIMULUS; VOICE;

EID: 85032225308     PISSN: 01915886     EISSN: 15733653     Source Type: Journal    
DOI: 10.1007/s10919-017-0268-x     Document Type: Article

References (103)

1. Aho, K. (2015). asbio: A collection of statistical tools for biologists. R package version 1.2.5.
2. Altman, D. G. (1991). Practical statistics for medical research. London: Chapman and Hall.
3. Aubergé, V., Audibert, N., & Rilliard, A. (2004). E-Wiz: A trapper protocol for hunting the expressive speech corpora in lab. In M. T. Lino et al. (Eds.), Proceedings of the fourth international conference on language resources and evaluation (pp. 179–182). Paris: European Language Resources Association.
4. Audibert, N., Aubergé, V., & Rilliard, A. (2008). How we are not all equally competent for discriminating acted from spontaneous expressive speech. In P. A. Barbosa, S. Madureira, & C. Reis (Eds.), Fourth conference on speech prosody (pp. 693–696). Campinas: International Speech Communication Association.
5. Audibert, N., Aubergé, V., & Rilliard, A. (2010). Prosodic correlates of acted vs. spontaneous discrimination of expressive speech: A pilot study. In Proceedings of Speech Prosody 2010 (pp. 1–4).
6. Bachorowski, J.-A. (1999). Vocal expression and perception of emotion. Current Directions in Psychological Science, 8, 53–57.
7. Bachorowski, J.-A., & Owren, M. J. (1995). Vocal expression of emotion: Acoustic properties of speech are associated with emotional intensity and context. Psychological Science, 6, 219–224.
8. Banse, R., & Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 614–636.
9. Bänziger, T., Mortillaro, M., & Scherer, K. R. (2012). Introducing the geneva multimodal expression corpus for experimental research on emotion perception. Emotion, 12, 1161–1179.
10. Barrett, J., & Paus, T. (2002). Affect-induced changes in speech production. Experimental Brain Research, 146, 531–537.
11. Brunner, E., Dette, H., & Munk, A. (1997). Box-type approximations in nonparametric factorial designs. Journal of the American Statistical Association, 92, 1494–1502.
12. Buck, R. (2014). Emotion: A biosocial synthesis. Cambridge: Cambridge University Press.
13. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005). A database of German emotional speech. In Proceedings of the 9th European conference on speech communication and technology, Interspeech 2005 (pp. 1517–1520). Lisbon: International Speech Communication Association.
14. Caffi, C., & Janney, R. W. (1994). Toward a pragmatics of emotive communication. Journal of Pragmatics, 22, 325–373.
15. Carletta, J. C. (2007). Unleashing the killer corpus: Experiences in creating the multi-everything AMI meeting corpus. Language Resources and Evaluation, 41, 181–190.
16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah: Erlbaum.
17. Cowie, R., & Cornelius, R. (2003). Describing the emotional states that are expressed in speech. Speech Communication, 40, 5–32.
18. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., et al. (2001). Emotion recognition in human–computer interaction. IEEE Signal Processing Magazine, 18, 32–80.
19. Cullen, C., Vaughan, B., Kousidis, S., & McAuley, J. (2008). Emotional speech corpus construction, annotation, and distribution. In L. Devillers, J.-C., Martin, R. Cowie, E. Douglas-Cowie, & A. Batliner (Eds.), Proceedings of the LREC 2008 workshop on corpora for research on emotion and affect (pp. 32–37). Marrakesh: ELRA.
20. Davitz, J. R. (1964). Auditory correlates of vocal expression of emotional feeling. In J. R. Davitz (Ed.), The communication of emotional meaning (pp. 101–112). New York: McGraw-Hill.
21. Dinno, A. (2009). Exploring the sensitivity of Horn’s parallel analysis to the distributional form of simulated data. Multivariate Behavioral Research, 44, 362–388.
22. Douglas-Cowie, E., Campbell, N., Cowie, R., & Roach, P. (2003). Emotional speech: Towards a new generation of data bases. Speech Communication, 40, 33–60.
23. Douglas-Cowie, E., Cowie, R., & Romano, A. (1999). Changing emotional tone in dialogue and its prosodic correlates. In M. Swerts & J. Terken (Eds.), Proceedings of the ESCA workshop on dialogue and prosody (pp. 41–46). Eindhoven: Eindhoven University.
24. Douglas-Cowie, E., Cowie, R., & Schröder, M. (2000). A new emotion database: Considerations, sources, and scope. In R. Cowie, E. Douglas-Cowie, & M. Schröder (Eds.), Proceedings of the ISCA workshop on speech and emotion (pp. 39–44). Belfast: International Speech Communication Association.
25. Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., et al. (2007). The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data. In A. Paiva, R. Prada, & R. W. Picard (Eds.), Affective computing and intelligent interaction, ACII 2007 (pp. 488–500). Berlin: Springer.
26. Douglas-Cowie, E., Cox, C., Martin, J.-C., Devillers, L., Cowie, R., Sneddon, I., et al. (2011). The HUMAINE database. In P. Petta, C. Pelachaud, & R. Cowie (Eds.), Emotion-oriented systems: The Humaine handbook (pp. 243–284). Berlin: Springer.
27. Ekman, P. (1997). Should we call it expression or communication? Innovation, 10, 333–344.
28. Ekman, P. (Ed.). (1973). Darwin and facial expression. New York: Academic Press.
29. Ekman, P., & Cordaro, D. (2011). What is meant by calling emotions basic? Emotion Review, 3, 364–370.
30. Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49–98.
31. El Ayadi, M., Kamel, M. S., & Karray, F. (2011). Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44, 572–587.
32. Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., André, E., Busso, C., et al. (2016). The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7, 190–202.
33. Eyben, F., Weninger, F., Gross, F., & Schuller, B. (2013). Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In J. Alejandro et al. (Eds.), Proceedings of ACM multimedia 2013 (pp. 835–838). Barcelona: Association for Computing Machinery.
34. Frank, M. G., Juslin, P. N., & Harrigan, J. A. (2005). Technical issues in recording nonverbal behavior. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research (pp. 449–470). New York: Oxford University Press.
35. Frank, M. G., & Stennett, J. (2001). The forced-choice paradigm and the perception of facial expressions of emotion. Journal of Personality and Social Psychology, 80, 75–85.
36. Frick, R. W. (1985). Communicating emotion: The role of prosodic features. Psychological Bulletin, 97, 412–429.
37. Fridlund, A. (1994). Human facial expression: An evolutionary view. San Diego: Academic Press.
38. Frommer, J., Michaelis, B., Rösner, D., Wendemuth, A., Friesen, R., Haase, M., et al. (2012). Towards emotion and affect detection in the multimodal LAST MINUTE corpus. In N. Calzolari et al. (Eds.), Proceedings of the 8th international conference on language resources and evaluation, LREC 2012 (pp. 3064–3069). Istanbul: European Language Resources Association.
39. Gnjatović, M., & Rösner, D. (2010). Inducing genuine emotions in simulated speech-based human–machine interaction: The NIMITEK corpus. IEEE Transactions on Affective Computing, 1, 132–144.
40. Greasley, P., Sherrard, C., & Waterman, M. (2000). Emotion in language and speech: Methodological issues in naturalistic settings. Language and Speech, 43, 355–375.
41. Grimm, M., Kroschel, K., & Narayanan, S. (2008). The Vera am Mittag German audio-visual emotional speech database. In Proceedings of the 2008 IEEE international conference on multimedia and expo, ICME 2008 (pp. 865–868). Piscataway, NJ: Institute of Electrical and Electronics Engineers.
42. Hansen J. H. L., & Bou-Ghazale, S. E. (1997). Getting started with SUSAS: A speech under simulated and actual stress database. In Proceedings of the 5th European conference on speech communication and technology, Eurospeech’97 (pp. 1743–1746). Rhodes: European Speech Communication Association.
43. Haq, S., & Jackson, P. J. B. (2009). Speaker-dependent audio-visual emotion recognition. In B.-J. Theobald & R. Harvey (Eds.), Proceedings of the international conference on auditory-visual speech processing, AVSP 2009 (pp. 53–58). Norwich: International Speech Communication Association.
44. Hawk, S. T., van Kleef, G. A., Fischer, A. H., & van der Schalk, J. (2009). ‘Worth a thousand words’: Absolute and relative decoding of nonlinguistic affect vocalizations. Emotion, 9, 293–305.
45. Izard, C. E. (1993). Organizational and motivational functions of discrete emotions. In M. Lewis & J. M. Haviland (Eds.), Handbook of emotions (pp. 631–641). New York: Guilford Press.
46. Jürgens, R., Grass, A., Drolet, M., & Fischer, J. (2015). Effect of acting experience on emotion expression and recognition in voice: Non-actors provide better stimuli than expected. Journal of Nonverbal Behavior, 39, 195–214.
47. Jürgens, R., Hammerschmidt, K., & Fischer, J. (2011). Authentic and play-acted vocal emotion expressions reveal acoustic differences. Frontiers in Psychology, 2, 180.
48. Juslin, P. N. (2013). Vocal expression of affect: Promises and problems. In E. Altenmüller, S. Schmidt, & E. Zimmerman (Eds.), Evolution of emotional communication (pp. 252–273). New York: Oxford University Press.
49. Juslin, P. N., & Laukka, P. (2001). Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion. Emotion, 1, 381–412.
50. Juslin, P. N., & Laukka, P. (2003). Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin, 129, 770–814.
51. Juslin, P. N., & Laukka, P. (2017). The truly intense vocal affect collection (TIVAC): Presentation and validation. Manuscript submitted for publication.
52. Juslin, P. N., & Scherer, K. R. (2005). Vocal expression of affect. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research (pp. 65–135). New York: Oxford University Press.
53. Kappas, A., & Hess, U. (1995). Nonverbal aspects of oral communication. In U. M. Quasthoff (Ed.), Aspects of oral communication (pp. 169–180). Berlin: DeGruyter.
54. Kehrein, R. (2002). The prosody of authentic emotions. In B. Bel & I. Marlien (Eds.), Proceedings of the speech prosody 2002 conference (pp. 423–426). Aix-en-Provence: Université de Provence.
55. Kim, S., Valente, F., Filippone, M., & Vinciarelli, A. (2014). Predicting continuous conflict perception with Bayesian Gaussian processes. IEEE Transactions on Affective Computing, 5, 187–200.
56. Klasmeyer, G., Johnstone, T., Bänziger, T., Sappok, C., & Scherer, K. R. (2000). Emotional voice variability in speaker verification. In R. Cowie, E. Douglas-Cowie, & M. Schröder (Eds.), Proceedings of the ISCA workshop on speech and emotion (pp. 212–217). Belfast: International Speech Communication Association.
57. Konietschke, F., Placzek, M., Schaarschmidt, F., & Hothorn, L. A. (2015). nparcomp: An R software package for nonparametric multiple comparisons and simultaneous confidence intervals. Journal of Statistical Software, 64, 1–17.
58. Krahmer, E., & Swerts, M. (2008). On the role of acting skills for the collection of simulated emotional speech. In Proceedings of the international conference on spoken language processing (Interspeech 2008). Brisbane: Interspeech.
59. Krebs, J. R., & Davies, N. B. (1993). An introduction to behavioural ecology (3rd ed.). Oxford: Blackwell.
60. Laukka, P., Audibert, N., & Aubergé, V. (2012). Exploring the determinants of the graded structure of vocal emotion expressions. Cognition and Emotion, 26, 710–719.
61. Laukka, P., Elfenbein, H. A., Chui, W., Thingujam, N. S., Iraki, F. K., Rockstuhl, T., & Althoff, J. (2010). Presenting the VENEC corpus: Development of a cross-cultural corpus of vocal emotion expressions and a novel method of annotating emotion appraisals. In L. Devillers, B. Schuller, R. Cowie, E. Douglas-Cowie, & A. Batliner (Eds.), Proceedings of the LREC 2010 workshop on corpora for research on emotion and affect (pp. 53–57). Valletta: European Language Resources Association.
62. Laukka, P., Elfenbein, H. A., Söder, N., Nordström, H., Althoff, J., Chui, W., et al. (2013). Cross-cultural decoding of positive and negative non-linguistic vocalizations. Frontiers in Psychology, 4, 353.
63. Laukka, P., Neiberg, D., Forsell, M., Karlsson, I., & Elenius, K. (2011). Expression of affect in spontaneous speech: Acoustic correlates, perception, and automatic detection of irritation and resignation. Computer Speech and Language, 25, 84–104.
64. Levenson, R. W. (1994). Human emotion: A functional view. In P. Ekman & R. J. Davidson (Eds.), The nature of emotion: Fundamental questions (pp. 123–126). Oxford: Oxford University Press.
65. Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006). The eNTERFACE’05 audio-visual emotion database. In Proceedings of the 22nd international conference on data engineering workshops, ICDEW’06. Piscataway, NJ: Institute of Electrical and Electronics Engineers.
66. McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schröder, M. (2012). The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Transactions on Affective Computing, 3, 5–17.
67. Morton, E. S. (1977). On the occurrence and significance of motivation-structural rules in some bird and mammal sounds. American Naturalist, 111, 855–869.
68. Murray, I. R., & Arnott, J. L. (1993). Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. Journal of the Acoustical Society of America, 93, 1097–1108.
69. Neiberg, D., Elenius, K., & Laskowski, K. (2006). Emotion recognition in spontaneous speech using GMMs. In Proceedings of the 9th international conference on spoken language processing, Interspeech 2006 (pp. 809–812). Pittsburgh, PA: International Speech Communication Association.
70. Nordström, H., & Laukka, P. (2017). The time course of emotion recognition in speech and music. Manuscript submitted for publication.
71. Norman, N. (2014). Method acting: ‘Faking it’ is so last year—Commitment is everything. Newsweek. Retrived from http://europe.newsweek.com/.
72. Owren, M. J., & Bachorowski, J.-A. (2007). Measuring emotion-related vocal acoustics. In J. Coan & J. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 239–266). New York: Oxford University Press.
73. Pell, M. D., & Skorup, V. (2008). Implicit processing of emotional prosody in a foreign versus native language. Speech Communication, 50, 519–530.
74. Pittermann, J., Pittermann, A., & Minker, W. (2010). Handling emotions in human–computer dialogues. Dordrecht: Springer.
75. Planalp, S. (1998). Communicating emotion in everyday life: Cues, channels, and processes. In P. A. Andersen & L. K. Guerrero (Eds.), Handbook of communication and emotion (pp. 29–48). New York: Academic Press.
76. Plutchik, R. (1994). The psychology and biology of emotion. New York: Harper-Collins College Publishers.
77. Rosenthal, R. (1987). Judgment studies: Design, analysis, and meta-analysis. Cambridge: Cambridge University Press.
78. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.
79. Russell, J. A., Bachorowski, J.-A., & Fernandez-Dols, J. M. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329–349.
80. Scheiner, E., & Fischer, J. (2011). Emotion expression—The evolutionary heritage in the human voice. In W. Welsch, W. Singer, & A. Wunder (Eds.), Interdisciplinary anthropology: The continuing evolution of man (pp. 105–130). Heidelberg: Springer.
81. Scherer, K. R. (1985). Vocal affect signalling: A comparative approach. In J. Rosenblatt, C. Beer, M.-C. Busnel, & P. J. B. Slater (Eds.), Advances in the study of behavior (Vol. 15, pp. 189–244). New York: Academic Press.
82. Scherer, K. R. (1986). Vocal affect expression: A review and a model for future research. Psychological Bulletin, 99, 143–165.
83. Scherer, K. R. (2013). Vocal markers of emotion: Comparing induction and acting elicitation. Computer Speech and Language, 27, 40–58.
84. Scherer, K. R., Banse, R., Wallbott, H. G., & Goldbeck, T. (1991). Vocal cues in emotion encoding and decoding. Motivation and Emotion, 15, 123–148.
85. Scherer, K. R., & Bänziger, T. (2010). On the use of actor portrayals in research on emotional expression. In K. R. Scherer, T. Bänziger, & E. B. Roesch (Eds.), Blueprint for affective computing: A sourcebook (pp. 166–176). Oxford: Oxford University Press.
86. Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011). Recognizing realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge. Speech Communication, 53, 1062–1087.
87. Sneddon, I., McRorie, M., McKeown, G., & Hanratty, J. (2012). The Belfast induced natural emotion database. IEEE Transactions on Affective Computing, 3, 32–41.
88. Snowdon, C. T. (2003). Expression of emotion in nonhuman animals. In R. J. Davidson, K. R. Scherer, & H. H. Goldsmith (Eds.), Handbook of affective sciences (pp. 457–480). New York: Oxford University Press.
89. Spencer, H. (1857). The origin and function of music. Fraser’s Magazine, 56, 396–408.
90. Sundberg, J. (1998). Expressivity in singing: A review of some recent investigations. Logopedics Phoniatrics Vocology, 23, 121–127.
91. Sundberg, J., Patel, S., Björkner, E., & Scherer, K. (2011). Interdependencies among voice source parameters in emotional speech. IEEE Transactions on Affective Computing, 2, 162–174.
92. Tassinary, L. G., & Cacioppo, J. T. (1992). Unobservable facial actions and emotion. Psychological Science, 3, 28–33.
93. Tatham, M., & Morton, K. (2004). Expression in speech: Analysis and synthesis. Oxford: Oxford University Press.
94. Truong, K. P., van Leeuwen, D. A., & de Jong, F. M. G. (2012). Speech-based recognition of self-reported and observed emotion in a dimensional space. Speech Communication, 54, 1049–1063.
95. van Bezooijen, R. (1984). Characteristics and recognizability of vocal expressions of emotion. Dordrecht: Foris.
96. Ververidis, D., & Kotropoulos, C. (2006). Emotional speech recognition: Resources, features, and methods. Speech Communication, 48, 1162–1181.
97. Visser, P. S., Krosnick, J. A., & Lavrakas, P. J. (2000). Survey research. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 223–252). New York: Cambridge University Press.
98. Wagner, H. L. (1993). On measuring performance in categorical judgment studies of nonverbal behavior. Journal of Nonverbal Behavior, 17, 3–28.
99. Wilcox, R. R. (2012). Modern statistics for the social and behavioral sciences: A practical introduction. New York: CRC Press.
100. Williams, C. E., & Stevens, K. N. (1972). Emotions and speech: Some acoustical correlates. Journal of the Acoustical Society of America, 52, 1238–1250.
101. Wilting, J., Krahmer, E., & Swerts, M. (2006). Real vs. acted emotional speech. In Proceedings of Interspeech 2006 (pp. 805–808). Pittsburgh, PA.
102. With, S., & Kaiser, S. (2011). Sequential patterning of facial actions in the production and perception of emotional expressions. Swiss Journal of Psychology, 70, 241–252.
103. Zuckerman, M., Larrance, D. T., Hall, J. A., DeFrank, R. S., & Rosenthal, R. (1979). Posed and spontaneous communication of emotion via facial and vocal cues. Journal of Personality, 47, 712–733.