Influence of contextual information in emotion annotation for spoken dialogue systems

Speech Communication

Volumn 50, Issue 5, 2008, Pages 416-433

  Callejas, Zoraida ^a   López Cózar, Ramón ^a  

^a UNIVERSITY OF GRANADA   (Spain)

Author keywords

Acoustic context; Affective computing; Dialogue context; Emotion annotation; Emotion recognition; Emotional speech; Spoken dialogue system

Indexed keywords

ACOUSTICS; SPEECH INTELLIGIBILITY; SPEECH PROCESSING; SPEECH RECOGNITION; USER INTERFACES;

ACOUSTIC CONTEXT; AFFECTIVE COMPUTING; CONTEXTUAL INFORMATION; DIALOGUE CONTEXT; EMOTION ANNOTATION; SPOKEN DIALOGUE SYSTEMS;

SPEECH SYNTHESIS;

EID: 41049085760     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2008.01.001     Document Type: Article

References (68)

1. Adell J., Bonafonte A., and Escudero D. Analysis of prosodic features towards modelling of emotional and pragmatic attributes of speech. Procesamiento de Lenguaje Natural 35 (2005) 277-284
2. Ai, H., Litman, D., Forbes-Riley, K., Rotaru, M., Tetreault, J., Purandare, A., 2006. Using systems and user performance features to improve emotion detection in spoken tutoring dialogs. In: Proceedings of the Ninth International Conference on Spoken Language Processing (Interspeech 2006 - ICSLP), Pittsburgh, PA, USA, pp. 797-800.
3. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., Stolcke, A., 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In: Proceedings of the Seventh International Conference on Spoken Language Processing (Interspeech 2002 - ICSLP), Denver, USA, pp. 2037-2040.
4. 3 = alpha (or beta). Tech. rep., University of Essex.
5. Bickmore, T., Giorgino, T., 2004. Some novel aspects of health communication from a dialogue systems perspective. In: Proceedings of the AAAI Fall Symposium on Dialogue Systems for Health Communication, Washington, DC, USA, pp. 275-291.
6. Bishop C.M. Pattern Recognition and Machine Learning (2006), Springer
7. Boehner K., DePaula R., Dourish P., and Sengers P. How emotion is made and measured. International Journal of Human-Computer Studies, Special Issue on Evaluating Affective Interactions 65 4 (2007) 275-291
8. Boersma, P., 1993. Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound. In: Proceedings of Institute of Phonetic Sciences, University of Amsterdam, Vol. 17, pp. 97-110.
9. Burkhardt, F., van Ballegooy, M., Englert, R., Huber, R., 2005. An emotion-aware voice portal. In: Proceedings of the Electronic Speech Signal Processing (ESSP), Prague, Czech Republic, pp. 123-131.
10. Callejas, Z., López-Cózar, R., 2005. Implementing modular dialogue systems: a case study. In: Proceedings of the Applied Spoken Language Interaction in Distributed Environments (ASIDE 2005).
11. Camurri A., Mazzarino B., and Volpe G. Expressive interfaces. Cognition, Technology and Work 6 1 (2004) 15-22
12. Cowie R., and Schröder M. Piecing together the emotion jigsaw. Lecture Notes on Computer Science 3361 (2005) 305-317
13. Cicchetti D.V., and Feinstein A.R. High agreement but low Kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 43 6 (1990) 551-558
14. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70 4 (1968) 213-220
15. Corradini, A., Mehta, M., Bernsen, N.O., Charfuelan, M., 2005. Animating an interactive conversational character for an educational game system. In: Proceedings of the Tenth International Conference on Intelligent User Interfaces, San Diego, CA, USA, pp. 183-190.
16. Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., Schröder, M., 2000. 'FEELTRACE': an instrument for recording perceived emotion in real time. In: Proceedings of the International Speech Communication Association (ISCA) Workshop on Speech and Emotion, Belfast Northern Ireland, pp. 19-24.
17. Cowie R., Douglas-Cowie E., Tsapatsoulis N., Votsis G., Kollias S., Fellenz W., and Taylor J. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18 1 (2001) 32-80
18. Craggs, R., Wood, M.M., 2003. Annotating emotion in dialogue. In: Proceedings of the Fourth SIGdial Workshop on Discourse and Dialogue, Sapporo, Japan, pp. 218-225.
19. Critchley H.D., Rotshtein P., Nagai Y., O'Doherty J., Mathias C.J., and Dolan R.J. Activity in the human brain predicting differential heart rate responses to emotional facial expressions. NeuroImage 24 (2005) 751-762
20. Davies M., and Fleiss J.L. Measuring agreement for multinomial data. Biometrics 38 4 (1982) 1047-1051
21. Devillers L., Vidrascu L., and Lamel L. Challenges in real-life emotion annotation and machine learning based detection. Neural Networks 18 (2005) 407-422
22. Douglas-Cowie E., Campbell N., Cowie R., and Roach P. Emotional speech: towards a new generation of databases. Speech Communication, Special Issue on Speech and Emotion 40 (2003) 33-60
23. Dunn G. Design and Analysis of Reliability Studies: The Statistical Evaluation of Measurement Errors (1989), Edward Arnold
24. Feinstein A.R., and Cicchetti D.V. High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43 6 (1990) 543-549
25. Fleiss J.L. Measuring nominal scale agreement among many raters. Psychological Bulletin 76 5 (1971) 378-382
26. Fleiss J.L., and Cohen J. The equivalence of weighted kappa and the interclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 33 (1973) 613-619
27. Forbes-Riley, K., Litman, D.J., 2004. Predicting emotion in spoken dialogue from multiple knowledge sources. In: Proceedings of the Human Language Technology Conference - North American Chapter of the Association for Computational Linguistics Annual Meeting (HLT-NAACL 2004), pp. 201-208.
28. Gebhard, P., Klesen, M., Rist, T., 2004. Coloring multi-character conversations through the expression of emotions. In: Proceedings of the Tutorial and Research Workshop on Affective Dialogue Systems, Kloster Irsee, Germany, pp. 128-141.
29. Gerfen C. Andalusian codas. Probus 14 (2002) 247-277
30. González, G.M., 1999. Bilingual computer-assisted psychological assessment: an innovative approach for screening depression in chicanos/latinos. Tech. Rep. 39, University of Michigan.
31. Gut, U., Bayerl, P.S., 2004. Measuring the reliability of manual annotations of speech corpora. In: Proceedings of the Second International Conference on Speech Prosody (SP2004), Nara, Japan, pp. 565-568.
32. Guyon I., and Elisseeff A. An introduction to variable and feature selection. Journal of Machine Learning Research 3 (2003) 1157-1182
33. Hall, L., Woods, S., Aylett, R., Paiva, A., Newall, L., 2005. Achieving empathic engagement through affective interaction with synthetic characters. In: Proceedings of the First International Conference on Affective Computing and Intelligent Interaction, Beijing, China, pp. 731-738.
34. Hansen J.H.L. Analysis and compensation of speech under stress and noise for environmental robustness in speech recognition. Speech Communication, Special Issue on Speech under Stress 20 2 (1996) 151-170
35. Hozjan, V., Kacic, Z., Moreno, A., Bonafonte, A., Nogueiras, A., 2002. Interface databases: design and collection of a multilingual emotional speech database. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002), Las Palmas de Gran Canaria, Spain, pp. 2019-2023.
36. Iriondo, I., Guaus, R., Rodríguez, A., Lázaro, P., Montoya, N., Blanco, J.M., Bernadas, D., Oliver, J.M., Tena, D., Longhi, L., 2000. Validation of an acoustical modelling of emotional expression in Spanish using speech synthesis techniques. In: Proceedings of the International Speech Communication Association (ISCA) Workshop on Speech and Emotion, Belfast Northern Ireland, pp. 161-166.
37. Johnstone, T., 1996. Emotional speech elicited using computer games. In: Proceedings of the Fourth International Conference on Spoken Language Processing (ICSLP 1996), Vol. 3, Philadelphia, PA, pp. 1985-1988.
38. Krippendorff K. Content Analysis: An Introduction to its Methodology (2003), Sage Publications Inc.
39. Landis J.R., and Koch G.G. The measurement of observer agreement for categorical data. Biometrics 33 (1977) 159-174
40. Lantz C.A., and Nebenzahl E. Behavior and interpretation of the κ statistic: resolution of the two paradoxes. Journal of Clinical Epidemiology 49 4 (1996) 431-434
41. Lee C.M., and Narayanan S.S. Toward detecting emotions in spoken dialogs. IEEE Transactions on Speech and Audio Processing 13 2 (2005) 293-303
42. Lee, C., Yoo, S.K., Park, Y.J., Kim, N.H., Jeong, K.S., Lee, B.C., 2005. Using neural network to recognize human emotions from heart rate variability and skin resistance. In: Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China, pp. 5523-5525.
43. Liscombe, J., Riccardi, G., Hakkani-Tür, D., 2005. Using context to improve emotion detection in spoken dialog systems. In: Proceedings of the Ninth European Conference on Speech Communication and Technology (Interspeech 2005 - Eurospeech), Lisbon, Portugal, pp. 1845-1848.
44. Litman D.J., and Forbes-Riley K. Recognizing student emotions and attitudes on the basis of utterances in spoken tutoring dialogues with both human and computer tutors. Speech Communication 48 5 (2006) 559-590
45. Mahlke, S., 2006. Emotions and EMG measures of facial muscles in interactive contexts. In: Proceedings of the 2006 Conference on Human Factors in Computer Systems (CHI 2006), Montréal, Canada.
46. Montero, J.M., Gutiérrez-Arriola, J., Enríquez, E., Pardo, J.M., 1999. Analysis and modelling of emotional speech in Spanish. In: Proceedings of the 14th International Conference of Phonetic, San Francisco, USA, pp. 957-960.
47. Morrison D., Wang R., and De Silva L.C. Ensemble methods for spoken emotion recognition in call-centers. Speech Communication 49 (2007) 98-112
48. O'Neill P. Utterance final /s/ in Andalusian Spanish. The phonetic neutralization of a phonological contrast. Language Design 7 (2005) 151-166
49. Picard, R.W., Daily, S.B., 2005. Evaluating affective interactions: alternatives to asking what users feel. In: Proceedings of the 2005 Conference on Human Factors in Computer Systems (CHI 2005), Workshop: Evaluating Affective Interfaces - Innovative Approaches, In CHI 2005, Portland, Oregon, USA.
50. Pitterman, J., Pitterman, A., 2006. Integrating emotion recognition into an adaptive spoken language dialogue system. In: Proceedings of the Second IEEE International Conference on Intelligent Environments, Athens, Greece, pp. 197-202.
51. Plutchik R. EMOTION: A Psychoevolutionary Synthesis (1980), Harper and Row Publishers
52. Riccardi G., and Hakkani-Tür D. Grounding emotions in human-machine conversational systems. Lecture Notes in Computer Science (2005) 144-154
53. Rotaru, M., Litman, D.J., 2006. Discourse structure and speech recognition problems. In: Proceedings of the Ninth International Conference on Spoken Language Processing (Interspeech 2006 - ICSLP), Pittsburgh, PA, USA, pp. 53-56.
54. Rumelhart D.E., Hinton G.E., and Williams R.J. Learning Internal Representations by Error Propagation (1986), MIT Press
55. Russell J.A. A circumplex model of affect. Journal of Personality and Social Psychology 39 (1980) 1161-1178
56. Scherer K.R. What are emotions? and how can they be measured?. Social Science Information 44 4 (2005) 694-729
57. Scott W. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly 19 3 (1955) 321-325
58. Shafran, I., Mohri, M., 2005. A comparison of classifiers for detecting emotion from speech. In: Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP 05), pp. 341-344.
59. Shafran, I., Riley, M., Mohri, M., 2003. Voice signatures. In: Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pp. 31-36.
60. Stibbard, R.M., 2000. Automated extraction of ToBI annotation data from the Reading/Leeds emotional speech corpus. In: Proceedings of the International Speech Communication Association (ISCA) Workshop on Speech and Emotion, Belfast, Northern Ireland, UK, pp. 60-65.
61. Streit M., Batliner A., and Portele T. Emotion analysis and emotion-handling subdialogues. In: Wahlster W. (Ed). SmartKom - Foundations of Multimodal Dialogue Systems (2006), Springer-Verlag 317-332
62. Ververidis D., and Kotropoulos C. Emotional speech recognition: resources, features and methods. Speech Communication 48 (2006) 1162-1181
63. Vidrascu L., and Devillers L. Real-life emotion representation and detection in call centers data. Lecture Notes on Computer Science 3784 (2005) 739-746
64. Vogt, T., André, E., 2005. Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), pp. 474-477.
65. Wilks, Y., 2006. Artificial companions as a new kind of interface to the future internet. Tech. Rep. 13, Oxford Internet Institute.
66. Wilting, J., Krahmer, E., Swerts, M., 2006. Real vs. acted emotional speech. In: Proceedings of the Ninth International Conference on Spoken Language Processing (Interspeech 2006 - ICSLP), Pittsburgh, PA, USA, pp. 805-808.
67. Witten I.H., and Frank E. Data Mining: Practical Machine Learning Tools and Techniques (2005), Morgan Kaufmann
68. Zeng, Z., Hu, Y., Fu, Y., Huang, T.S., Roisman, G.I., Wen, Z., 2006. Audio-visual emotion recognition in adult attachment interview. In: Proceedings of the Eighth International Conference on Multimodal Interfaces, Banff, Alberta, Canada, pp. 828-831.