Inter-rater reliability for emotion annotation in human-computer interaction: Comparison and methodological improvements

Journal on Multimodal User Interfaces

Volumn 8, Issue 1, 2014, Pages 17-28

  Siegert, Ingo ^a   Böck, Ronald ^a   Wendemuth, Andreas ^a  

^a OTTO VON GUERICKE UNIVERSITY   (Germany)

Author keywords

Affective state; Annotation; Context influence; Inter rater agreement; Labelling

Indexed keywords

HUMAN COMPUTER INTERACTION; LABELING;

AFFECTIVE STATE; ANNOTATION; CONTEXT INFLUENCE; HUMAN COMMUNICATIONS; INTER-RATER AGREEMENTS; INTER-RATER RELIABILITIES; KRIPPENDORFF'S ALPHAS; RECOGNITION OF EMOTION;

RELIABILITY;

EID: 84902364868     PISSN: 17837677     EISSN: 17838738     Source Type: Journal    
DOI: 10.1007/s12193-013-0129-9     Document Type: Article

References (74)

1. Altman DG (1991) Practical statistics for medical research. Chapman & Hall, London.
2. Artstein R, Poesio M (2008) Inter-coder agreement for computational linguistics. Comput Linguist 34(4): 555-596.
3. Batliner A, Hacker C, Steidl S, Nöth E, Russell M, Wong M (2004) "You stupid tin box"-children interacting with the AIBO robot: a cross-linguistic emotional speech corpus. In: Proceedings of LREC, pp 865-868.
4. Böck R, Siegert I, Vlasenko B, Wendemuth A, Haase M, Lange J (2011) A processing tool for emotionally coloured speech. In: Proceedings of ICME, s. p.
5. Bradley M, Lang P (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psy 25(1): 49-59.
6. Burger S, MacLaren V, Yu H (2002) The ISL meeting corpus: the impact of meeting type on speech style. In: Proceedings of the international conference on spoken language processing, pp 301-304.
7. Burkhardt F, Paeschke A, Rolfes M, Sendlmeier W, Weiss B (2005) A database of german emotional speech. In: Proceedings of interspeech, pp 1517-1520.
8. Callejas Z, Lpez-Czar R (2008) Influence of contextual information in emotion annotation for spoken dialogue systems. Speech Commun 50(5): 416-433.
9. Cauldwell RT (2000) Where did the anger go? The role of context in interpreting emotion in speech. In: Proceedings of ITRW on speech and, emotion, pp 127-131.
10. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 24(1): 37-46.
11. Cowie R, Cornelius RR (2003) Describing the emotional states that are expressed in speech. Speech Commun 40(1-2): 5-32.
12. Crawford JR, Henry JD (2004) The positive and negative affect schedule (PANAS): construct validity, measurement properties and normative data in a large non-clinical sample. Br J Clin Psychol 43(3): 245-265.
13. Cronbach L (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3): 297-334.
14. Devillers L, Vasilescu I (2004) Reliability of lexical and prosodic cues in two real-life spoken dialog corpora. In: Proceedings of LREC, pp 865-868.
15. Devillers L, Vidrascu L, Lamel L (2005) Challenges in real-life emotion annotation and machine learning based detection. Neural Netw 18(4): 407-422.
16. Douglas-Cowie E, Cowie R, Schröder M (2000) A new emotion database: considerations, sources and scope. In: Proceedings of ITRW on speech and, emotion, pp 39-44.
17. Douglas-Cowie E, Cowie R, Sneddon I, Cox C, Lowry O, McRorie M, Martin JC, Devillers L, Abrilian S, Batliner A, Amir N, Karpouzis K (2007) The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data. In: Proceedings of ACII. Berlin, Heidelberg, pp 488-500.
18. Douglas-Cowie E, Devillers L, Martin JC, Cowie R, Savvidou S, Abrilian S, Cox C (2005) Multimodal databases of everyday emotion: facing up to complexity. In: Proceedings of EUROSPEECH, pp 813-816.
19. Eggink J, Bland D (2012) A large scale experiment for mood-based classification of TV programmes. In: Proceedings of ICME, pp 140-145.
20. Ekman P (1992) Are there basic emotions? Psychol Rev 99(3): 550-553.
21. El Ayadi M, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit 44(3): 572-587.
22. Engberg IS, Hansen AV (1996) Documentation of the danish emotional speech database (DES). Technical report, Center for Person, Kommunikation, Aalborg University, Denmark. Internal aau report.
23. Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 43(6): 543-549.
24. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5): 378-382.
25. Fleiss JL, Levin B, Paik MC (1991) Statistical methods for rates & proportions, 3rd edn. Wiley, Hoboken.
26. Fragopanagos N, Taylor J (2005) Emotion recognition in human-computer interaction. Neural Netw 18(4): 389-405.
27. Frommer J, Michaelis B, Rösner D, Wendemuth A, Friesen R, Haase M, Kunze M, Andrich R, Lange J, Panning A, Siegert I (2012) Towards emotion and affect detection in the multimodal LAST MINUTE corpus. In: Proceedings of LREC, pp 3064-3069.
28. Frommer J, Rösner D, Haase M, Lange J, Friesen R, Otto M (2012) Detection and avoidance of failures in dialogues-Wizard of Oz Experiment Operator's Manual. Pabst Science Publishers.
29. Gehm T, Scherer K (1988) Factors determining the dimensions of subjective emotional space. In: Scherer K (ed) Facets of emotion: recent research. Erlbaum, Hillsdale, NJ, pp 99-114.
30. Gnjatović M, Rösner D (2008) The NIMITEK corpus of affected behavior in human-machine interaction. In: Proceedings of LREC, pp 5-8.
31. Grandjean D, Sander D, Scherer K (2008) Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Conscious Cogn 17(2): 484-495.
32. Grimm M, Kroschel K (2005) Evaluation of natural emotions using self assessment manikins. In: IEEE workshop on automatic speech recognition and understanding, pp 381-385.
33. Grimm M, Kroschel K, Narayanan S (2008) The vera am mittag german audio-visual emotional speech database. In: Proceedings of ICME, pp 865-868.
34. Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61(1): 29-48.
35. Gwet KL (2008) Intrarater reliability. In: D'Agostino RB, Sullivan L, Massaro J (eds) Wiley encyclopedia of clinical trials. Wiley, Hoboken, pp 473-485.
36. Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Commun Methods Meas 24(1): 77-89.
37. Ibáñez J (2011) Showing emotions through movement and symmetry. Comput Hum Behav 27(1): 561-567.
38. Izard CE, Libero DZ, Putnam P, Haynes OM (1993) Stability of emotion experiences and their relations to traits of personality. J Pers Soc Psychol 64(5): 847-860.
39. Krippendorff K (2007) Computing Krippendorff's alpha reliability. University of Pennsylvania, Annenberg School for Communication, Technical report.
40. Krippendorff K (2012) Content analysis: an introduction to its methodology, 3rd edn. SAGE Publications, Thousand Oaks.
41. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1): 159-174.
42. Lang PJ (1980) Behavioral treatment and bio-behavioral assessment: computer applications. In: Sidowski JB, Johnson JH, Williams TA (eds) Technology in mental health care delivery systems. Ablex Pub. Corp., pp 119-137.
43. Lee CM, Narayanan S (2005) Toward detecting emotions in spoken dialogs. IEEE Trans Speech Audio Process 13(2): 293-303.
44. McDougall W (1926) An introduction to social psychology, revised edn. John W. Luce & Co, Boston.
45. McKeown G, Valstar M, Cowie R, Pantic M (2010) The semaine corpus of emotionally coloured character interactions. In: Proceedings of ICME, pp 1079-1084.
46. McKeown G, Valstar M, Cowie R, Pantic M, Schroder M (2012) The semaine database: annotated multimodal records of emotionally coloured conversations between a person and a limited agent. IEEE Trans Affect Comput 3(1): 5-17.
47. Mehrabian A (1970) A semantic space for nonverbal behavior. J Consult Clin Psychol 35(2): 248-257.
48. Morris JD (1995) SAM: the self-assessment manikin an efficient cross-cultural measurement of emotional response. J Advert Res 35(6): 63-68.
49. Morris JD, McMullen JS (1994) Measuring multiple emotional responses to a single television commercial. Adv Consum Res 21: 175-180.
50. Osgood CE, Miron MS, May WH (1975) Cross-cultural universals of affective meaning. University of Illinois Press, Urbana.
51. Plutchik R (1980) Emotion, a psychoevolutionary synthesis. Harper & Row, New York.
52. Pugmire D (1994) Real emotion. Philos Phenomen Res 54(1): 105-122.
53. Rösner D, Friesen R, Otto M, Lange J, Haase M, Frommer J (2011) Intentionality in interacting with companion systems G an empirical approach. In: Human-Computer interaction. Towards mobile and intelligent interaction environments, LNCS, vol 6763. Springer, Berlin, Heidelberg, pp 593-602.
54. Russel J, Mehrabian A (1974) Distinguishing anger and anxiety in terms of emotional response factors. J Consult Clin Psychol 42: 79-83.
55. Russel JA (1980) Three dimensions of emotion. J Pers Soc Psychol 39(9): 1161-1178.
56. Sacharin V, Schlegel K, Scherer KR (2012) Geneva emotion wheel rating study. Center for Person, Kommunikation, Aalborg University, NCCR Affective Sciences, Technical report.
57. Scherer K (2005) What are emotions? and how can they be measured? Soc Sci Inform 44(4): 695-729.
58. Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking, vol 92. Oxford University Press, Oxford, pp. 92-120.
59. Schimmack U (1997) The Berlin everyday language mood inventory (BELMI): toward the content valid assessment of moods. Diagnostica 43(2): 150-173.
60. Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8(4): 350-353.
61. Schröder M, Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M (2000) Feeltrace: An instrument for recording perceived emotion in real time. In: Proceedings of ITRW on speech and, emotion, pp 19-24.
62. Sharp H, Rogers Y, Preece J (2007) Interaction design: beyond human-computer interaction, 2nd edn. Wiley, London.
63. Siegert I, Böck R, Wendemuth A (2013) The influence of context knowledge for multimodal affective annotation. In: Human-computer interaction, Part V, HCII 2013, LNCS, vol 8008. Springer, Berlin, pp 381-390.
64. Siegert I, Böck R, Philippou-Hübner D, Vlasenko B, Wendemuth A (2011) Appropriate emotional labeling of non-acted speech using basic emotions, Geneva emotion wheel and self assessment Manikins. In: Proceedings of ICME, s. p.
65. Siegert I, Böck R, Wendemuth A (2012) The influence of context knowledge for multimodal annotation on natural material. In: Joint proceedings of the IVA 2012 workshops, pp 25-32.
66. Sijtsma K (2009) On the use, the misuse, and the very limited usefulness of cronbachGs alpha. Psychometrika 74(1): 107-120.
67. Sojka P, Horak A, Kopecek I, Pala K (eds) (2012) Aggression detection in speech using sensor and semantic information, vol 7499. Springer, Berlin.
68. Truong KP, van Leeuwen DA, de Jong FM (2012) Speech-based recognition of self-reported and observed emotion in a dimensional space. Speech Commun 54(9): 1049-1063.
69. Truong KP, Neerincx MA, van Leeuwen DA (2008) Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data. In: Proceedings of interspeech, pp 318-321.
70. Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 54(6): 1063-1070.
71. Wendemuth A, Biundo S (2012) A companion technology for cognitive technical systems. In: Cognitive behavioural systems, Lecture Notes in Computer Science, vol 7403, Springer, Berlin, pp 89-103.
72. Wundt W (1922/1863) Vorlesungen über die Menschen- und Tierseele. L. Voss, Leipzig.
73. Yang YH, Lin YC, Su YF, Chen H (2007) Music emotion classification: a regression approach. In: Proceedings of ICME, pp 208-211.
74. Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31(1): 39-58.