The influence of annotation, corpus design, and evaluation on the outcome of automatic classification of human emotions

Frontiers in ICT

Volumn 3, Issue NOV, 2016, Pages

  Kächele, Markus ^a   Schels, Martin ^a   Schwenker, Friedhelm ^a  

^a UNIVERSITY OF ULM   (Germany)

Author keywords

Affective computing; Affective labeling; Human computer interaction; Machine guided labeling; Performance measures

Indexed keywords

EID: 85050468803     PISSN: None     EISSN: 2297198X     Source Type: Journal    
DOI: 10.3389/fict.2016.00027     Document Type: Article

References (67)

1. Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., and Christopherson, R. (2009). "Emotion sensors go to school," in Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling (Amsterdam: IOS Press), 17-24
2. Bishop, C. M. (1995). Neural Networks for Pattern Recognition. New York: Oxford University Press
3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. New York: Springer
4. Blum, A., and Mitchell, T. (1998). "Combining labeled and unlabeled data with co-training," in COLT: Proceedings of the Workshop on Computational Learning Theory (New York: Morgan Kaufmann Publishers), 92-100
5. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., and Weiss, B. (2005). "A database of German emotional speech," in INTERSPEECH'05 (Lisbon: ISCA), 1517-1520
6. Calvo, R., and D'Mello, S. (2010). Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 18-37. doi:10.1109/T-AFFC.2010.1
7. Celiktutan, O., Eyben, F., Sariyanidi, E., Gunes, H., and Schuller, B. (2014). "Maptraits 2014: the first audio/visual mapping personality traits challenge," in Proceedings of the 2014 Workshop on Mapping Personality Traits Challenge and Workshop, MAPTRAITS '14 (New York: ACM), 3-9
8. Chao, L., Tao, J., Yang, M., Li, Y., and Wen, Z. (2014). "Multi-scale temporal modeling for dimensional emotion recognition in video," in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC '14 (New York, NY: ACM), 11-18
9. Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., and Schröder, M. (2000). "'FEELTRACE': an instrument for recording perceived emotion in real time," in Proceedings of the ISCA Workshop on Speech and Emotion (Newcastle, Northern Ireland: ISCA), 19-24
10. Cowie, R., McKeown, G., and Douglas-Cowie, E. (2012). Tracing emotion: an overview. Int. J. Synth. Emot. 3, 1-17. doi:10.4018/jse.2012010101
11. D'Mello, S., and Calvo, R. A. (2013). "Beyond the basic emotions: what should affective computing compute?," in CHI '13 Extended Abstracts on Human Factors in Computing Systems, CHI EA '13 (New York: ACM), 2287-2294
12. D'Mello, S., Picard, R. W., and Graesser, A. (2007). Toward an affect-sensitive autotutor. IEEE Intell. Syst. 22, 53-61. doi:10.1109/MIS.2007.79
13. Dubois, D., and Prade, H. (1980). Fuzzy Sets and Systems: Theory and Applications. New York: Academic Press
14. Ekman, P., and Friesen, W. V. (1978). Facial Action Coding System (FACS): A Technique for the Measurement of Facial Action. Palo Alto: CPP
15. Girard, J. M. (2014). Carma: software for continuous affect rating and media annotation. J. Open Res. Softw. 2, e5. doi:10.5334/jors.ar
16. Glodek, M., Schels, M., Palm, G., and Schwenker, F. (2012). "Multiple classifier combination using reject options and Markov fusion networks," in Proceedings of the 14th ACM International Conference on Multimodal Interaction (New York: ACM), 465-472
17. Gnjatovic, M., and Rösner, D. (2008). "On the role of the NIMITEK corpus in developing an emotion adaptive spoken dialogue system," in Proc. of the 7th LREC (Marrakech, Morocco)
18. Graves, A., and Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18, 602-610. doi:10.1016/j.neunet.2005.06.042
19. Grimm, M., and Kroschel, K. (2005). "Evaluation of natural emotions using self assessment manikins," in IEEE Workshop on Automatic Speech Recognition and Understanding (Cancun: IEEE), 381-385
20. Gupta, R., Malandrakis, N., Xiao, B., Guha, T., Van Segbroeck, M., Black, M., et al. (2014). "Multimodal prediction of affective dimensions and depression in human-computer interactions," in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC '14 (New York, NY: ACM), 33-40
21. Hart, S. G. (2006). "Nasa-task load index (nasa-tlx); 20 years later," in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 50 (New York: Sage Publications), 904-908
22. Hassenzahl, M., Burmester, M., and Koller, F. (2003). "AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität," in Mensch & Computer 2003 (Wiesbaden: Springer), 187-196
23. Kächele, M., Schels, M., and Schwenker, F. (2014). "Inferring depression and affect from application dependent meta knowledge," in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC '14 (New York: ACM), 41-48
24. Kanade, T., Cohn, J., and Tian, Y. (2000). "Comprehensive database for facial expression analysis," in Automatic Face and Gesture Recognition, 2000 (Washington: IEEE), 46-53
25. Kelley, J. F. (1983). "An empirical methodology for writing user-friendly natural language computer applications," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '83 (New York: ACM), 193-196
26. Kessler, V., Schels, M., Kächele, M., Palm, G., and Schwenker, F. (2015). "On the effects of continuous annotation tools and the human factor on the annotation outcome," in Proceedings of the International Symposium on Companion Technology (Ulm: Ulm University Open Access Repository), 17
27. Kipp, M. (2001). "Anvil-a generic annotation tool for multimodal dialogue," in Proc. of 7th European Conference on Speech Communication and Technology (Eurospeech) (Aalborg: ISCA), 1367-1370
28. Kuncheva, L. (2004). Combining Pattern Classifiers: Methods and Algorithms. Hoboken, NJ: Wiley
29. Lang, P. J., Bradley, M. M., and Cuthbert, B. N. (1999). International Affective Picture System (IAPS): Technical Manual and Affective Ratings. Technical report. Gainesville: Center for Research in Psychophysiology, Gainesville, University of Florida
30. Lang, P. J., Bradley, M. M., and Cuthbert, B. N. (2005). International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual. Technical Report A-6. Gainesville, FL: University of Florida
31. Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45, 255-268. doi:10.2307/2532051
32. Madsen, M., and Gregor, S. (2000). "Measuring human-computer trust," in Proceedings of Eleventh Australasian Conference on Information Systems (Brisbane: ACS), 6-8
33. McCallum, A., and Nigam, K. (1998). "A comparison of event models for naive Bayes text classification," in AAAI-98 Workshop on Learning for Text Categorization (AAAI Press), 41-48
34. McKeown, G., Valstar, M. F., Cowie, R., and Pantic, M. (2010). "The SEMAINE corpus of emotionally coloured character interactions," in IEEE International Conference on Multimedia and Expo (ICME) (Singapore: IEEE), 1079-1084
35. Meudt, S., Bigalke, L., and Schwenker, F. (2012). "Atlas-an annotation tool for HCI data utilizing machine learning methods," in Proceedings of the 1st International Conference on Affective and Pleasurable Design (APD'12) (Boca Raton, FL: CRC Pess), 5347-5352
36. Palm, G., and Glodek, M. (2013). "Towards emotion recognition in human computer interaction," in Neural Nets and Surroundings, Vol. 19, eds B. Apolloni, B. Simone, A. Esposito, and F. C. Morabito (Berlin: Springer), 323-336
37. Picard, R. W. (2000). Affective Computing. Cambridge: MIT Press
38. Prylipko, D., Rösner, D., Siegert, I., Gnther, S., Friesen, R., Haase, M., et al. (2014). Analysis of significant dialog events in realistic human-computer interaction. J. Multimodal User Interfaces 8, 75-86. doi:10.1007/s12193-013-0144-x
39. Ringeval, F., Schuller, B., Valstar, M., Jaiswal, S., Marchi, E., Lalanne, D., et al. (2015). "AV+EC 2015-the first affect recognition challenge bridging across audio, video, and physiological data," in Proceedings of AVEC (New York: ACM), 3-8
40. Ringeval, F., Sonderegger, A., Sauer, J., and Lalanne, D. (2013). "Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions," in Proceedings of Face & Gestures 2013, 2nd IEEE International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space (EmoSPACE) (Shanghai: IEEE), 1-8
41. Rösner, D., Frommer, J., Friesen, R., Haase, M., Lange, J., and Otto, M. (2012). "LAST MINUTE: a multimodal corpus of speech-based user-companion interactions," in Proc. of LREC (Istanbul: ELRA), 2559-2566
42. Rukavina, S., Gruss, S., Walter, S., Hoffmann, H., and Traue, H. C. (2015). "Open_emorec II-a multimodal corpus of human-computer interaction," in Proceedings of ICACII (Berlin: Springer), 356-367
43. Russell, J. A., and Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. J. Res. Pers. 11, 273-294. doi:10.1016/0092-6566(77)90037-X
44. Schels, M., Kächele, M., Glodek, M., Hrabal, D., Walter, S., and Schwenker, F. (2014). Using unlabeled data to improve classification of emotional states in human computer interaction. J. Multimodal User Interfaces 8, 5-16. doi:10.1007/s12193-013-0133-0
45. Schölkopf, B., Smola, A. J., Williamson, R. C., and Bartlett, P. L. (2000). New support vector algorithms. Neural Comput. 12, 1207-1245. doi:10.1162/089976600300015565
46. Schuller, B., Müller, R., Eyben, F., Gast, J., Hörnler, B., Wöllmer, M., et al. (2009). Being bored? Recognising natural interest by extensive audiovisual integration for real-life application. Image Vision Comput. 27, 1760-1774. doi:10.1016/j.imavis.2009.02.013
47. Schuller, B., Valstar, M., Cowie, R., and Pantic, M. (2012). "Avec 2012: the continuous audio/visual emotion challenge-an introduction," in Proc. of the 14th ACM ICMI'12 (New York: ACM), 361-362
48. Schuller, B., Valstar, M., Eyben, F., McKeown, G., Cowie, R., and Pantic, M. (2011). "AVEC 2011-the first international audio visual emotion challenges," in Proceedings of ACII (2011), volume 6975 of LNCS, Part II (Berlin: Springer), 415-424
49. Schüssel, F., Honold, F., Schmidt, M., Bubalo, N., Huckauf, A., and Weber, M. (2014). "Multimodal interaction history and its use in error detection and recovery," in Proceedings of the 16th International Conference on Multimodal Interaction, ICMI '14 (New York: ACM), 164-171
50. Schwenker, F., and Trentin, E. (2014). Pattern classification and clustering: a review of partially supervised learning approaches. Pattern Recognit. Lett. 37, 4-14. doi:10.1016/j.patrec.2013.10.017
51. Stemmler, G. (1989). The autonomic differentiation of emotions revisited: convergent and discriminant validation. Psychophysiology 26, 617-632. doi:10.1111/j.1469-8986.1989.tb03163.x
52. Strauss, P.-M., Hoffmann, H., Minker, W., Neumann, H., Palm, G., Scherer, S., et al. (2008). "The PIT corpus of German multi-party dialogues," in Proc. of LREC (Marrakech: ELRA), 2442-2445
53. Strauß, P.-M., Hoffmann, H., Minker, W., Neumann, H., Palm, G., Scherer, S., et al. (2006). "Wizard-of-oz data collection for perception and interaction in multi-user environments," in Proceedings of LREC (Genoa: ELRA), 2014-2017
54. Tax, D. M. J., and Duin, R. P. W. (1999). Support vector domain description. Pattern Recognit. Lett. 20, 1191-1199. doi:10.1016/S0167-8655(99)00087-2
55. Theodoridis, S., and Koutroumbas, K. (2009). Pattern Recognition, 4th Edn. Burlington, MA: Academic Press
56. Thiam, P., Meudt, S., Kächele, M., Palm, G., and Schwenker, F. (2014). "Detection of emotional events utilizing support vector methods in an active learning HCI scenario," in Proceedings of the 2014 Workshop on Emotion Representations and Modelling for HCI Systems, ERM4HCI '14 (New York: ACM), 31-36
57. Valstar, M., Schuller, B., Smith, K., Almaev, T., Eyben, F., Krajewski, J., et al. (2014a). "Avec 2014: 3d dimensional affect and depression recognition challenge," in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC '14 (New York: ACM), 3-10
58. Valstar, M., Schuller, B., Smith, K., Almaev, T., Eyben, F., Krajewski, J., et al. (2014b). "AVEC 2014: 3D dimensional affect and depression recognition challenge," in Proceedings of ACM Multimedia 2014 (New York: ACM), 3-10
59. Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., et al. (2013). "Avec 2013: the continuous audio/visual emotion and depression recognition challenge," in Proceedings of AVEC 2013, AVEC '13 (New York: ACM), 3-10
60. Walter, S., Kim, J., Hrabal, D., Crawcour, S., Kessler, H., and Traue, H. (2013a). Transsituational individual-specific biopsychological classification of emotions. IEEE Trans. Syst. Man Cybern. 43, 988-995. doi:10.1109/TSMCA.2012.2216869
61. Walter, S., Wendt, C., Bhnke, J., Crawour, S., Tan, J., Chan, A., et al. (2013b). Similarities and differences of emotions in human-machine and human-human interaction: what kind of emotions are relevant for future companion systems. Ergonomics 57, 374-386. doi:10.1080/00140139.2013.822566
62. Walter, S., Scherer, S., Schels, M., Glodek, M., Hrabal, D., Schmidt, M., et al. (2011). "Multimodal emotion classification in naturalistic user behavior," in Human-Computer Interaction. Towards Mobile and Intelligent Interaction Environments, Volume 6763 of LNCS, ed. J. Jacko (Berlin, Heidelberg: Springer), 603-611
63. Williamson, J. R., Quatieri, T. F., Helfer, B. S., Horwitz, R., Yu, B., and Mehta, D. D. (2013). "Vocal biomarkers of depression based on motor incoordination," in Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge, AVEC '13 (New York: ACM), 41-48
64. Wu, C.-H., Lin, J.-C., and Wei, W.-L. (2014). Survey on audiovisual emotion recognition: databases, features, and data fusion strategies. APSIPA Trans. Signal Inform. Process. 3, e12. doi:10.1017/ATSIP.2014.11
65. Yarowski, D. (1995). "Unsupervised word sense disambiguation rivaling supervised methods," in Proceedings Thirty-Third Meeting of the ACL (Stroudsburg: ACL), 189-196
66. Zadeh, L. A. (1965). Fuzzy sets. Inform. Control 8, 338-353. doi:10.1016/S0019-9958(65)90241-X
67. Zeng, Z., Pantic, M., Roisman, G., and Huang, T. (2009). A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39-58. doi:10.1109/TPAMI.2008.52