A functional data analysis approach for continuous 2-D emotion annotations

Web Intelligence

Volumn 17, Issue 1, 2019, Pages 41-52

  Sharma, Karan ^a   Wagner, Marius ^a   Castellini, Claudio ^a   Van Den Broek, Egon L ^b   Stulp, Freek ^a   Schwenker, Friedhelm ^c  

^a GERMAN AEROSPACE CENTER DLR   (Germany)

^b UTRECHT UNIVERSITY   (Netherlands)

^c UNIVERSITY OF ULM   (Germany)

Author keywords

affective computing; continuous annotation; Functional data analysis; self reporting

Indexed keywords

DATA HANDLING; INFORMATION ANALYSIS; MULTIVARIANT ANALYSIS; PHYSIOLOGICAL MODELS; PRINCIPAL COMPONENT ANALYSIS; TIME SERIES ANALYSIS;

AFFECTIVE COMPUTING; CONTINUOUS ANNOTATION; EMOTIONAL EXPERIENCES; FUNCTIONAL ANNOTATION; FUNCTIONAL DATA ANALYSIS; FUNCTIONAL PRINCIPAL COMPONENT ANALYSIS; MULTIVARIATE STATISTICAL TECHNIQUES; SELF-REPORTING;

QUALITY CONTROL;

EID: 85062167067     PISSN: 24056456     EISSN: 24056464     Source Type: Journal    
DOI: 10.3233/WEB-190399     Document Type: Article

References (43)

1. J. Antony, K. Sharma, E.L. van den Broek, C. Castellini and C. Borst, Continuous affect state annotation using a joystick-based user interface, in: Proceedings of Measuring Behavior 2014: 9th International Conference on Methods and Techniques in Behavioral Research, 2014, pp. 268-271. doi: 10.13140/2.1.2507.3929.
2. M.E. Ayadi, M.S. Kamel and F. Karray, Survey on speech emotion recognition: Features, classification schemes, and databases, Pattern Recognition 44(3) (2011), 572-587. doi: 10.1016/j.patcog.2010.09.020.
3. Y. Baveye, E. Dellandréa, C. Chamaret and L. Chen, Deep learning vs. kernel methods: Performance for emotion prediction in videos, in: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE, 2015, pp. 77-83. doi: 10.1109/ACII.2015.7344554.
4. M.M. Bradley and P.J. Lang, Measuring emotion: The self-assessment manikin and the semantic differential, Journal of behavior therapy and experimental psychiatry 25(1) (1994), 49-59. doi: 10.1016/0005-7916(94)90063-9.
5. C.J.C. Burges, A tutorial on Support Vector Machines for pattern recognition, Knowledge Discovery and Data Mining 2(2) (1998). doi: 10.1023/A:1009715923555.
6. C.-C. Chang and C.-J. Lin, LIBSVM: A library for support vector machines, ACM Transactions on Intelligent Systems and Technology 2 (2011), 27:1-27:27.
7. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey and M. Schröder, 'FEELTRACE': An instrument for recording perceived emotion in real time, in: ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000, pp. 19-24.
8. R. Cowie, M. Sawey, C. Doherty, J. Jaimovich, C. Fyans and P. Stapleton, GTrace: General trace program compatible with EmotionML, in: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE, 2013, pp. 709-710. doi: 10.1109/ACII.2013.126.
9. C. De Boor, A Practical Guide to Splines, Applied Mathematical Sciences, Springer, Berlin, 2001.
10. L. Fahrmeir, A. Hamerle and G. Tutz, Multivariate Statistische Verfahren, Walter de Gruyter GmbH & Co KG, 1996.
11. L. Fahrmeir, T. Kneib, S. Lang and B. Marx, Regression: Models, Methods and Applications, Springer Science & Business Media, 2013.
12. P. Filzmoser, R.G. Garrett and C. Reimann, Multivariate outlier detection in exploration geochemistry, Computers & Geosciences 31(5) (2005), 579-587. doi: 10.1016/j.cageo.2004.11.013.
13. P. Filzmoser and M. Gschwandtner, mvoutlier: Multivariate outlier detection based on robust methods, R package version 2.0.6, 2015.
14. O. Grewe, F. Nagel, R. Kopiez and E. Altenmüller, Emotions over time: Synchronicity and development of subjective, physiological, and facial affective reactions to music, Emotion 7(4) (2007), 774. doi: 10.1037/1528-3542.7.4.774.
15. C. Happ, funData: An S4 class for functional data, R package version 1.0, 2016.
16. C. Happ and S. Greven, Multivariate functional principal component analysis for data observed on different (dimensional) domains, Journal of the American Statistical Association 113(522) (2018), 649-659.
17. T. Hastie, R. Tibshirani and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2nd edn, Springer, 2008.
18. C.-W. Hsu and C.-J. Lin, A comparison of methods for multiclass support vector machines, IEEE Transactions on Neural Networks 13(2) (2002), 415-425. doi: 10.1109/72.991427.
19. I.T. Jolliffe and J. Cadima, Principal component analysis: A review and recent developments, Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 374(2065) (2016). doi: 10.1098/rsta.2015.0202.
20. M. Kächele, M. Schels and F. Schwenker, The influence of annotation, corpus design, and evaluation on the outcome of automatic classification of human emotions, Front. ICT 3 (2016), 27.
21. M. Kächele, P. Thiam, M. Amirian, F. Schwenker and G. Palm, Methods for person-centered continuous pain intensity assessment from bio-physiological channels, IEEE Journal of Selected Topics in Signal Processing 10(5) (2016), 854-864. doi: 10.1109/JSTSP.2016.2535962.
22. K. Karhunen, Über lineare Methoden in der Wahrscheinlichkeitsrechnung, Annales Academiae Scientarum Fennicae 37 (1947), 3-79.
23. N. Malandrakis, A. Potamianos, G. Evangelopoulos and A. Zlatintsi, A supervised approach to movie emotion tracking, in: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 2376-2379. doi: 10.1109/ICASSP.2011.5946961.
24. A. Metallinou and S. Narayanan, Annotation and processing of continuous emotional attributes: Challenges and opportunities, in: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), IEEE, 2013, pp. 1-8.
25. F. Nagel, R. Kopiez, O. Grewe and E. Altenmueller, Emujoy: Software for continuous measurement of perceived emotions in music, Behavior Research Methods 39(2) (2007), 283-290. doi: 10.3758/BF03193159.
26. R.W. Picard, Affective Computing, MIT Press, Cambridge, MA, USA, 1997.
27. J. Ramsay and B. Silverman, Functional Data Analysis, Springer Science & Business Media, 2005.
28. V.C. Raykar, S. Yu, L.H. Zhao, G.H. Valadez, C. Florin, L. Bogoni and L. Moy, Learning from crowds, Journal of Machine Learning Research 11 (2010), 1297-1322.
29. A.M. Ruef and R.W. Levenson, Continuous measurement of emotion, in: Handbook of Emotion Elicitation and Assessment, 2007, pp. 286-297.
30. J.A. Russell, Core affect and the psychological construction of emotion, Psychological Review 110(1) (2003), 145. doi: 10.1037/0033-295X.110.1.145.
31. F. Schwenker, R. Böck, M. Schels, S. Meudt, I. Siegert, M. Glodek, M. Kächele, M. Schmidt-Wack, P. Thiam, A. Wendemuth and G. Krell, Multimodal affect recognition in the context of human-computer interaction for companion-systems, in: Companion Technology-A Paradigm Shift in Human-Technology Interaction, 2017, pp. 387-408.
32. F. Schwenker and S. Scherer (eds), Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction-4th IAPR TC 9 Workshop, MPRSS 2016, Cancun, Mexico, December 4, 2016, Revised Selected Papers, Lecture Notes in Computer Science, Vol. 10183, Springer, 2017. doi: 10.1007/978-3-319-59259-6.
33. K. Sharma, C. Castellini, F. Stulp and E.L.V. den Broek, Continuous, real-time emotion annotation: A novel joystick-based analysis framework, IEEE Transactions on Affective Computing (2018), 1-1. doi: 10.1109/TAFFC.2017.2772882.
34. K. Sharma, C. Castellini and E.L. van den Broek, Continuous affect state annotation using a joystick-based user interface: Exploratory data analysis, in: Measuring Behavior 2016: 10th International Conference on Methods and Techniques in Behavioral Research, 2016. doi: 10.13140/RG.2.1.2285.7841.
35. I. Sneddon, G. McKeown, M. McRorie and T. Vukicevic, Cross-cultural patterns in dynamic ratings of positive and negative natural emotional behaviour, PloS One 6(2) (2011), e14679. doi: 10.1371/journal.pone.0014679.
36. M. Soleymani, S. Koelstra, I. Patras and T. Pun, Continuous emotion detection in response to music videos, in: Face and Gesture 2011, 2011, pp. 803-808. doi: 10.1109/FG.2011.5771352.
37. M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie and M. Pantic, Avec 2014: 3d dimensional affect and depression recognition challenge, in: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC '14, ACM, New York, NY, USA, 2014, pp. 3-10.
38. E.L. van den Broek, Monitoring Technology: The 21st Century's Pursuit of Well-being? Eu-osha Discussion Paper, European Agency for Safety and Health at Work (EUOSHA), Bilbao, Spain, 6 July 2017.
39. E.L. van den Broek, J.H. Janssen and J.H.D.M. Westerink, Autonomous closed-loop biofeedback: An introduction and a melodious application, in: The Oxford Handbook of Affective Computing, Chapter 35 (Section 5: Applications of Affective Computing), Oxford Library of Psychology, Oxford University Press, Inc., New York, NY, USA, 2015, pp. 472-482.
40. E.L. van den Broek, V. Lisý, J.H. Janssen, J.H.D.M. Westerink, M.H. Schut and K. Tuinenbreijer, Affective Man-Machine Interface: Unveiling Human Emotions Through Biosignals, Springer, Berlin, Heidelberg, 2010, pp. 21-47.
41. C. Vondrick, D. Patterson and D. Ramanan, Efficiently scaling up crowdsourced video annotation, International Journal of Computer Vision 101(1) (2013), 184-204. doi: 10.1007/s11263-012-0564-1.
42. J.-L. Wang, J.-M. Chiou and H.-G. Müller, Functional data analysis, Annual Review of Statistics and Its Application 3 (2016), 257-295. doi: 10.1146/annurev-statistics-041715-033624.
43. G.N. Yannakakis and H.P. Martinez, Grounding truth via ordinal annotation, in: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), 2015, pp. 574-580. doi: 10.1109/ACII.2015.7344627.