Continuous, Real-Time Emotion Annotation: A Novel Joystick-Based Analysis Framework

IEEE Transactions on Affective Computing

Volumn 11, Issue 1, 2020, Pages 78-84

  Sharma, Karan ^a   Castellini, Claudio ^a   Stulp, Freek ^a   Van Den Broek, Egon L ^b  

^a GERMAN AEROSPACE CENTER DLR   (Germany)

^b UTRECHT UNIVERSITY   (Netherlands)

Author keywords

Affective computing; change point analysis; emotion in human computer interaction; pattern recognition; time series analysis; tools and methods of annotation

Indexed keywords

HUMAN COMPUTER INTERACTION; INTERACTIVE COMPUTER SYSTEMS; PATTERN RECOGNITION; PATTERN RECOGNITION SYSTEMS; REAL TIME SYSTEMS; STANDARDS; SUPPORT VECTOR MACHINES; SURVEYS; TOOLS; USABILITY ENGINEERING;

AFFECTIVE COMPUTING; CHANGE-POINT ANALYSIS; TOOLS AND METHODS; TWO-DIMENSIONAL DISPLAYS; USABILITY; VIDEOS;

TIME SERIES ANALYSIS;

EID: 85034235402     PISSN: None     EISSN: 19493045     Source Type: Journal    
DOI: 10.1109/TAFFC.2017.2772882     Document Type: Article

References (36)

1. P. Rani, N. Sarkar, C. A. Smith, and L. D. Kirby, "Anxiety detecting robotic system-towards implicit human-robot collaboration, " Robotica, vol. 22, pp. 85-95, 1 2004.
2. E. L. van den Broek, V. Lisy, J. H. Janssen, J. H. D. M. Westerink, M. H. Schut, and K. Tuinenbreijer, Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals. Berlin, Germany: Springer, 2010, pp. 21-47.
3. E. L. van den Broek, "Affective Signal Processing (ASP): Unraveling the mystery of emotions, " Ph.D. dissertation, Human Media Interaction (HMI), Faculty Electr. Eng., Math., Comput. Sci., Univ. Twente, Enschede, The Netherlands, 2011.
4. F. Nagel, R. Kopiez, O. Grewe, and E. Altenmueller, "Emujoy: Software for continuous measurement of perceived emotions in music, " Behavior Res. Methods, vol. 39, no. 2, pp. 283-290, 2007.
5. M. Soleymani, S. Asghari Esfeden, Y. Fu, and M. Pantic, "Analysis of EEG signals and facial expressions for continuous emotion detection, " IEEE Trans. Affective Comput., vol. 7, no. 1, pp. 17-28, Jan.-Mar. 2016.
6. M. E. Ayadi, M. S. Kamel, and F. Karray, "Survey on speech emotion recognition: Features, classification schemes, and databases, " Pattern Recognit., vol. 44, no. 3, pp. 572-587, 2011.
7. M. Soleymani, S. Koelstra, I. Patras, and T. Pun, "Continuous emotion detection in response to music videos, " in Proc. IEEE Int. Conf. Face Gesture Recognit. Workshops, Mar. 2011, pp. 803-808.
8. J. A. Russell, "Core affect and the psychological construction of emotion, " Psychological Rev., vol. 110, pp. 145-172, 2003.
9. D. S. Messinger, T. D. Cassel, S. I. Acosta, Z. Ambadar, and J. F. Cohn, "Infant smiling dynamics and perceived positive emotion, " J. Nonverbal Behavior, vol. 32, no. 3, 2008, Art. no. 133.
10. R. Cowie, M. Sawey, C. Doherty, J. Jaimovich, C. Fyans, and P. Stapleton, "Gtrace: General Trace Program Compatible with EmotionML, " Humaine Assoc. Conf. Affective Comput. Intell. Interaction, pp. 709-710, Sep. 2013, doi: 10.1109/ACII.2013.126
11. J. Girard, "Carma: Software for continuous affect rating and media annotation, " J. Open Res. Softw., vol. 2, no. 1, p. e5, 2014.
12. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schröder, "'feeltrace': An instrument for recording perceived emotion in real time, " in Proc. ISCA Tutorial Res. Workshop (ITRW) Speech Emotion, 2000, pp. 19-24.
13. J. M. Girard and A. G. C. Wright, "DARMA: Software for dual axis rating and media annotation, " Behavior Res. Methods, pp. 1-8, Jun. 2017 doi: 10.3758/s13428-017-0915-5
14. N. Malandrakis, A. Potamianos, G. Evangelopoulos, and A. Zlatintsi, "A supervised approach to movie emotion tracking, " in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2011, pp. 2376-2379.
15. Y. Baveye, E. Dellandréa, C. Chamaret, and L. Chen, "Deep learning versus kernel methods: Performance for emotion prediction in videos, " in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2015, pp. 77-83.
16. G. N. Yannakakis and H. P. Martnez, "Grounding truth via ordinal annotation, " in Proc. Int. Conf. Affective Comput. Intell. Interaction, Sep. 2015, pp. 574-580.
17. A. Metallinou and S. Narayanan, "Annotation and processing of continuous emotional attributes: Challenges and opportunities, " in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recognit., Apr. 2013, pp. 1-8.
18. H. Egermann, F. Nagel, E. Altenmüller, and R. Kopiez, "Continuous measurement of musically-induced emotion: A web experiment, " Int. J. Internet Sci., vol. 4, no. 1, pp. 4-20, 2009.
19. M. A. Nicolaou, Y. Panagakis, S. Zafeiriou, and M. Pantic, "Robust canonical correlation analysis: Audio-visual fusion for learning continuous interest, " in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2014, pp. 1522-1526.
20. O. Grewe, F. Nagel, R. Kopiez, and E. Altenmüller, "Emotions over time: Synchronicity and development of subjective, physiological, and facial affective reactions to music, " Emotion, vol. 7, no. 4, 2007, Art. no. 774.
21. E. Dellandréa, et al., "The MediaEval 2016 emotional impact of movies task, " in Proc. MediaEval Multimedia Benchmark Workshop Working Notes Proc. MediaEval Workshop, 2016.
22. J. Antony, K. Sharma, E. L. van den Broek, C. Castellini, and C. Borst, "Continuous affect state annotation using a joystick-based user interface, " in Proc. Measuring Behavior 9th Int. Conf. Methods Techn. Behavioral Res., 2014, pp. 268-271.
23. J. J. Gross and R. W. Levenson, "Emotion elicitation using films, " Cognition Emotion, vol. 9, no. 1, pp. 87-108, 1995.
24. J. Hewig, D. Hagemann, J. Seifert, M. Gollwitzer, E. Naumann, and D. Bartussek, "A revised film set for the induction of basic emotions, " Cognition Emotion, vol. 19, no. 7, pp. 1095-1109, 2005.
25. E. E. Bartolini, "Eliciting emotion with film: Development of a stimulus set, " Master's thesis, Wesleyan University, Middletown, CT, USA, 2011.
26. C. Elliott, V. Vijayakumar, W. Zink, and R. Hansen, "National instruments labview: A programming environment for laboratory automation and measurement, " JALA: J. Assoc. Laboratory Autom., vol. 12, no. 1, pp. 17-24, 2007.
27. M. M. Bradley and P. J. Lang, "Measuring emotion: The self-assessment manikin and the semantic differential, " J. Behavior Therapy Experimental Psychiatry, vol. 25, no. 1, pp. 49-59, 1994.
28. I. D. Loram, H. Gollee, M. Lakie, and P. J. Gawthrop, "Human control of an inverted pendulum: Is continuous control necessary? is intermittent control effective? is intermittent control physiological?" J. Physiology, vol. 589, no. 2, pp. 307-324, 2011.
29. J. Sauro and J. R. Lewis, "When designing usability questionnaires, does it hurt to be positive?" in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2011, pp. 2215-2224.
30. S. Mariooryad and C. Busso, "Correcting time-continuous emotional labels by modeling the reaction lag of evaluators, " IEEE Trans. Affective Comput., vol. 6, no. 2, pp. 97-108, Apr.-Jun. 2015.
31. W. L. Martinez and A. R. Martinez, Computational Statistics Handbook with MATLAB, 3rd ed. Boca Raton, FL, USA: CRC press, 2007.
32. C. C. Aggarwal, Data Mining: The Textbook. Berlin, Germany: Springer, 2015.
33. A. Gabadinho, G. Ritschard, N. S. Mueller, and M. Studer, "Analyzing and visualizing state sequences in r with traminer, " J. Statistical Softw., vol. 40, no. 4, pp. 1-37, 2011, iD: unige:16809.
34. D. S. Matteson and N. A. James, "A nonparametric approach for multiple change point analysis of multivariate data, " J. Amer. Statistical Assoc., vol. 109, no. 505, pp. 334-345, 2014.
35. N. A. James, A. Kejariwal, and D. S. Matteson, "Leveraging cloud data to mitigate user experience from breaking bad, " IEEE Int. Conf. Big Data, pp. 3499-3508, Dec. 2016, doi: 10.1109/BigData.2016.7841013
36. C. J. C. Burges, "A tutorial on support vector machines for pattern recognition, " Knowl. Discovery Data Mining, vol. 2, no. 2, pp. 121-167, 1998.