Multimodal emotion recognition in response to videos (Extended abstract)

2015 International Conference on Affective Computing and Intelligent Interaction, ACII 2015

Volumn , Issue , 2015, Pages 491-497

  Soleymani, Mohammad ^a   Pantic, Maja ^b   Pun, Thierry ^a  

^a UNIVERSITY OF GENEVA   (Switzerland)

^b UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

EEG signals; emotions; implicit tagging

Indexed keywords

ELECTROENCEPHALOGRAPHY; IMAGE RETRIEVAL; SPEECH RECOGNITION; VIDEO CAMERAS;

CLASSIFICATION ACCURACY; CLASSIFICATION PERFORMANCE; EEG SIGNALS; ELECTRO-ENCEPHALOGRAM (EEG); EMOTION RECOGNITION; EMOTIONS; IMPLICIT TAGGING; MULTIMODAL EMOTION RECOGNITION;

INTELLIGENT COMPUTING;

EID: 84964034554     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ACII.2015.7344615     Document Type: Conference Paper

References (41)

1. J. Kim and E. André, "Emotion Recognition Based on Physiological Changes in Music Listening," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 30, no. 12, pp. 2067-2083, 2008.
2. J. Rottenberg, R. D. Ray, and J. J. Gross, Emotion elicitation using films, ser. Series in affective science. Oxford University Press, 2007, pp. 9-28.
3. M. Soleymani, G. Chanel, J. J. M. Kierkels, and T. Pun, "Affective Characterization of Movie Scenes Based on Content Analysis and Physiological Changes," International Journal of Semantic Computing, vol. 3, no. 2, pp. 235-254, June 2009.
4. R. W. Picard and S. B. Daily, "Evaluating Affective Interactions: Alternatives to Asking What Users Feel," in CHI Workshop on Evaluating Affective Interfaces: Innovative Approaches, 2005.
5. K. R. Scherer, "What are emotions? And how can they be measured?" Social Science Information, vol. 44, no. 4, pp. 695-729, December 2005.
6. A. Hanjalic and L.-Q. Xu, "Affective video content representation and modeling," Multimedia, IEEE Transactions on, vol. 7, no. 1, pp. 143-154, 2005.
7. M. Tkalčič, U. Burnik, and A. Košir, "Using affective parameters in a content-based recommender system for images," User Modeling and User-Adapted Interaction, pp. 1-33-33, September 2010.
8. M. K. Shan, F. F. Kuo, M. F. Chiang, and S. Y. Lee, "Emotion-based music recommendation by affinity discovery from film music," Expert Syst. Appl., vol. 36, no. 4, pp. 7666-7674, September 2009.
9. H. Joho, J. M. Jose, R. Valenti, and N. Sebe, "Exploiting facial expressions for affective video summarisation," in Proceeding of the ACM International Conference on Image and Video Retrieval, ser. CIVR '09. New York, NY, USA: ACM, 2009.
10. C. L. Lisetti and F. Nasoz, "Using noninvasive wearable computers to recognize human emotions from physiological signals," EURASIP J. Appl. Signal Process., vol. 2004, no. 1, pp. 1672-1687, January 2004.
11. K. Takahashi, "Remarks on Emotion Recognition from BioPotential Signals," in in 2nd Int. Conf. on Autonomous Robots and Agents, 2004, 2005.
12. S. Koelstra, A. Yazdani, M. Soleymani, C. Muhl, J.-S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, "Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos," in Brain Informatics, ser. Lecture Notes in Computer Science, Y. Yao, R. Sun, T. Poggio, J. Liu, N. Zhong, and J. Huang, Eds. Berlin, Heidelberg: Springer, 2010, vol. 6334, ch. 9, pp. 89-100.
13. H. Joho, J. Staiano, N. Sebe, and J. Jose, "Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents," Multimedia Tools and Applications, pp. 1-19, October 2010.
14. I. Arapakis, I. Konstas, and J. M. Jose, "Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance," in Proceedings of the seventeen ACM international conference on Multimedia, ser. MM '09. New York, NY, USA: ACM, 2009, pp. 461-470.
15. P. Lang, M. Bradley, and B. Cuthbert, "International affective picture system (IAPS): Affective ratings of pictures and instruction manual," University of Florida, Gainesville, Florida, US, Tech. Rep. A-8, 2008.
16. P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, "Looking at pictures: Affective, facial, visceral, and behavioral reactions," Psy-chophysiology, vol. 30, no. 3, pp. 261-273, 1993.
17. J. Wang and Y. Gong, "Recognition of multiple drivers' emotional state," in In ICPR'08: The 19th International Conference on Pattern Recognition, 2008.
18. J. A. Healey, "Wearable and automotive systems for affect recognition from physiology," Ph.D. dissertation, MIT, 2000.
19. G. Chanel, J. J. M. Kierkels, M. Soleymani, and T. Pun, "Short-term emotion assessment in a recall paradigm," International Journal of Human-Computer Studies, vol. 67, no. 8, pp. 607-627, August 2009.
20. V. Kolodyazhniy, S. D. Kreibig, J. J. Gross, W. T. Roth, and F. H. Wilhelm, "An affective computing approach to physiological emotion specificity: Toward subject-independent and stimulus-independent classification of film-induced emotions," Psychophysiology, vol. 7, no. 48, pp. 908-922, 2011.
21. Bradley, M. Margaret, Miccoli, Laura, Escrig, A. Miguel, Lang, and J. Peter, "The pupil as a measure of emotional arousal and autonomic activation," Psychophysiology, vol. 45, no. 4, pp. 602-607, July 2008.
22. T. Partala and V. Surakka, "Pupil size variation as an indication of affective processing," International Journal of Human-Computer Studies, vol. 59, no. 1-2, 2003.
23. Y. Gao, A. Barreto, and M. Adjouadi, "Monitoring and processing of the pupil diameter signal for affective assessment of a computer user," in Proceedings of the 13th International Conference on Human-Computer Interaction. Part I: New Trends. Berlin, Heidelberg: Springer-Verlag, 2009, pp. 49-58.
24. J. J. M. Kierkels, M. Soleymani, and T. Pun, "Queries and tags in affectbased multimedia retrieval," in ICME'09: Proceedings of the 2009 IEEE international conference on Multimedia and Expo. Piscataway, NJ, USA: IEEE Press, 2009, pp. 1436-1439.
25. J. A. Russell, "Culture and the Categorization of Emotions," Psycholog-ical Bulletin, vol. 110, no. 3, pp. 426-450, 1991.
26. J. A. Russell and A. Mehrabian, "Evidence for a three-factor theory of emotions," Journal of Research in Personality, vol. 11, no. 3, pp. 273-294, September 1977.
27. M. Soleymani, J. Davis, and T. Pun, "A collaborative personalized affective video retrieval system," in Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Confer-ence on, sep 2009.
28. A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, "Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers," Cognition &Emotion, vol. 24, no. 7, pp. 1153-1172, 2010.
29. J. Lichtenauer, M. Valstar, J. Shen, and M. Pantic, "Cost-Effective Solution to Synchronized Audio-Visual Capture Using Multiple Sensors," in AVSS '09: Proceedings of the 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance. Washington, DC, USA: IEEE Computer Society, 2009, pp. 324-329.
30. R. Adolphs, D. Tranel, and A. R. Damasio, "Dissociable neural systems for recognizing emotions," Brain and Cognition, vol. 52, no. 1, pp. 61-69, June 2003.
31. A. R. Damasio, T. J. Grabowski, A. Bechara, H. Damasio, L. L. B. Ponto, J. Parvizi, and R. D. Hichwa, "Subcortical and cortical brain activity during the feeling of self-generated emotions," Nature Neuro-science, vol. 3, no. 10, pp. 1049-1056, October 2000.
32. R. J. Davidson, "Affective neuroscience and psychophysiology: toward a synthesis." Psychophysiology, vol. 40, no. 5, pp. 655-665, September 2003.
33. L. I. Aftanas, N. V. Reva, A. A. Varlamov, S. V. Pavlov, and V. P. Makhnev, "Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics." Neuroscience and behavioral physiology, vol. 34, no. 8, pp. 859-867, October 2004.
34. P. D. Welch, "The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms," IEEE Transactions on Audio and Electroacoustics, vol. 15, pp. 70-73, 1967.
35. S. K. Sutton and R. J. Davidson, "Prefrontal Brain Asymmetry: A Biological Substrate of the Behavioral Approach and Inhibition Systems," Psychological Science, vol. 8, no. 3, pp. 204-210, 1997.
36. V. F. Pamplona, M. M. Oliveira, and G. V. G. Baranoski, "Photorealistic models for pupil light reflex and iridal pattern deformation," ACM Trans. Graph., vol. 28, no. 4, pp. 1-12, 2009.
37. A. Longtin and J. Milton, "Modelling autonomous oscillations in the human pupil light reflex using non-linear delay-differential equations," Bulletin of Mathematical Biology, vol. 51, no. 5, pp. 605-624, September 1989.
38. H. Bouma and L. C. J. Baghuis, "Hippus of the pupil: Periods of slow oscillations of unknown origin," Vision Research, vol. 11, no. 11, 1971.
39. F. H. Kanfer, "Verbal rate, eyeblink, and content in structured psychiatric interviews," Journal of Abnormal and Social Psychology, vol. 61, no. 3, pp. 341-347, 1960.
40. G. Chanel, C. Rebetez, M. Bétrancourt, and T. Pun, "Emotion assessment from physiological signals for adaptation of game difficulty," Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, vol. PP, no. PP, p. In Press, 2011.
41. S. D. Kreibig, G. Schaefer, and T. Brosch, Psychophysiological response patterning in emotion: Implications for affective computing. Oxford, UK: Oxford University Press, 2010, ch. 2.4, pp. 105-130.