Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection

IEEE Transactions on Affective Computing

Volumn 7, Issue 1, 2016, Pages 17-28

  Soleymani, Mohammad ^a   Asghari Esfeden, Sadjad ^b   Fu, Yun ^b   Pantic, Maja ^c,d  

^a UNIVERSITY OF GENEVA   (Switzerland)

^b Northeastern University   (United States)

^c IMPERIAL COLLEGE LONDON   (United Kingdom)

^d UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Affect; EEG; facial expressions; implicit tagging; video highlight detection

Indexed keywords

ELECTROENCEPHALOGRAPHY; PHYSIOLOGICAL MODELS; RECURRENT NEURAL NETWORKS; SIGNAL DETECTION;

AFFECT; CONDITIONAL RANDOM FIELD; ELECTROENCEPHALOGRAM SIGNALS; FACIAL EXPRESSIONS; HIGHLIGHT DETECTION; IMPLICIT TAGGING; LONG SHORT TERM MEMORY; PHYSIOLOGICAL RESPONSE;

FACE RECOGNITION;

EID: 84963876689     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAFFC.2015.2436926     Document Type: Article

References (48)

1. P. G. Hunter, E. G. Schellenberg, and A. T. Griffith, "Misery loves company:Mood-congruent emotional responding to music," Emotion, vol. 11, no. 5, pp. 1068-1072, 2011.
2. M. Soleymani, M. Larson, T. Pun, and A. Hanjalic, "Corpus development for affective video indexing," IEEE Trans. Multimedia, vol. 16, no. 4, pp. 1075-1089, Jun. 2014.
3. H. Joho, J. Staiano, N. Sebe, and J. Jose, "Looking at the viewer: Analysing facial activity to detect personal highlights of multimedia contents," Multimedia Tools Appl., vol. 51, no. 2, pp. 505-523, 2010.
4. F. Silveira, B. Eriksson, A. Sheth, and A. Sheppard, "Predicting audience responses to movie content from electro-dermal activity signals," in Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput., 2013, pp. 707-716.
5. N. Malandrakis, A. Potamianos, G. Evangelopoulos, and A. Zlatintsi, "A supervised approach to movie emotion tracking," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., May 2011, pp. 2376-2379.
6. K. R. Scherer, "What are emotions? And how can they be measured?" Soc. Sci. Inf., vol. 44, no. 4, pp. 695-729, 2005.
7. J. A. Russell, "Culture and the categorization of emotions," Psych. Bul., vol. 110, no. 3, pp. 426-450, 1991.
8. J. A. Russell and A. Mehrabian, "Evidence for a three-factor theory of emotions," J. Res. Personality, vol. 11, no. 3, pp. 273-294, 1977.
9. H. Gunes and B. Schuller, "Categorical and dimensional affect analysis in continuous input: Current trends and future directions," Image Vis. Comput., vol. 31, no. 2, pp. 120-136, 2013.
10. C. W. Granger, "Investigating causal relations by econometric models and cross-spectral methods," Econometrica: J. Econ. Soc., vol. 37, pp. 424-438, 1969.
11. M. Soleymani, S. Asghari-Esfeden, M. Pantic, and Y. Fu, "Continuous emotion detection using EEG signals and facial expressions," in Proc. IEEE Int. Conf. Multimedia Expo, 2014, pp. 1-6.
12. M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie, "Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies," in Proc. 9th Annu. Conf. Int. Speech Commun. Assoc., 2008, pp. 597-600.
13. M. Nicolaou, H. Gunes, and M. Pantic, "Continuous prediction of spontaneous affect from multiple cues and modalities in valencearousal space," IEEE Trans. Affective Comput., vol. 2, no. 2, pp. 92-105, Apr.-Jun. 2011.
14. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, "The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 5-17, Jan.-Mar. 2012.
15. M. A. Nicolaou, H. Gunes, and M. Pantic, "Output-associative RVM regression for dimensional and continuous emotion prediction," Image Vis. Comput., Special Issue Best Automatic Face Gesture Recognit., vol. 30, no. 3, pp. 186-196, 2012.
16. B. Schuller, M. Valster, F. Eyben, R. Cowie, and M. Pantic, "AVEC 2012: The continuous audio/visual emotion challenge," in Proc. ACM Int. Conf. Multimodal Interaction, 2012, pp. 449-456.
17. T. Baltrusaitis, N. Banda, and P. Robinson, "Dimensional affect recognition using continuous conditional random fields," in Proc. IEEE Int. Conf. Automatic Face Gesture Recognit., 2013, pp. 1-8.
18. M. Pantic and A. Vinciarelli, "Implicit human-centered tagging," IEEE Signal Process. Mag., vol. 26, no. 6, pp. 173-180, Nov. 2009.
19. X. Bao, S. Fan, A. Varshavsky, K. Li, and R. Roy Choudhury, "Your reactions suggest you liked the movie: Automatic content rating via reaction sensing," in Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput., New York, NY, USA, 2013, pp. 197-206.
20. D. McDuff, R. El Kaliouby, and R. W. Picard, "Crowdsourcing facial responses to online videos," IEEE Trans. Affective Comput., vol. 3, no. 4, pp. 456-468, Oct.-Dec. 2012.
21. D. McDuff, R. el Kaliouby, D. Demirdjian, and R. Picard, "Predicting online media effectiveness based on smile responses gathered over the internet," in Proc. 10th IEEE Int. Conf. Workshops Automatic Face Gesture Recognition., 2013, pp. 1-7.
22. H. Joho, J. M. Jose, R. Valenti, and N. Sebe, "Exploiting facial expressions for affective video summarisation," in Proc. ACM Int. Conf. Image Video Retrieval, New York, NY, USA, 2009, p. 31.
23. C. Chênes, G. Chanel, M. Soleymani, and T. Pun, "Highlight detection in movie scenes through inter-users, physiological linkage," in Social Media Retrieval, N. Ramzan, R. van Zwol, J.-S. Lee, K. Clüver, and X.-S. Hua, Eds. New York, NY, USA: Springer, 2013, pp. 217-237.
24. J. Fleureau, P. Guillotel, and I. Orlac, "Affective benchmarking of movies based on the physiological responses of a real audience," in Proc. 3rd Int. Conf. Affective Comput. Intell. Interaction Workshops, Sep. 2013, pp. 73-77.
25. M. Soleymani, G. Chanel, J. J. M. Kierkels, and T. Pun, "Affective characterization of movie scenes based on content analysis and physiological changes," Int. J. Semantic Comput., vol. 3, no. 2, pp. 235-254, Jun. 2009.
26. J. J. M. Kierkels, M. Soleymani, and T. Pun, "Queries and tags in affect-based multimedia retrieval," in Proc. IEEE Int. Conf. Multimedia Expo, Piscataway, NJ, USA, 2009, pp. 1436-1439.
27. M. Soleymani, S. Koelstra, I. Patras, and T. Pun, "Continuous emotion detection in response to music videos," in Proc. IEEE Int. Conf. Automatic Face Gesture Recognit., Mar. 2011, pp. 803-808.
28. S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Y. Patras, "DEAP: A database for emotion analysis using physiological signals," IEEE Trans. Affective Comput., vol. 3, no. 2, pp. 18-31, Jan.-Mar. 2012.
29. M. Soleymani, M. Pantic, and T. Pun, "Multimodal emotion recognition in response to videos," IEEE Trans. Affective Comput., vol. 3, no. 2, pp. 211-223, Apr.-Jun. 2012.
30. M. K. Abadi, S. M. Kia, R. Subramanian, P. Avesani, and N. Sebe, "User-centric affective video tagging from MEG and peripheral physiological responses," in Proc. 3rd Int. Conf. Affective Comput. Intell. Interaction Workshops, Sep. 2013, pp. 582-587.
31. M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, "A multimodal database for affect recognition and implicit tagging," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 42-55, Jan.-Mar. 2012.
32. M. M. Bradley and P. J. Lang, "Measuring emotion: The selfassessment manikin and the semantic differential," J. Behavior Therapy Experimental Psychiatry, vol. 25, no. 1, pp. 49-59, Mar. 1994.
33. A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, "Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers," Cogn. Emotion, vol. 24, no. 7, pp. 1153-1172, 2010.
34. J. Rottenberg, R. D. Ray, and J. J. Gross, Emotion Elicitation Using Films. series Affective Science. London, U.K.: Oxford Univ. Press, 2007, pp. 9-28.
35. R. J. Barry and E. N. Sokolov, "Habituation of phasic and tonic components of the orienting reflex," Int. J. Psychophysiol., vol. 15, no. 1, pp. 39-42, 1993.
36. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. Mcmahon, M. Sawey, and M. Schröder, "'feeltrace': An instrument for recording perceived emotion in real time," in Proc. ISCA Workshop Speech Emotion, 2000, pp. 19-24.
37. R. J. Davidson, "Affective neuroscience and psychophysiology: Toward a synthesis.," Psychophysiology, vol. 40, no. 5, pp. 655-665, 2003.
38. Xuehan-Xiong and F. De la Torre, "Supervised descent method and its application to face alignment," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 532-539.
39. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
40. I. Goncharova, D. McFarland, T. Vaughan, and J. Wolpaw, "EMG contamination of EEG: Spectral and topographical characteristics," Clin. Neurophysiol., vol. 114, no. 9, pp. 1580-1593, Sep. 2003.
41. E. Niedermeyer, "The normal EEG of the waking adult," in Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Philadelphia, PA, USA: Lippincott Williams & Wilkins, 2005, p. 167.
42. S. Mariooryad and C. Busso, "Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations," in Proc. Humaine Assoc. Conf. Affect. Comput. Intell. Interaction, Sep. 2013, pp. 85-90.
43. F. Weninger, F. Eyben, and B. Schuller, "On-line continuous-time music mood regression with deep recurrent neural networks," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 5412-5416.
44. R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, "Liblinear: A library for large linear classification," J. Mach. Learn. Res., vol. 9, pp. 1871-1874, 2008.
45. S. Koelstra and I. Patras, "Fusion of facial expressions and eeg for implicit affective tagging," Image Vis. Comput., vol. 31, no. 2, pp. 167-174, 2013.
46. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani, "Robust continuous prediction of human emotions using multiscale dynamic cues," in Proc. ACM Int. Conf. Multimodal Interaction, 2012, pp. 501-508.
47. Y. Song, L.-P. Morency, and R. Davis, "Learning a sparse codebook of facial and body microexpressions for emotion recognition," in Proc. ACM Int. Conf. Multimodal Interaction, 2013, pp. 237-244.
48. D. J. McDuff, "Crowdsourcing affective responses for predicting media effectiveness," Ph.D. dissertation, Massachusetts Inst. Technol., Cambridge, MA, USA, 2014.