Fusion of facial expressions and EEG for implicit affective tagging

Image and Vision Computing

Volumn 31, Issue 2, 2013, Pages 164-174

  Koelstra, Sander ^a   Patras, Ioannis ^a  

^a QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Affective computing; EEG; Emotion classification; Facial expressions; Pattern classification; Signal processing

Indexed keywords

ELECTROENCEPHALOGRAPHY; ELECTROPHYSIOLOGY; PATTERN RECOGNITION; SIGNAL PROCESSING; SPEECH RECOGNITION;

AFFECTIVE COMPUTING; DECISION LEVEL FUSION; EMOTION CLASSIFICATION; FACIAL EXPRESSIONS; INTENSIVE RESEARCH; MULTI-MODAL APPROACH; MULTIMEDIA CONTENTS; SEARCH AND RETRIEVAL;

USER INTERFACES;

EID: 84885627733     PISSN: 02628856     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.imavis.2012.10.002     Document Type: Article

References (45)

1. A. Vinciarelli, N. Suditu, M. Pantic, Implicit human-centered tagging, IEEE Int'l Conf. Multimedia and Expo, 2009, pp. 1428-1431.
2. L. von Ahn, L. Dabbish, Labeling images with a computer game, Proc. Conf. Human Factors in Computing Systems, 2004, pp. 319-326.
3. L. von Ahn, R. Liu, M. Blum, Peekaboom, Proc. Conf. Human Factors in Computing Systems, 2006, pp. 55-64.
4. M. Mandel, D. Ellis, A web-based game for collecting music metadata, J. New Music Res. 37 (2) (2008) 151-165.
5. Y. Kim, E. Schmidt, L. Emelle, Moodswings: A collaborative game for music mood label collection, Proc. Int'l Conf. Music Information Retrieval, 2008, pp. 231-236.
6. A.D. Gerson, L.C. Parra, P. Sajda, Cortically coupled computer vision for rapid image search, IEEE Trans. Neural Syst. Rehabil. Eng. 14 (2) (2006) 174-179. (Pubitemid 44150235)
7. A. Kapoor, P. Shenoy, D. Tan, Combining brain computer interfaces with vision for object categorization, Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008, pp. 1-8.
8. A. Kapoor, D. Tan, P. Shenoy, E. Horvitz, Complementary computing for visual tasks: meshing computer vision with human visual processing, Proc. IEEE Conf. Face and Gesture Recognition, 18 (4), 2008, pp. 1-7.
9. A.J. Cowell, K. Hale, C. Berka, S. Fuchs, A. Baskin, D. Jones, G. Davis, R. Johnson, R. Patch, E. Marshall, Brainwave-based imagery analysis, digital human modeling: Trends in human algorithms, Lect. Notes Comput. Sci. 4650 (2008) 17-27.
10. S. Koelstra, C. Mühl, I. Patras, EEG analysis for implicit tagging of video data, Proc. Workshop on Affective Brain-Computer Interfaces, 2009, pp. 27-32.
11. G. Chanel, J.J.M. Kierkels, M. Soleymani, T. Pun, Short-term emotion assessment in a recall paradigm, Int. J. Hum. Comput. Stud. 67 (8) (2009) 607-627.
12. S. Koelstra, C.Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras, DEAP: A database for emotion analysis using physiological signals, IEEE Trans. Affect. Comput. 3 (1) (2012) 18-31.
13. M. Soleymani, S. Koelstra, I. Patras, T. Pun, Continuous emotion detection in response to music Videos, IEEE Int'l Conf. Automatic Face & Gesture Recognition and Workshops, 2011, pp. 803-808.
14. K. Takahashi, A. Tsukaguchi, Remarks on emotion recognition from multi-modal bio-potential signals, IEEE Int'l Conf. Systems, Man and Cybernetics, vol. 2, 2003, pp. 1654-1659.
15. J.K. Olofsson, S. Nordin, H. Sequeira, J. Polich, Affective picture processing: An integrative review of ERP findings, Biol. Psychol. 77 (3) (2008) 247-265.
16. B. Cuthbert, H. Schupp, M. Bradley, N. Birbaumer, P. Lang, Brain potentials in affective picture processing: covariation with autonomic arousal and affective report, Biol. Psychol. 52 (2) (2000) 95-111. (Pubitemid 30122969)
17. J. Polich, Updating P300: An integrative theory of P3a and P3b, Clin. Neurophysiol. 118 (10) (2007) 2128-2148. (Pubitemid 47385341)
18. M. Pantic, L.J.M. Rothkrantz, Automatic analysis of facial expressions - the state of the art, IEEE Trans. Pattern Anal. Mach. Intell. 22 (12) (2000) 1424-1445. (Pubitemid 32088469)
19. M. Pantic, M.S. Bartlett, Machine Analysis of Facial Expressions, I-Tech Education and Publishing, Vienna, Austria, 2007.
20. Z. Zeng, M. Pantic, G.I. Roisman, T.S. Huang, A survey of affect recognition methods: Audio, visual, and spontaneous expressions, IEEE Trans. Pattern Anal. Mach. Intell. 31 (1) (2009) 39-58.
21. P. Ekman, W.V. Friesen, J.C. Hager, The Facial Action Coding System: A Technique for the Measurement of Facial Movement, A Human Face, Salt Lake City, UT, 2002.
22. P. Ekman, E.L. Rosenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS), Oxford University Press, 2005.
23. A.C. Williams, Facial expression of pa an evolutionary account, Behav. Brain Sci. 25 (4) (2002) 439-488.
24. R. Kaliouby, Real-time inference of complex mental states from facial expressions and head gestures, B. Kisačanin, V. Pavlović, T. Huang (Eds.), Real-time Vision for Human-Computer Interaction, Springer, US, 2005, pp. 181-200.
25. H. Gunes, B. Schuller, M. Pantic, R. Cowie, Emotion representation, analysis and synthesis in continuous space: A survey, Proc. IEEE Conf. Face and Gesture Recognition, 2011, pp. 827-834.
26. J. Russell, A circumplexmodel of affect, J. Pers. Soc. Psychol. 39 (6) (1980) 1161-1178.
27. D.McDuff, R.E. Kaliouby, Affect valence inference fromfacial action unit spectrograms, Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010, pp. 17-24.
28. M. Hussein, T. Elsayed, Studying Facial Expressions as an Implicit Feedback in Information Retrieval Systems, 2008.
29. I. Arapakis, I. Konstas, J.M. Jose, Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance, Proc. ACM Int'l Conf. Multimedia, 2009, pp. 461-470.
30. J. Jiao, M. Pantic, Implicit image tagging via facial information, Proc. Int'l Workshop on Social Signal Processing, 2010, pp. 59-64.
31. J.-S. Lee, C.H. Park, Robust audio-visual speech recognition based on late integration, IEEE Trans. Multimed. 10 (5) (2008) 767-779.
32. P.K. Atrey, M.A. Hossain, A. El-Saddik, M.S. Kankanhalli, Multimodal fusion for multimedia analysis: A survey, Multimed. Syst. 16 (6) (2010) 345-379.
33. M. Soleymani, J. Lichtenauer, T. Pun, M. Pantic, A multi-modal affective database for affect recognition and implicit tagging, IEEE Trans. Affect. Comput. 3 (1) (2012) 42-55.
34. N. Sebe, I. Cohen, T. Huang, Multimodal emotion recognition, Handbook of Pattern Recognition and Computer Vision, 2005, p. 387-419.
35. M.M. Bradley, P.J. Lang, Measuring emotion: The Self-Assessment Manikin and the Semantic Differential, J. Behav. Ther. Exp. Psychiatry 25 (1) (1994) 49-59. (Pubitemid 24214918)
36. J.D. Morris, Observations: SAM: The Self-Assessment Manikin; an efficient cross-cultural measurement of emotional response, J. Advert. Res. 35 (8) (1995) 38-63.
37. S. Koelstra, M. Pantic, I. Patras, A dynamic texture-based approach to recognition of facial actions and their temporal models, IEEE Trans. Pattern Anal. Mach. Intell. 32 (11) (2010) 1940-1954.
38. K. Scherer, What are emotions? And how can they be measured? Soc. Sci. Inf. 44 (4) (2005) 695-729. (Pubitemid 41666684)
39. A. Bell, T. Sejnowski, An information-maximization approach to blind separation and blind deconvolution, Neural Comput. 7 (6) (1995) 1129-1159.
40. A. Hyvärinen, Fast and robust fixed-point algorithms for independent component analysis, IEEE Trans. Neural Netw. 10 (3) (1999) 626-634.
41. D. Mackay, Bayesian interpolation, Neural Comput. 4 (3) (1992) 415-447.
42. S. Koelstra, A. Yazdani, M. Soleymani, C. Muehl, J. Lee, A. Nijholt, T. Pun, T. Ebrahimi, I. Patras, Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos, Proc. Brain Informatics, 2010, pp. 89-100.
43. R.J. Barry, A.R. Clarke, S.J. Johnstone, C.A. Magee, J.A. Rushby, EEG differences between eyes-closed and eyes-open resting conditions, Clin. Neurophysiol. 118 (12) (2007) 2765-2773. (Pubitemid 350180601)
44. J. Onton, S. Makeig, High-frequency broadband modulations of electroencephalographic spectra, Front. Hum. Neurosci. 3 (61) (2009) 1-18.
45. P. Lang, M. Bradley, B. Cuthbert, International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual, Tech. Rep. A-8, University of Florida, 2008.