Multi-modal emotion analysis from facial expressions and electroencephalogram

Computer Vision and Image Understanding

Volumn 147, Issue , 2016, Pages 114-124

  Huang, Xiaohua ^a   Kortelainen, Jukka ^a   Zhao, Guoying ^a   Li, Xiaobai ^a   Moilanen, Antti ^a   Seppänen, Tapio ^a   Pietikäinen, Matti ^a  

^a UNIVERSITY OF OULU   (Finland)

Author keywords

Electroencephalogram; Facial expression; Feature representation; Multi modal emotion recognition; Pattern recognition; Signal processing

Indexed keywords

BEHAVIORAL RESEARCH; COMPUTER VISION; ELECTROENCEPHALOGRAPHY; MODAL ANALYSIS; PATTERN RECOGNITION; SIGNAL PROCESSING; SPEECH RECOGNITION;

AUTOMATIC ANALYSIS; ELECTRO-ENCEPHALOGRAM (EEG); EMOTION ANALYSIS; EMOTION RECOGNITION; FACIAL EXPRESSIONS; FEATURE REPRESENTATION; HUMAN PERFORMANCE; MULTI-MODAL APPROACH;

FACE RECOGNITION;

EID: 84990050332     PISSN: 10773142     EISSN: 1090235X     Source Type: Journal    
DOI: 10.1016/j.cviu.2015.09.015     Document Type: Article

References (54)

1. [1] Ahonen, T., Hadid, A., Pietikäinen, M., Face description with local binary patterns: Application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28:12 (2006), 2037–2041.
2. [2] Atiya, A.F., Estimating the posterior probabilities using the k-nearest neighbor rule. Neural Comput. 17:3 (2005), 731–740.
3. [3] Bailenson, J.N., Pontikakis, E.D., Mauss, I.B., Gross, J.J., Jabon, M.E., Hutcherson, C., Nass, C., John, O., Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int. J. Hum. Comput. Stud. 66:5 (2008), 303–317.
4. [4] Chan, C., Goswami, B., Kittler, J., Christmas, W., Local ordinal contrast pattern histograms for spatiotemporal, lip-based speaker authentication. IEEE Trans. Inf. Forensics Security 7:2 (2012), 602–612.
5. [5] Chang, C., Lin, C., Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol., 2(3), 2011, 27.
6. [6] Chang, C., Tsai, J., Wang, C., Chung, P., Emotion recognition with consideration of facial expression and physiological signals. Proceeding of IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology, 2009, 278–283.
7. [7] Davidson, R., Fox, N., Asymmetrical brain activity discriminates between positive and negative affective stimuli in human infants. Science 218:4578 (1982), 1235–1237.
8. [8] Delorme, A., Makeig, S., EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134:1 (2004), 9–21.
9. [9] Draghici, A., Petca, A., Hillebrandt, H., Inferring emotion from facial expression in social context. A role of self-construal?. J. Eur. Psychol. Students 1 (2009), 1–8.
10. [10] Ekman, P., Expression and the nature of emotion. Scherer, K., Ekman, P., (eds.) Approach. Emotion, 1984, Hillsdale, NJ, Erlbaum, 319–344.
11. [11] Farquhar, J., Hardoon, D., Meng, H., Shawe-taylor, J., Szedmak, S., Two view learning: SVM-2K, theory and practice. Proceeding of Neural Information Processing Systems, 2006, 355–362.
12. [12] Fontaine, J., Scherer, K., Roesch, E., Ellsworth, P., The world of emotions is not two-dimensional. Psychol. Sci. 18:12 (2007), 1050–1057.
13. [13] Hardoon, D., Szedmak, S., Shawe-Taylor, J., Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16:12 (2004), 2639–2664.
14. [14] He, X., Niyogi, P., Locality preserving projections. Proceedings of Neural Information Processing Systems, 2003, 153–160.
15. [15] Huang, X., Zhao, G., Pietikäinen, M., Zheng, W., Expression recognition in videos using a weighted component-based feature descriptor. Proceedings of the 17th Scandinavian Conference on Image Analysis, 2011, 569–578.
16. [16] Huang, X., Zhao, G., Zheng, W., Pietikäinen, M., Spatiotemporal local monogenic binary patterns for facial expression recognition. IEEE Signal Process. Lett. 19:5 (2012), 243–246.
17. [17] Huang, X., Zhao, G., Zheng, W., Pietikäinen, M., Towards a dynamic expression recognition system under facial occlusion. Pattern Recognit. Lett. 33:16 (2012), 2181–2191.
18. [18] Jiang, B., Valstar, M., Pantic, M., Action unit detection using sparse appearance descriptors in space-time video volumes. Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition, 2011, 314–321.
19. [19] Kaltwang, S., Rudovic, O., Pantic, M., Continuous pain intensity estimation from facial expressions. Proceedings of International Symposium on Advances in Visual Computing, 2012, 368–377.
20. [20] Kittler, J., Hatef, M., Duin, R., Matas, J., On combining classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 20:3 (1998), 226–239.
21. [21] Koelstra, S., Patras, I., Fusion of facial expressions and EEG for implicit affective tagging. Image Vision Comput. 31:2 (2013), 164–174.
22. [22] Kolodyazhniy, V., Kreibig, S., Gross, J., Roth, W., Wilhelm, F., An affective computing approach to physiological emotion specificity: toward subject independent and stimulus independent classification of film induced emotions. Psychophysiology 48:7 (2011), 908–922.
23. [23] Kortelainen, J., Seppänen, T., EEG-based recognition of video-induced emotions: selecting subject-independent feature set. Proceedings of IEEE International Conference on Engineering in Medicine and Biology Society, 2013, 4287–4290.
24. [24] Kortelainen, J., Tiinanen, S., Huang, X., Li, X., Laukka, S., Pietikäinen, M., Seppänen, T., Multimodal emotion recognition by combining physiological signals and facial expressions: a preliminary study. Proceedings of IEEE International Conference on Engineering in Medicine and Biology Society, 2012, 5238–5241.
25. [25] Kortelainen, J., Väyrynen, E., Seppänen, T., High-frequency electroencephalographic activity in left temporal area is associated with pleasant emotion induced by video clips. Comput. Intell. Neurosci., vol. 2015, 2015, 10.1155/2015/762769 Article ID 762769, 14 pages.
26. [26] Le Ngo, A., Phan, R., See, J., Spontaneous subtle expression recognition: imbalanced databases & solutions. Proceedings of Asian Conference on Computer Vision, 2014.
27. [27] Lee, J., Park, C., Robust audio–visual speech recognition based on late integration. IEEE Trans. Multimed. 10:5 (2008), 767–779.
28. [28] Lin, H., Lin, C., Weng, R., A note on platts probabilistic outputs for support vector machines. Mach. Learn. 68:3 (2007), 267–276.
29. [29] Lin, Y., Wang, C., Jung, T., Wu, T., Jeng, S., Duann, J., Chen, J., EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57:7 (2010), 1798–1806.
30. [30] Mahoor, M., Cadavid, S., Messinger, D., Cohn, J., A framework for automated measurement of the intensity of non-posed facial action units. Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition Workshops, 2009, 74–80.
31. [31] Mehrabian, A., Communication without words. Psychol. Today 2 (1968), 53–55.
32. [32] Meng, H., Huang, D., Wang, H., Yang, H., AI-Shuraifi, M., Wang, Y., Depression recognition based on dynamic facial and vocal expression features using partial least square regression. Proceedings of ACM International Conference on Multimedia Interaction, 2013, 21–30.
33. [33] Murugappan, M., Ramachandran, N., Sazali, Y., Classification of human emotion from EEG using discrete wavelet transform. J. Biomed. Sci. Eng. 3:4 (2010), 390–396.
34. [34] Nie, D., Wang, X., Shi, L., Lu, B., EEG-based emotion recognition during watching movies. Proceedings of IEEE International Conference on Neural Engineering, 2011, 667–670.
35. [35] Niu, Z., Shan, S., Yan, S., Chen, X., Gao, W., 2D cascaded Adaboost for eye localization. Proceedings of IEEE International Conference on Pattern Recognition, 2006, 1216–1219.
36. [36] Ojala, T., Matti, M.P., Mäenpää, T., Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24:7 (2002), 971–987.
37. [37] Orabona, F., Jie, L., Ultra-fast optimization algorithm for sparse multi kernel learning. Proceedings of International Conference on Machine Learning, 2011, 249–256.
38. [38] Pantic, M., Machine analysis of facial behaviour: naturalistic and dynamic behaviour. Philos. Trans. R. Soc. B: Biol. Sci. 364:1535 (2009), 3505–3513.
39. [39] Park, S., Kim, D., Subtle facial expression recognition using motion magnification. Pattern Recognit. Lett. 30:7 (2009), 708–716.
40. [40] Pavlidis, I., Tsiamyrtzis, P., Shastri, D., Wesley, A., Zhou, Y., Lindner, P., Buddharaju, P., Joseph, R., Mandapati, A., Dunkin, B., Bass, B., Fast by nature—how stress patterns define human experience and performance in dexterous tasks. Sci. Rep. 2:305 (2007), 1–9.
41. [41] Petratonakis, P., Hadjileontiadis, L., Emotion recognition from EEG using higher order crossings. IEEE Trans. Inf. Technol. Biomed. 14:2 (2009), 186–197.
42. [42] Pudil, P., Novovicova, J., Kittler, J., Floating search methods in feature selection. Pattern Recognit. Lett. 15:11 (1994), 1119–1125.
43. [43] Russell, J., A circumplex model of affect. J Personality Social Psychol. 39:6 (1980), 1161–1178.
44. [44] Savran, A., Cao, H., Shah, M., Nenkova, A., Verma, R., Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering. Proceedings of ACM International Conference on Multimedia Interaction, 2012, 485–492.
45. [45] Sebe, N., Cohen, I., Gevers, T., Huang, T., Multimodal approaches for emotion recognition: a survey. Electronic Imaging 2005, 2005, International Society for Optics and Photonics, 56–67.
46. [46] Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M., A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affective Comput. 3:1 (2012), 42–55.
47. [47] Takahashi, K., Remarks on emotion recognition from multi-modal bio-potential signals. Proceedings of IEEE International Conference on Industrial Technology, 2004, 1138–1143.
48. [48] Valstar, M., Pantic, M., Ambadar, Z., Cohn, J., Spontaneous vs. posed facial behavior: automatic analysis of brow actions. Proceedings of ACM International Conference on Multimodal Interfaces, 2006, 162–170.
49. [49] Wan, S., Aggarwal, J., Spontaneous facial expression recognition: a robust metric learning approach. Pattern Recognit. 47:5 (2014), 1859–1868.
50. [50] Welch, P., The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15:2 (1967), 70–73.
51. [51] Wesley, A., Lindner, P., Pavlidis, I., Eustressed or distressed?: combining physiology with observation in user studies. Proceedings of CHI’12 Extended Abstracts on Human Factors in Computing Systems, 2012, 327–330.
52. [52] Zeng, Z., Pantic, M., Roisman, G., Huang, T., A survey of affect recognition methods: audio, visual and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31:1 (2009), 39–58.
53. [53] Zhao, G., Pietikäinen, M., Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29:6 (2007), 915–928.
54. [54] Zheng, W., Zhu, J., Peng, Y., Lu, B., EEG-based emotion classification using deep belief networks. Proceedings of IEEE International Conference on Multimedia and Expo, 2014, 1–6.