Affect recognition based on physiological changes during the watching of music videos

ACM Transactions on Interactive Intelligent Systems

Volumn 2, Issue 1, 2012, Pages 1-26

  Yazdani, Ashkan ^a   Lee, Jong Seok ^b   Vesin, Jean Marc ^a   Ebrahimi, Touradj ^a  

^a EPFL   (Switzerland)

^b Yonsei University   (South Korea)

Author keywords

Affective computing; EEG; Emotion classification; Pattern classification; Physiological signals; Signal processing

Indexed keywords

CLASSIFICATION (OF INFORMATION); ELECTROENCEPHALOGRAPHY; PATTERN RECOGNITION; PHYSIOLOGY; SIGNAL PROCESSING; SPEECH RECOGNITION; VIDEO CAMERAS;

AFFECTIVE COMPUTING; BINARY CLASSIFICATION; CLASSIFICATION RESULTS; CLASSIFICATION SYSTEM; ELECTRO-ENCEPHALOGRAM (EEG); EMOTION CLASSIFICATION; MULTIMEDIA CONTENT DISTRIBUTION; PHYSIOLOGICAL SIGNALS;

BIOMEDICAL SIGNAL PROCESSING;

EID: 84983569154     PISSN: 21606455     EISSN: 21606463     Source Type: Journal    
DOI: 10.1145/2133366.2133373     Document Type: Article

References (51)

1. AFTANAS, L., REVA, N., VARLAMOV, A., PAVLOV, S., AND MAKHNEV, V. 2004. Analysis ofevoked EEG synchronization and desynchronization in conditions of emotional activation in humans: Temporal and topographic characteristics. Neurosci. Behav. Physiol. 34, 8, 859-867..
2. BLANCO, S., FIGLIOLA, A., QUIROGA, R., ROSSO, O., AND SERRANO, E. 1998. Time-Frequency analysis of electroen-cephalogram series. III. Wavelet packets and information cost function. Phys. Rev. E 57, 1, 932-940..
3. CHANEL, G., KIERKELS, J., SOLEYMANI, M., AND PUN, T. 2009. Short-term emotion assessment in a recall paradigm. Int. J. Hum.-Comput. Stud. 67, 8, 607-627..
4. CHANEL, G., KRONEGG, J., GRANDJEAN, D., AND PUN, T. 2006. Emotion assessment: Arousal evaluation using EEG's and peripheral physiological signals. In Proceedings of the Conference on Multimedia Content Representation, Classification and Security. 530-537..
5. COAN, J. AND ALLEN, J. 2004. Frontal EEG asymmetry as a moderator and mediator of emotion. Biol. Psychol. 67, 1-2, 7-50..
6. COWIE, R. 2010. Emotion-Oriented Systems: The Humaine Handbook. Springer..
7. COWIE, R., DOUGLAS-COWIE, E., TSAPATSOULIS, N., VOTSIS, G., KOLLIAS, S., FELLENZ, W., AND TAYLOR, J. 2002. Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 18, 1, 32-80..
8. EKMAN, P., FRIESEN, W. V., O'SULLIVAN, M., CHAN, A., DIACOYANNI- TARLATZIS, I., HEIDER, K., KRAUSE, R., LECOMPTE, W. A., PITCAIRN, T., AND RICCI-BITTI, P. E. 1987. Universals and cultural differences in thejudgments of facial expressions of emotion. J. Person. Social Psychol. 53, 4, 712-717..
9. EKMAN, P., LEVENSON, R. W., AND FRIESEN, W. V. 1983. Autonomic nervous system activity distinguishes among emotions. Sci. 221, 1208-1210..
10. FRAGOPANAGOS, N. AND TAYLOR, J. 2005. Emotion recognition in human-computer interaction. Neural Netw. 18, 4, 389-405..
11. GILLICK, L. AND COX, S. J. 1989. Some statistical issues in the comparison ofspeech recognition algorithms. In Proceedings of the International Conference Acoustics, Speech and Signal Processing. 532-535..
12. GUO, L., RIVERO, D., SEOANE, J., AND PAZOS, A. 2009. Classification of EEG signals using relative wavelet energy and artificial neural networks. In Proceedings of the ACM/SIGEVO Summit on Genetic and Evolutionary Computation. 177-184..
13. HANJALIC, A. AND XU, L. 2005. Affective video content representation and modeling. IEEE Trans. Multimedia 7, 1, 143-154..
14. HEALEY, J. A. 2000. Wearable and automotive systems for affect recognition from physiology. Ph.D. thesis, MIT..
15. ISHINO, K. AND HAGIWARA, M. 2003. A feeling estimation system using a simple electroencephalograph. In Proceedings of the IEEE International Conference Systems, Man and Cybernetics. Vol. 5. 4204-4209..
16. JAIMES, A. AND SEBE, N. 2007. Multimodal human-computer interaction: A survey. Comput. Vis. Image Under-stand. 108, 1-2, 116-134..
17. KIM, J. AND ANDRÉ, E. 2008. Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 12, 2067-2083..
18. KIM, K., BANG, S., AND KIM, S. 2004. Emotion recognition system using short-term monitoring ofphysiological signals. Med. Biol. Engin. Comput. 42, 3, 419-427..
19. KOELSTRA, S., YAZDANI, A., SOLEYMANI, M., MÜHL, C., LEE, J.-S., NIJHOLT, A., PUN, T., EBRAHIMI, T., AND PATRAS, I. 2010. Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos. In Proceedings of the International Conference Brain Informatics. Springer, 89-100..
20. KOLEV, V., DEMIRALP, T., YORDANOVA, J., ADEMOGLU, A., AND ISOGLU-ALKAÇ, Ü. 1997. Time-frequency analysis reveals multiple functional components during oddball P300. NeuroRep. 8, 8, 2061-2065..
21. KOSTYUNINA, M. AND KULIKOV, M. 1996. Frequency characteristics of EEG spectra in the emotions. Neurosci. Behav. Physiol. 26, 4, 340-343..
22. KRAUSE, C., VIEMERÖ, V., ROSENQVIST, A., SILLANMÄKI, L., AND ASTRÖM, T. 2000. Relative electroencephalographic desynchronization and synchronization in humans to emotional film content: An analysis of the 4-6, 6-8, 8-10 and 10-12 Hz frequency bands. Neurosci. Lett. 286, 1, 9-12..
23. LANG, P. 1995. The emotion prob: Studies of motivation and attention.Amer. Psychol. 50, 5, 372-385..
24. LANG, P., BRADLEY, M., AND CUTHBERT, B. 2008. International affective picture system (IAPS): Affective ratings ofpictures and instruction manual. Tech. rep. A-8, University of Florida, Gainesville, FL..
25. LANG, P., GREENWALD, M., BRADELEY, M., AND HAMM, A. 1993. Looking at pictures- affective, facial, visceral, and behavioral reactions. Psychophysiol. 30, 3, 261-273..
26. LIN, Y., WANG, C., JUNG, T., WU, T., JENG, S., DUANN, J., AND CHEN, J. 2010. Eeg-Based emotion recognition in music listening. IEEE Trans. Biomed. Engin. 57, 7, 1798-1806..
27. LISETTI, C. L. AND NASOZ, F 2004. Using noninvasive wearable computers to recognize human emotions from physiologicalsignals.EURASIPJ.Appl. SignalProcess., 1, 1672-1687..
28. MCFARLAND, R. A. 1985. Relationship of skin temperature changes to the emotions accompanying music. Biofeedback Self-Regul. 10, 3, 255-267..
29. MCNEMAR, E. L. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 153-157..
30. MORRIS, J. D. 1995. SAM: the self-assessment manikin. An efficient cross-cultural measurement of emotional response. J. Advertis. Res. 35, 8, 63-68..
31. MOTA, S. AND PICARD, R.2003. Automated posture analysis for detecting learner's interest level. In Proceedings of the Computer Vision and Pattern Recognition Workshop. 49-49..
32. PETRANTONAKIS, P. AND HADJILEONTIADIS, L. 2010. Emotion recognition from eeg using higher order crossings. IEEE Trans. Inf. Technol. Biomed. 14, 2, 186-197..
33. PICARD, R. W., VYZAS, E., AND HEALEY, J. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23, 10, 1175-1191..
34. PLUTCHIK, R. 2001. The nature of emotions.Amer. Sci. 89, 344..
35. ROSSO, O., BLANCO, S., YORDANOVA, J., KOLEV, V., FIGLIOLA, A., SCHÜRMANN, M., AND BAAR, E. 2001. Wavelet entropy: a new tool for analysis of short duration brain electrical signals. J. Neurosci. Methods 105, 1, 65-75..
36. RUSSELL, J. A. 1980. A circumplex model of affect. J. Personal. Social Psychol. 39, 6, 1161-1178..
37. SAVRAN, A., CIFTCI, K., CHANEL, G., MOTA, J. C., VIET, L. H., SANKUR, B., AKARUN, L., CAPLIER, A., AND ROMBAUT, M. 2006. Emotion detection in the loop from brain signals and facial images. In Proceedings of the eNTERFACE 2006 Workshop..
38. SCHAAFF, K. AND SCHULTZ, T. 2009. Towards emotion recognition from electroencephalographic signals. In Proceedings of the International ConferenceAffective Computing and Intelligent Interaction and Workshops. 1-6..
39. SEBE, N., COHEN, I., GEVERS, T., AND HUANG, T. 2006. Emotion recognition based onjoint visual and audio cues. In Proceedings of the 18th International Conference on Pattern Recognition. Vol. 1. IEEE, 1136-1139..
40. SHANNON, C. 2001. A mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Comm. Rev. 5, 1, 3-55..
41. SUTTON, S. AND DAVIDSON, R. 1997. Prefrontal brain asymmetry: A biological substrate of the behavioral approach and inhibition systems. Psychol. Sci. 8, 3, 204-210..
42. UNSER, M. 2002. Splines: A perfect fit for signal and image processing. IEEE Signal Process. Mag. 16, 6, 22-38..
43. VERVERIDIS, D. AND KOTROPOULOS, C. 2006. Emotional speech recognition: Resources, features, and methods. Speech Comm. 48, 9, 1162-1181..
44. WANG, J. AND GONG, Y. 2008. Recognition ofmultiple drivers' emotional state. In Proceedings of the International Conference Pattern Recognition. 1-4..
45. WILCOXON, F. 1945. Individual comparisons by ranking methods. Biomet. Bull. 1, 6, 80-83..
46. YAZDANI, A., LEE, J., AND EBRAHIMI, T. 2009. Implicit emotional tagging of multimedia using eeg signals and brain computer interface. In Proceedings of the 1st SIGMM Workshop on Social Media. ACM, 81-88..
47. YAZDANI, A., VESIN, J., IZZO, D., AMPATZIS, C., AND EBRAHIMI, T. 2010a. The impact ofexpertise on brain computer interface based salient image retrieval. In Proceedings of the IEEE Annual International Conference on Engineering in Medicine and Biology Society (EMBC), IEEE, 1646-1649..
48. YAZDANI, A., VESIN, J., IZZO, D., AMPATZIS, C., AND EBRAHIMI, T. 2010b. Implicit retrieval ofsalient images using brain computer interface. In Proceedings of the 17th IEEE International Conference on Image Processing (ICIP). IEEE, 3169-3172..
49. YEASIN, M., BULLOT, B., AND SHARMA, R. 2006. Recognition offacial expressions and measurement of levels of interest from video. IEEE Trans. Multimedia 8, 3, 500-508..
50. YORDANOVA, J., KOLEV, V., ROSSO, O., SCHÜRMANN, M., SAKOWITZ, O., ÖZGÖREN, M., AND BASAR, E. 2002.Wavelet entropy analysis of event-related potentials indicates modality-independent theta dominance. J. Neurosci. Methods 117, 1, 99-109..
51. ZENG, Z., PANTIC, M., ROISMAN, G. I., AND HUANG, T. S. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 1, 39-58..