Short-term emotion assessment in a recall paradigm

International Journal of Human Computer Studies

Volumn 67, Issue 8, 2009, Pages 607-627

  Chanel, Guillaume ^a   Kierkels, Joep J M ^a   Soleymani, Mohammad ^a   Pun, Thierry ^a  

^a UNIVERSITY OF GENEVA   (Switzerland)

Author keywords

Affective computing; Emotion assessment and classification; Signal processing

Indexed keywords

ACQUISITION PROTOCOLS; AFFECTIVE COMPUTING; CLASSIFICATION ACCURACY; DECISION LEVELS; EEG SIGNALS; EMOTION ASSESSMENT AND CLASSIFICATION; FEATURE SETS; HUMAN EMOTION; MUTUAL INFORMATIONS; NEUTRAL STATE; PATTERN CLASSIFICATION; PHYSIOLOGICAL SIGNALS; SPECIFIC AREAS; SUMMATION RULES; TIME FREQUENCY FEATURES; TIME PERIODS;

ABERRATIONS; SIGNAL PROCESSING;

ELECTROENCEPHALOGRAPHY;

EID: 67349173572     PISSN: 10715819     EISSN: 10959300     Source Type: Journal    
DOI: 10.1016/j.ijhcs.2009.03.005     Document Type: Article

References (85)

1. Aboy M., McNames J., Thong T., Tsunami D., Ellenby M.S., and Goldstein B. An automatic beat detection algorithm for pressure signals. IEEE Transactions on Biomedical Engineering 52 10 (2005) 1662-1670
2. Adolphs R., Tranel D., and Damasio A.R. Dissociable neural systems for recognizing emotions. Brain and Cognition 52 1 (2003) 61-69
3. Aftanas L.I., Reva N.V., Varlamov A.A., Pavlov S.V., and Makhnev V.P. Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics. Neuroscience and Behavioral Physiology 34 8 (2004) 859-867
4. Ansari-Asl, K., Chanel, G., Pun, T., 2007. A channel selection method for EEG classification in emotion assessment based on synchronization likelihood. In: Proceedings of the 15th European Signal Processing Conference, Poznan.
5. Bailón, R., Laguna, P., Mainardi, L., Sörnmo, L., 2007. Analysis of heart rate variability using time-varying frequency bands based on respiratory frequency. In: IEEE 29th International Conference on EMBS, Lyon, France.
6. Benoit, A., Bonnaud, L., Caplier, A., Ngo, P., Lawson, L., Trevisan, D., Levacik, V., Mancas, C., Chanel, G., 2006. Multimodal focus attention and stress detection and feedback in an augmented driver simulator. In: 3rd IFIP Conference on Artificial Intelligence Applications & Innovations (AIAI), Athens, Greece.
7. Berntson G.G., Bigger J.T., Eckberg D.L., Grossman P., Kaufmann P.G., Malik M., Nagaraja H.N., Porges S.W., Saul J.P., Stone P.H., and VanderMolen M.W. Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology 34 6 (1997) 623-648
8. Bishop C.M. Pattern Recognition and Machine Learning (2006), Springer, Berlin
9. Blankertz B., Dornhege G., Schafer C., Krepki R., Kohlmorgen J., Muller K.R., Kunzmann V., Losch F., and Curio G. Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG analysis. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11 2 (2003) 127-131
10. Brave S., Nass C., and Hutchinson K. Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human Computer Studies 62 2 (2005) 161-178
11. Chanel, G., Ansari-Asl, K., Pun, T., 2007. Valence-arousal evaluation using physiological signals in an emotion recall paradigm. In: IEEE SMC and International Conference on Systems, Man and Cybernetics, Smart Cooperative Systems and Cybernetics: Advancing Knowledge and Security for Humanity, Montreal, Canada.
12. Chanel G., Kronegg J., Grandjean D., and Pun T. Emotion assessment: arousal evaluation using EEG's and peripheral physiological signals. In: Gunsel B., Tekalp A.M., Jain A.K., and Sankur B. (Eds). Multimedia Content Representation Classification and Security. Springer Lecture Notes in Computer Sciences vol. 4105 (2006) 530-537
13. Chanel, G., Rebetez, C., Bétrancourt, M., Pun, T., 2008. Boredom, engagement and anxiety as indicators for adaptation to difficulty in games. In: 12th International MindTrek Conference: Entertainment and Media in the Ubiquitous Era. ACM, Tampere, Finland.
14. Chang, C.-C., Lin, C.-J., 2001. LIBSVM: a library for support vector machines.
15. Chen J. Flow in games (and everything else)-a well-designed game transports its players to their personal flow zones, delivering genuine feelings of pleasure and happiness. Communications of the ACM 50 4 (2007) 31-34
16. Choi D.H., Kim J., and Kim S.H. ERP training with a web-based electronic learning system: the flow theory perspective. International Journal of Human-Computer Studies 65 3 (2007) 223-243
17. Cohen I., Sebe N., Garg A., Chen L.S., and Huang T.S. Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and Image Understanding 91 1-2 (2003) 160-187
18. Cornelius R.R. The Science of Emotion (1996), Prentice-Hall, Upper Saddle River, NJ
19. Coulson M. Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior 28 2 (2004) 117-139
20. Cowie, R., 2000. Describing the emotional states expressed in speech. In: ISCA Workshop on Speech and Emotion, Northern Ireland.
21. Cowie R., Douglas-Cowie E., Tsapatsoulis N., Votsis G., Kollias S., Fellenz W., and Taylor J.G. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18 1 (2001) 32-80
22. Damasio A.R., Grabowski T.J., Bechara A., Damasio H., Ponto L.L.B., Parvizi J., and Hichwa R.D. Subcortical and cortical brain activity during the feeling of self-generated emotions. Nature Neuroscience 3 10 (2000) 1049-1056
23. Davidson R.J. Affective neuroscience and psychophysiology: toward a synthesis. Psychophysiology 40 5 (2003) 655-665
24. Davidson R.J., Jackson D.C., and Kalin N.H. Emotion, plasticity, context and regulation: perspectives from affective neuroscience. Psychological Bulletin 126 6 (2000) 890-909
25. Devillers L., Vidrascu L., and Lamel L. Challenges in real-life emotion annotation and machine learning based detection. Neural Networks 18 (2005) 407-422
26. Duda R.O., Hart P.E., and Stork D.G. Pattern Classification (2001), Wiley Interscience, New York
27. Ekman P., Friesen W.V., O'Sullivan M., Chan A., Diacoyanni-Tarlatzis I., Heider K., Krause R., LeCompte W.A., Pitcairn T., and Ricci-Bitti P.E. Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology 53 4 (1987) 712-717
28. Ekman P., Levenson R.W., and Friesen W.V. Autonomic nervous-system activity distinguishes among emotions. Science 221 4616 (1983) 1208-1210
29. Fasel B., and Luettin J. Automatic facial expression analysis: a survey. Pattern Recognition 36 1 (2003) 259-275
30. Grandjean D., Sander D., and Scherer K.R. Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Consciousness and Cognition 17 2 (2008) 484-495
31. Guger C., Edlinger G., Harkam W., Niedermayer I., and Pfurtscheller G. How many people are able to operate an EEG-based brain-computer interface (BCI)?. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11 2 (2003) 145-147
32. Haag, A., Goronzy, S., Schaich, P., Williams, J., 2004. Emotion recognition using bio-sensors: first step toward an automatic system. In: Affective Dialog Systems: Tutorial and Research Workshop, Kloster Irsee, Germany.
33. Hanjalic A., and Xu L.-Q. Affective video content representation and modeling. IEEE Transactions on Multimedia 7 1 (2005) 143-154
34. Hazlett R.L., and Benedek J. Measuring emotional valence to understand the user's experience of software. International Journal of Human-Computer Studies 65 4 (2007) 306-314
35. Healey, J.A., 2000. Wearable and automotive systems for affect recognition from physiology. Doctor of Philosophy. Electrical Engineering and Computer Science Department, MIT.
36. Hua J., Xiong Z., Lowey J., Suh E., and Dougherty E.R. Optimal number of features as a function of sample size for various classification rules. Bioinformatics 21 8 (2005) 1509-1515
37. Isomursu M., Tahti M., Vainamo S., and Kuutti K. Experimental evaluation of five methods for collecting emotions in field settings with mobile applications. International Journal of Human-Computer Studies 65 4 (2007) 404-418
38. Kapoor A., Burleson W., and Picard R.W. Automatic prediction of frustration. International Journal of Human-Computer Studies 65 8 (2007) 724-736
39. Kierkels J.J.M., van Boxtel G.J.M., and Vogten L.L.M. A model-based objective evaluation of eye movement correction in EEG recordings. IEEE Transactions on Biomedical Engineering 53 2 (2006) 246-253
40. Kim, J., 2004. Emotion recognition from physiological measurement. In: Humaine European Network of Excellence Workshop, Santorini, Greece.
41. Kim, J., Andre, E., Rehm, M., Vogt, T., Wagner, J., 2005. Integrating information from speech and physiological signals to achieve emotional sensitivity. In: Proceedings of the 9th European Conference on Speech Communication and Technology, Lisboa, Portugal.
42. Kronegg J., Chanel G., Voloshynovskiy S., and Pun T. EEG-based synchronized brain-computer interfaces: a model for optimizing the number of mental tasks. IEEE Transactions on Neural Systems and Rehabilitation Engineering 15 1 (2007) 50-58
43. Lang, P.J., Bradley, M.M., Cuthbert, B.N., 2005. International affective picture system (IAPS): digitized photographs, instruction manual and affective ratings. Technical Report A-6, University of Florida, Gainesville, FL.
44. Lang P.J., Greenwald M.K., Bradley M.M., and Hamm A.O. Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30 3 (1993) 261-273
45. Leon E., Clarke G., Callaghan V., and Sepulveda F. A user-independent real-time emotion recognition system for software agents in domestic environments. Engineering Applications of Artificial Intelligence 20 3 (2007) 337-345
46. Lisetti C.L., and Nasoz F. Using noninvasive wearable computers to recognize human emotions from physiological signals. Journal on Applied Signal Processing 11 (2004) 1672-1687
47. Lotte F., Congedo M., Lecuyer A., Lamarche F., and Arnaldi B. A review of classification algorithms for EEG-based brain-computer interfaces. Journal of Neural Engineering 4 2 (2007) R1-R13
48. Mandryk R.L., and Atkins M.S. A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies 65 4 (2007) 329-347
49. Mandryk R.L., Inkpen K.M., and Calvert T.W. Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology 25 2 (2006) 141-158
50. MettingVanRijn, A.C., Kuiper, A.P., Dankers, T.E., Grimbergen, C.A., 1996. Low-cost active electrode improves the resolution in biopotential recordings. In: 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Amsterdam, The Netherlands.
51. Moddemeijer R. On estimation of entropy and mutual information of continuous distributions. Signal Processing 16 3 (1989) 233-248
52. Norman D.A. The Psychology of Everyday Things (1990), Doubleday/Currency, New York
53. Pantic M., and Rothkrantz L.J.M. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91 9 (2003) 1370-1390
54. Pelzer M., Schipke J.D., Horstkotte D., and Arnold G. Effect of respiration on short-term heart-rate-variability. Faseb Journal 8 5 (1994) A846
55. Picard R.W. Affective Computing (1997), The MIT Press, Cambridge
56. Picard, R.W., Daily, S.B., 2005. Evaluating affective interactions: alternatives to asking what users feel. In: CHI Workshop on Evaluating Affective Interfaces: Innovative Approaches, Portland.
57. Picard R.W., Vyzas E., and Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23 10 (2001) 1175-1191
58. Platt J. Probabilities for SV machines. In: Smola A., Bartlett P., Scholkopf B., and Schuurmans D. (Eds). Advances in Large Margin Classifiers (2000), MIT Press, Cambridge, MA 61-64
59. Pudil, P., Ferri, F., Novovicová, J., Kittler, J., 1994. Floating search methods for feature selection with nonmonotonic criterion functions. In: Proceedings of the IEEE International Conference on Pattern Recognition, vol. 2. pp. 279-283.
60. Rainville P., Bechara A., Naqvi N., and Damasio A.R. Basic emotions are associated with distinct patterns of cardiorespiratory activity. International Journal of Psychophysiology 61 1 (2006) 5-18
61. Rani P., Liu C., and Sarkar N. An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis and Applications 9 (2006) 58-69
62. Rani, P., Sarkar, N., Liu, C., 2005. Maintaining optimal challenge in computer games through real-time physiological feedback. In: 11th HCI International, Las Vegas, USA. Lawrence Erlbaum Associates, Inc.
63. Rolls E.T. Précis of the brain and emotion. Behavioral and Brain Sciences 23 2 (2000) 177-233
64. Romero S., Mananas M.A., and Barbanoj M.J. A comparative study of automatic techniques for ocular artifact reduction in spontaneous EEG signals based on clinical target variables: a simulation case. Computers in Biology and Medicine 38 3 (2008) 348-360
65. Russell J.A. A circumplex model of affect. Journal of Personality and Social Psychology 39 6 (1980) 1161-1178
66. Sakata, T., Watanuki, S., Sakamoto, H., Sumi, T., Kim, Y.-K., 2007. Objective evaluation of Kansei by a complementary use of physiological indexes, brain wave and facial expressions for user oriented designs. In: Proceedings of the10th Qmod Conference, Quality Management and Organisational Development: Our Dreams of Excellence, Helsingborg, Sweden.
67. Salahuddin, L., Cho, J., Jeong, M.G., Kim, D., 2007. Ultra short term analysis of heart rate variability for monitoring mental stress in mobile settings. In: IEEE 29th International conference of the EMBS, Lyon, France.
68. Sander D., Grandjean D., and Scherer K.R. A systems approach to appraisal mechanisms in emotion. Neural Networks 18 4 (2005) 317-352
69. Sanderson C., and Paliwal K.K. Identity verification using speech and face information. Digital Signal Processing 14 5 (2004) 449-480
70. Scherer K.R. Appraisal Considered as a Process of Multi-Level Sequential Checking (2001), Oxford University Press, New York, Oxford
71. Sinha R., Lovallo W.R., and Parsons O.A. Cardiovascular differentiation of emotions. Psychosomatic Medicine 54 4 (1992) 422-435
72. Sinha R., and Parsons O.A. Multivariate response patterning of fear and anger. Cognition & Emotion 10 2 (1996) 173-198
73. Smith A.P.R., Stephhan K.E., Rugg M.D., and Dolan R.J. Task and content modulate amygdala-hippocampal connectivity in emotional retrieval. Neuron 49 4 (2006) 631-638
74. Soleymani M., Chanel G., Kierkels J., and Pun T. Affective characterization of movie scenes based on multimedia content analysis and user's physiological emotional responses. In: IEEE International Symposium on Multimedia, Berkeley, USA (2008)
75. Stemmler G., Heldmann M., Pauls C.A., and Scherer T. Constraints for emotion specificity in fear and anger: the context counts. Psychophysiology 38 2 (2001) 275-291
76. Takahashi, K., 2004. Remarks on emotion recognition from bio-potential signals. In: Proceedings of the 2nd International Conference on Autonomous Robots and Agents, Palmerston North, New Zealand.
77. Tipping M.E. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research 1 3 (2001) 211-244
78. van den Broek E.L., Schut M.H., Westerink J.H.D.M., van Herk J., and Tuinenbreijer K. Computing emotion awareness through facial electromyography. Computer Vision in Human-Computer Interaction 3979 (2006) 52-63
79. Vaughan T.M., Heetderks W.J., Trejo L.J., Rymer W.Z., Weinrich M., Moore M.M., Kubler A., Dobkin B.H., Birbaumer N., Donchin E., Wolpaw E.W., and Wolpaw J.R. Brain-computer interface technology: a review of the second international meeting. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11 2 (2003) 94-109
80. Ververidis D., and Kotropoulos C. Emotional speech recognition: resources, features and methods. Speech Communication 48 9 (2006) 1162-1181
81. Wagner, J., Kim, J., André, E., 2005. From physiological signals to emotions: implementing and comparing selected methods for features extraction and classification. In: IEEE International Conference on Multimedia & Expo.
82. Wu T.F., Lin C.J., and Weng R.C. Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Research 5 (2004) 975-1005
83. Xu Z., John D., and Boucouvalas A.C. Expressive image generation: towards expressive Internet communications. Journal of Visual Languages and Computing 17 5 (2006) 445-465
84. Zeng Z., Tu J., Pianfetti B.M., and Huang T.S. Audio-visual affective expression recognition through multistream fused HMM. IEEE Transactions on Multimedia 10 4 (2008) 570-577
85. Zhang H., and Malik J. Selecting shape features using multi-class relevance vector machine. EECS Department, University of California, Berkeley (2005)