Toward emotion aware computing: An integrated approach using multichannel neurophysiological recordings and affective visual stimuli

IEEE Transactions on Information Technology in Biomedicine

Volumn 14, Issue 3, 2010, Pages 589-597

  Frantzidis, Christos A ^a   Bratsas, Charalampos ^a   Papadelis, Christos L ^b   Konstantinidis, Evdokimos ^a   Pappas, Costas ^a   Bamidis, Panagiotis D ^a  

^a ARISTOTLE UNIVERSITY OF THESSALONIKI   (Greece)

^b UNIVERSITY OF TRENTO   (Italy)

Author keywords

Affective computing; EEG; Emotion theory; Human computer interaction (HCI); Mahalanobis; Neurophysiological recordings; Support vector machines (SVMs)

Indexed keywords

AFFECTIVE COMPUTING; CLASSIFICATION MODELS; CLASSIFICATION PROCEDURE; CLASSIFICATION RATES; DISTANCE-BASED; EEG SIGNALS; EMOTION THEORIES; EMOTIONAL STATE; INTEGRATED APPROACH; MAHALANOBIS; MULTI-CHANNEL; NEUROPHYSIOLOGICAL RECORDINGS; PICTURE SYSTEM; ROBUST CLASSIFICATION; VISUAL STIMULUS;

ELECTROENCEPHALOGRAPHY; KNOWLEDGE MANAGEMENT; SPHERES; SUPPORT VECTOR MACHINES;

HUMAN COMPUTER INTERACTION;

ADULT; ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; COMPUTER INTERFACE; ELECTROENCEPHALOGRAPHY; EMOTION; EVOKED VISUAL RESPONSE; FEEDBACK SYSTEM; FEMALE; HUMAN; MALE; METHODOLOGY; PHOTOSTIMULATION; PHYSIOLOGY; SIGNAL PROCESSING;

ADULT; ALGORITHMS; ARTIFICIAL INTELLIGENCE; ELECTROENCEPHALOGRAPHY; EMOTIONS; EVOKED POTENTIALS, VISUAL; FEEDBACK; FEMALE; HUMANS; MALE; PHOTIC STIMULATION; SIGNAL PROCESSING, COMPUTER-ASSISTED; USER-COMPUTER INTERFACE;

EID: 77953158309     PISSN: 10897771     EISSN: None     Source Type: Journal    
DOI: 10.1109/TITB.2010.2041553     Document Type: Article

References (57)

1. M. Pantic and L. J.M. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction," Proc. IEEE, vol.91, no.9, pp. 1370-1390, Sep. 2003.
2. S. Petridis andM. Pantic, "Audiovisual laughter detection based on temporal features," in Proc. 10th Int. Conf. Multimodal Interfaces (ICMI 2008), Chania, Greece, Oct. 20-22, pp. 37-44.
3. C. Lisetti, F. Nasoz, C. LeRouge, O. Ozyer, and K. Alvarez, "Developing multimodal intelligent affective interfaces for tele-home health care," Int. J. Hum.-Comput. Stud., vol.59, no.1-2, pp. 245-255, Jul. 2003.
4. S. Favre, H. Salamin, J. Dines, and A. Vinciarelli, "Role recognition in multiparty recordings using social affiliation networks and discrete distributions," in Proc. 10th Int. Conf.Multimodal Interfaces (ICMI 2008), Chania, Greece, Oct. 20-22, pp. 29-36.
5. F. Pianesi, N. Mana, A. Cappelletti, B. Lepri, and M. Zancanaro, "Multimodal recognition of personality traits in social interactions," in Proc. 10th Int. Conf. Multimodal Interfaces (ICMI 2008), Chania, Greece, Oct. 20-22, pp. 29-36.
6. C. A. Frantzidis, C. Bratsas, M. Klados, E. Konstantinidis, C. D. Lithari, A. B. Vivas, C. Papadelis, E. Kaldoudi, C. Pappas, and P. D. Bamidis, "On the classification of emotional biosignals evoked while viewing affective pictures: An integrated data mining based approach for healthcare applications," IEEE Trans. Inf. Technol. Biomed., vol.14, no.2, 2010. DOI: 10.1109/TITB.2009.2038481.
7. A. Luneski, E. I. Konstantinidis, M. Hitoglou-Antoniadou, and P. D. Bamidis, "Affective computer-aided learning for autistic children," presented at the 1stWorkshop Child, Comput. Interact. (ICMI 2008), Chania, Greece, Oct. 20-23.
8. E. Hudlicka, "To feel or not to feel: The role of affect in human-computer interaction," Int. J. Hum.-Comput. Stud., vol.59, no.1-2, pp. 1-32, Jul. 2003.
9. B. Reeves and C. Nass, The Media Equation: How People Treat Computers. Cambridge, U.K.: Cambridge Univ. Press, 1996.
10. R. W. Picard, "Affecting Computing for HCI," in Proc. 8th Int. Conf. Human-Comput. Interact.: Ergonom. User Interfaces, vol.I, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1999.
11. P. D. Bamidis, C. A. Frantzidis, E. I.Konstantinidis,A. Luneski, C. Lithari, M. A. Klados, C. Bratsas, C. L. Papadelis, and C. Pappas, "An integrated approach to emotion recognition for advanced emotional intelligence," in Lecture Notes in Computer Science, Affective Dialogue Systems, vol.3068, Berlin,Germany/Heidelberg: Springer-Verlag, 2009, pp. 36-48.
12. J. A. Russell, "A circumplex model of affect," J. Pers. Soc. Psychol., vol.39, no.6, pp. 1161-1178, Dec. 1980.
13. D. Watson and A. Tellegen, "Toward a consensual structure of mood," Psychol. Bull., vol.98, no.2, pp. 219-235, Sep. 1985.
14. M. M. Bradley, M. Codispoti, B. N. Cuthbert, and P. J. Lang, "Emotion and motivation. I: Defensive and appetitive reactions in picture processing," Emotion, vol.1, no.3, pp. 276-298, Sep. 2001.
15. P. J. Lang, M. M. Bradley, and B. N. Cuthbert, "International affective picture system (IAPS): Technical manual and affective ratings," NIMH Center for the Study of Emotion and Attention, Gainesville, FL, 1997.
16. H. T. Schupp, "Affective picture processing: The late positive potential is modulated by motivational relevance," Psychophysiology, vol.37, no.2, pp. 257-261, Mar. 2003.
17. A. Keil, M. M. Bradley, O. Hauk, B. Rockstroh, T. Elbert, and P. J. Lang, "Large-scale neural correlates of affective picture processing," Psychophysiology, vol.39, no.5, pp. 641-649, Mar. 2003.
18. A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion recognition using bio-sensors: First steps towards an automatic system," in Lecture Notes in Computer Science, Affective Dialogue Systems, vol.3068, Berlin, Germany/Heidelberg: Springer-Verlag, 2004, pp. 36-48.
19. G. Chanel, J. Kronegg, D. Grandjean, and T. Pun, "Emotion assessment: Arousal evaluation using EEG's and peripheral physiological signals," in Lecture Notes in Computer Science, Multimedia Content Representation, Classification and Security. vol.4105, Berlin, Germany/Heidelberg: Springer-Verlag, 2006, pp. 530-537.
20. S. V. Ioannou, A. T. Raouzaiou, V. A. Tzouvaras, T. P. Mailis, K. C. Karpouzis, and S. D. Kollias, "Emotion recognition through facial expression analysis based on a neurofuzzy network," Neural Netw., vol.18, no.4, pp. 423-435, May 2005.
21. M.-H. Yang, D. J. Kriegman, and N. Ahuja, "Detecting faces in images: A survey," IEEE Trans. Pattern Anal.Mach. Intell., vol.24, no.1, pp. 34-58, Jan. 2002.
22. P.-Y Oudeyer, "The production and recognition of emotions in speech: Features and algorithms," Int. J. Hum.-Comput. Stud., vol.59, no.1-2, pp. 157-183, Jul. 2003.
23. V. I. Pavlovic, "Visual interpretation of hand gestures for human-computer interaction: A review," IEEE Trans. Pattern Anal. Mach. Intell., vol.19, no.7, pp. 677-695, Jul. 1997.
24. M. Miki, C. Miyajima, T. Nishino, N. Kitaoka, and K. Takeda, "An integrative recognition method for speech and gestures," in Proc. 10th Int. Conf. Multimodal Interfaces (ICMI 2008), Chania, Greece, Oct. 20-22, pp. 93-96.
25. W. Hu, T. Tan, L.Wang, andA.Maybank, "A survey on visual surveillance of object motion and behaviors," IEEE Trans. Syst.,Man, Cybern.C, Appl. Rev., vol.34, no.3, pp. 334-362, Aug. 2004.
26. A. T. Duchowski, "A breadth-first survey of eye-tracking applications," Behav. Res. Methods, Instrum. Comput., vol.34, no.4, pp. 455-470, Nov. 2002.
27. R. W. Picard, E. Vyzas, and J. Healey, "Toward machine emotional intelligence: Analysis of affective physiological state," IEEE Trans. Pattern Anal. Mach. Intell., vol.23, no.10, pp. 1175-1191, Oct. 2001.
28. B. N. Cuthbert, H. T. Schupp, M. M. Bradley, N. Birbaumer, and P. J. Lang, "Brain potentials in affective picture processing: Covariation with autonomic arousal and affective report," Biol. Psychol., vol.52, no.2, pp. 95-111, Mar. 2000.
29. C. Amrhein, A. Muhlberger, P. Pauli, G. Wiedermann et al., "Modulation of event-related brain potentials during affective picture processing: A complement to startle reflex and skin conductance response?," Int. J. Psychophysiol., vol.54, no.3, pp. 231-240, Nov. 2004.
30. P. J. Lang, M. M. Bradley, and B. N. Cuthbert, "Motivated attention: Affect, activation and action," in Attention and Orienting: Sensory and Motivational Processes, P. J. Lang, R. F. Simons, and M. T. Balaban, Eds. Hillsdale, NJ: Erlbaum, 1997.
31. A. Gasbarri, B. Arnone, A. Pompili, F. Pacitti, C. Pacitti, and L. Cahill, "Sex related hemispheric lateralization of electrical potentials evoked by arousing negative stimuli," Brain Res., vol.1138, pp. 178-186, Mar. 2007.
32. B. Rozenkrants and J. Polich, "Affective ERP processing in a visual oddball task: Arousal, valence and gender," Clin. Neurophysiol., vol.119, no.10, pp. 2260-2265, Oct. 2008.
33. C. Lithari, C. Frantzidis, C. Papadelis, A. B. Vivas, M. Klados, C. Kourtidou-Papadeli, C. Pappas, A. A. Ioannides, and P. D. Bamidis, "Are females more sensitive to emotional stimuli? A neurophysiological study across arousal and valence dimensions," Brain Topography, vol.23, no.1, pp. 27-40, Mar. 2010.
34. J. Healey and R. W. Picard, "Digital processing of affective signals," in Proc. 1998 IEEE Int. Conf. Acoust., Speech Signal Process., May 12-15, vol.6, pp. 3749-3752.
35. F. Nasoz, C. L. Lisetti, K. Alvarez, and N. Finkelstein, "Emotion recognition from physiological signals for user modeling of affect," presented at the 3rd Workshop Affective and Attitude User Modeling, Pittsburgh, PA, 2003.
36. The Affection Project: Enabling multi-channel physiological sensing for direct adaptation of computer user interfaces in affective interaction paradigms. (2008, Nov.). [Online]. Available: http://kedip.med.auth.gr/ affection
37. A. Delaurme and S. Makeig, "EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis," J. Neurosci. Methods, vol.134, no.1, pp. 9-21, Mar. 2004.
38. A. G. Correa, E. Laciar, H. D. Patiño, and M. E. Valentinuzzi, "Artifact removal from EEG signals using adaptive filters in cascade," presented at the 16th Argentine Bioeng. Congr. The 5th Conf. Clin. Eng., J. Phys.: Conf. Series. Bristol, U.K.: IOP Publishing, 2007, vol.90, 012081. DOI: 10.1088/1742-6596/90/1/012081.
39. M. A. Klados, C. Frantzidis, A. B. Vivas, C. Papadelis, C. Pappas, and P. D. Bamidis. (2009, Jul. 8). A framework combining delta eventrelated oscillations (EROs) and synchronization effects (ERD/ERS) to study emotional processing, Comput. Intell. Neurosci. [Online],Available: http://www.hindawi.com
40. G. A. Giannakakis, N. N. Tsiaparas, M.-F. S. Xenikou, C. Papageorgiou, and K. S. Nikita, "Wavelet entropy differentiations of event related potentials in dyslexia," presented at the 8th IEEE Int. Conf. BioInformatics Bioeng. (BIBE 2008), Athens, Greece, Oct. 8-10.
41. R. Q. Quiroga, O. W. Sakowitz, E. Basar, and M. Schurmann, "Wavelet transform in the analysis of the frequency composition of evoked potentials," Brain Res. Protoc., vol.8, no.1, pp. 16-24, Aug. 2001.
42. H. Adeli, Z. Zhou, and N. Dadmehr, "Analysis of EEG records in an epileptic patient usingwavelet transform," J. Neurosci. Methods, vol.123, no.1, pp. 69-87, Feb. 2003.
43. S. G. Mallat, "Multiresolution approximations and wavelet orthonormal bases of L2(R)," Trans. Amer. Math. Soc., vol.315, no.1, pp. 69-87, Sep. 1989.
44. B. Krishnapuram, "A Bayesian approach to joint feature selection and classifier design," IEEE Trans. Pattern Anal. Mach. Intell., vol.26, no.9, pp. 1105-1111, Sep. 2004.
45. E. Kaimakamis, C. Bratsas, L. Sichletidis, C. Karvounis, and N. Maglaveras, "Screening of patients with obstructive sleep apnea syndrome using C4.5 algorithm based on nonlinear analysis of respiratory signals during sleep," presented at the 31st IEEE Int. Conf. Eng. Med. Biol. Soc., Minneapolis, MN, Sep. 2-6, 2009, Piscataway, NJ: IEEE Press.
46. I. Guyon, J.Weston, S. Barnhill, and V. Vapnik, "Gene selection for cancer classification using support vector machines," Mach. Learn., vol.46, no.1-3, pp. 389-422, Jan. 2002.
47. C. A. Frantzidis, C. D. Lithari, A. B. Vivas, C. L. Papadelis, C. Pappas, and P. D. Bamidis, "Towards emotion aware computing: A study of arousal modulation with multichannel event related potentials, delta oscillatory activity and skin conductiviy responses," presented at the 8th IEEE Int. Conf. BioInformatics Bioeng. (BIBE 2008), Athens, Greece, Oct. 8-10.
48. C. Cortes and V. Vapnik, "Support-vector networks," Mach. Learn., vol.20, no.3, pp. 273-297, Sep. 1995.
49. C. D. Katsis,N.Katertsidis,G.Ganiatsas, and D. I. Fotiadis, "Toward emotion recognition in car-racing drivers: A biosignal processing approach," IEEE Trans. Syst.,Man, Cybern. A, Syst. Hum., vol.38, no.3, pp. 502-512, May 2008.
50. M. M. Bradley, M. Codispoti, D. Sabatinelli, and P. J. Lang, "Emotion and motivation. I: Defensive and appetitive reactions in picture processing," Emotion, vol.1, no.3, pp. 300-319, Sep. 2001.
51. P. J. Lang, M. M. Bradley, and B. N. Cuthbert, "Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology," Biol. Psychiatry, vol.44, no.12, pp. 1248-1263, Dec. 1998.
52. F. Cincotti, D. Mattia, C. Babiloni, F. Carducci, S. Salinari, L. Bianchi, M. G. Marciani, and F. Babiloni, "The use of EEG modifications due to motor imagery for brain-computer interfaces," IEEE Trans. Neural Syst. Rehabill. Eng., vol.11, no.2, pp. 131-133, Jun. 2003.
53. P. D. Bamidis, C. Papadelis, C. Kourtidou-Papadeli, C. Pappas, and A. B. Vivas, "Affective computing in the era of contemporary neurophysiology and health informatics," Interacting Comput., vol.16, no.4, pp. 715-721, Aug. 2004.
54. K. H. Kim, S. W. Bang, and S. R. Kim, "Emotion recognition system using short-term monitoring of physiological signals," Med. Biol. Eng. Comput., vol.42, no.3, pp. 419-427, Apr. 2006.
55. C. L. Lisetti and F. Nasoz, "Using noninvasive wearable computers to recognize human emotions from physiological signals," EURASIP J. Appl. Signal Process., vol.2004, no.11, pp. 1672-1687, Apr. 2004.
56. S. K. D'Mello, S. D. Craig, J. Sullins, and A. C. Graesser, "Predicting affective states expressed through an emote-aloud procedure from autotutor's mixed-initiative dialogue," Int. J. Artif. Intell. Educ., vol.16, no.1, pp. 3-28, Jan. 2006.
57. M. N. K. Boulos and S. Wheeler, "The emerging Web 2.0 social software: An enabling suite of sociable technologies in health and health care education," Health Inf. Libraries J., vol.24, no.1, pp. 2-23, Mar. 2007.