A multimodal emotion detection system during human-robot interaction

Sensors (Switzerland)

Volumn 13, Issue 11, 2013, Pages 15549-15581

  Alonso Martín, Fernando ^a   Malfaz, María ^a   Sequeira, João ^b   Gorostiza, Javier F ^a   Salichs, Miguel A ^a  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

^b INSTITUTE FOR SYSTEMS AND ROBOTICS   (Portugal)

Author keywords

Affective computing; Dialog systems; Emotion recognition; FACS; Human robot interaction

Indexed keywords

COMPUTER KEYBOARDS; EXPERIMENTS; HUMAN ROBOT INTERACTION; MAN MACHINE SYSTEMS; OBJECT RECOGNITION; ROBOTICS;

AFFECTIVE COMPUTING; COMPUTER EXPRESSIONS; DIALOG SYSTEMS; EMOTION RECOGNITION; FACS; INFORMATION CHANNELS; ROBOT OPERATING SYSTEM; SATISFACTION DEGREES;

HUMAN COMPUTER INTERACTION;

EID: 84887740934     PISSN: 14248220     EISSN: None     Source Type: Journal    
DOI: 10.3390/s131115549     Document Type: Article

References (96)

1. Picard, R. Affective Computing for HCI; MIT Press: Munich, Germany, 1999.
2. Alonso-Martín, F.; Gorostiza, J.F.; Salichs, M.A. Descripción General del Sistema de Interacción Humano-Robot Robotics Dialog System (RDS). In Proceedings of the Robocity 2030 12th Workshop: Robótica Cognitiva, Madrid, Spain, July 2013; pp. 1-13.
3. Alonso-Martin, F.; Gorostiza, J.F.; Salichs, M.A. Preliminary Experiments on HRI for Improvement the Robotic Dialog System (RDS). In Proceedings of the Robocity 2030 11th Workshop: Robots Sociales, Leganés, Spain, March 2013.
4. Rich, C.; Ponsler, B. Recognizing Engagement in Human-Robot Interaction. In Proceedings of the 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Osaka, Japan, 2-5 March 2010; pp. 375-382.
5. Arnold, M. Emotion and Personality; Columbia University Press: New York, NY, USA, 1960.
6. Ekman, P.; Friesen, W.; Ellsworth, P. Emotion in the Human Face: Guidelines for Research and an Integration of Findings; Pergamon Press: Oxford, UK, 1972.
7. Plutchik, R. Emotion, a Psychoevolutionary Synthesis; Harper and Row: New York, NY, USA, 1980.
8. Cowie, R.; Douglas-Cowie, E. 'FEELTRACE': An Instrument for Recording Perceived Emotion in Real Time. In Proceedings of the ISCA Tutorial and ResearchWorkshop (ITRW) on Speech and Emotion, Newcastle, Northern Ireland, UK, 5-7 September 2000.
9. Bradley, M.; Lang, P. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 1994, 25, 49-59.
10. Bradley, M.; Lang, P. Measuring Emotion: Behavior, Feeling, and Physiology. In Cognitive Neuroscience of Emotion; Oxford University Press: New York, NY, USA, 2000; pp. 242-276.
11. Chakraborty, A. Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Trans. Syst. Man Cybern. A 2009, 39, 726-743.
12. Cheng, J.; Deng, Y. A Facial Expression Based Continuous Emotional State Monitoring System with GPU Acceleration. In Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China, 22-26 April, 2013; pp. 1-6.
13. Shan, C.; Gong, S.; McOwan, P. Facial expression recognition based on Local Binary Patterns: A comprehensive study. Image Vis. Comput. 2009, 27, 803-816.
14. Cowie, R.; Douglas-Cowie, E.; Tsapatsoulis, N.; Votsis, G.; Kollias, S.; Fellenz, W.; Taylor, J. Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 2001, 18, 32-80.
15. Dar, S.; Khaki, Z. Emotion recognition based on audio speech. IOSR J. Comput. Eng. 2013, 1, 46-50.
16. Pantic, M.; Sebe, N.; Cohn, J.F.; Huang, T. Affective Multimodal Human-Computer Interaction. In Proceedings of the 13th Annual ACM International Conference on Multimedia (MULTIMEDIA '05), New York, NY, USA, 2005; ACM Press: New York, NY, USA, 2005; p. 669.
17. Roy, D.; Pentland, A. Automatic Spoken Affect Classification and Analysis. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, Killington, VT, USA, 14-16 October 1996; IEEE Computer Society Press: Killington, VT, USA. pp. 363-367.
18. Van der Wal, C.; Kowalczyk, W. Detecting changing emotions in human speech by machine and humans. Appl. Intell. 2013, 39, 46-50.
19. Yu, C.; Tian, Q.; Cheng, F.; Zhang, S. Speech Emotion Recognition Using Support Vector Machines. In Advanced Research on Computer Science and Information Engineering; Springer: Berlin/Heidelberg, Germany; pp. 215-220.
20. Bänziger, T.; Grandjean, D.; Scherer, K. Emotion recognition from expressions in face, voice, and body: The multimodal emotion tecognition test (MERT). Emotion 2009, 9, 691-704.
21. Busso, C.; Deng, Z.; Yildirim, S.; Bulut, M. Analysis of Emotion Recognition Using Facial Expressions, Speech and Multimodal Information. In Proceedings of the 6th International Conference on Multimodal Interfaces, New York, NY, USA, 13-15 October 2004; pp. 205-211.
22. Castellano, G.; Kessous, L.; Caridakis, G. Emotion Recognition Through Multiple Modalities: Face, Body Gesture, Speech. In Affect and Emotion in Human-Computer Interaction; Springer: Berlin/Heidelberg, Germany, 2008; pp. 92-103.
23. De Silva, L.; Miyasato, T.; Nakatsu, R. Facial Emotion Recognition Using Multi-Modal Information. In Proceedings of the IEEE 6th International Conference on Information, Communications and Signal Processing, Singapore, 10-13 December 2007; Volume 1, pp. 397-401.
24. Chen, L.; Huang, T.; Miyasato, T.; Nakatsu, R. Multimodal Human Emotion/Expression Recognition. In Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 14-16 April, 1998; pp. 366-371.
25. Tu, B.; Yu, F. Bimodal emotion recognition based on speech signals and facial expression. Found. Intell. Syst. 2012, 122, 691-696.
26. Zeng, Z.; Pantic, M. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 31, 39-58.
27. Yoshitomi, Y.; Kawano, T.; Kilazoe, T. Effect of Sensor fusion for Recognition of Emotional States Using Voice, Face Image and Thermal Image of Face. In Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication (IEEE RO-MAN 2000), Osaka, Japan, 27-29 September 2000; pp. 178-183.
28. Merchant, J. Lymbix Sentiment Analysis Reinvented. 2013.
29. Rao, Y.; Lei, J.; Wenyin, L.; Li, Q.; Chen, M. Building emotional dictionary for sentiment analysis of online news. World Wide Web 2013.
30. Serban, O.; Pauchet, A.; Pop, H. Recognizing Emotions in Short Texts. In Proceedings of the ICAART: 4th International Conference on Agents and Artificial Intelligence, Vilamoura, Portugal, 6-8 February 2012.
31. Alghowinem, S.; Goecke, R.; Wagner, M. Head Pose and Movement Analysis as an Indicator of Depression. In Proceedings of the Fifth Biannual Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII2013), Geneva, Switzerland, 2-5 September 2013.
32. Russell, J.; Dols, J. The Psychology of Facial Expression; Cambridge University Press: Cambridge, UK, 1997.
33. Liu, Y.; Sourina, O.; Nguyen, M. Real-Time EEG-Based Human Emotion Recognition and Visualization. In Proceedings of the 2010 International Conference on Cyberworlds (CW), Singapore, Republic of Singapore, 20-22 October 2010; pp. 262-269.
34. Valderrama Cuadros, C.E.; Ulloa Villegas, G.V. Spectral analysis of physiological parameters for consumers' emotion detection. 2012.
35. Mehrabian, A.; Ferris, S. Inference of attitudes from nonverbal communication in two channels. J. Consult. Psychol. 1967, 31, 248-252
36. Mehrabian, A.; Wiener, M. Decoding of inconsistent communications. J. Personal. Soc. Psychol. 1967, 6, 109-114.
37. Truong, K.; van Leeuwen, D.; Neerincx, M. Unobtrusive Multimodal Emotion Detection in Adaptive Interfaces: Speech and Facial Expressions. In Foundations of Augmented Cognition; Springer: Beijing, China, 2007; pp. 354-363.
38. Busso, C.; Deng, Z.; Yildirim, S.; Bulut, M.; Lee, C.M.; Kazemzadeh, A.; Lee, S.; Neumann, U.; Narayanan, S. Analysis of Emotion Recognition Using Facial Expressions, Speech and Multimodal Information. In Proceedings of the 6th International Conference on Multimodal Interfaces, (ICMI '04); ACM Press: New York, NY, USA, 2004; p. 205.
39. Sebe, N.; Cohen, I.; Gevers, T.; Huang, T. Emotion Recognition Based on Joint Visual and Audio Cues. In Proceedings of the IEEE 18th International Conference on Pattern Recognition, (ICPR'06), Hong Kong, 20-24 August 2006; pp. 1136-1139.
40. Yildirim, S.; Narayanan, S.; Potamianos, A. Detecting emotional state of a child in a conversational computer game. Comput. Speech Lang. 2011, 25, 29-44.
41. Microsoft Kinect. Available online at http://en.wikipedia.org/wiki/Kinect#Kinect on the Xbox-One (accessed on 2 November 2013).
42. Toledo-Ronen, O.; Sorin, A. Emotion Detection for Dementia Patients Monitoring. In Proceedings of the Afeka Center for Language Procesing (ACLP'2012), Haifa, Israel, 1-2 July 2012.
43. Komlósi, S.; Csukly, G.; Stefanics, G.; Czigler, I. Fearful face recognition in schizophrenia: An electrophysiological study. Schizophr. Res. 2013, 149, 135-140.
44. Harms, M.; Martin, A.; Wallace, G. Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies. Neuropsychol. Rev. 2010, 20, 290-322.
45. Petrushin, V. Emotion in Speech: Recognition and Application to Call Centers. In Proceedings of the Artificial Neural Networks in Engineering, 1999; pp. 7-10.
46. Inanoglu, Z.; Caneel, R. Emotive Alert: HMM-Based Emotion Detection in Voicemail Messages. In Proceedings of the 10th International Conference on Intelligent User Interfaces, New York, NY, USA, 9-12 January 2005; pp. 251-253.
47. Ramírez, A.P. Uso del Cry Translator (Traductor del llanto del bebé) de Biloop Technologic SL (España) como identificador del llanto en el niño y pautas a seguir. Ped. Rur. Ext. 2009, 39.
48. XML. Available online: http://en.wikipedia.org/wiki/Extensible Markup Language (accessed on 2 November 2013).
49. VoiceXML. Available online: http://en.wikipedia.org/wiki/VoiceXML (accessed on 2 November 2013).
50. Bladeware. Available online: http://sourceforge.net/projects/bladeware-vxml (accessed on 2 November 2013).
51. Litman, D.J.; Forbes-Riley, K. Predicting Student Emotions in Computer-Human Tutoring Dialogues. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, (ACL '04), Barcelona, Spain, 21-26 July. 2004; p. 351.
52. Cowie, R.; Douglas-Cowie, E.; Romano, Changing Emotional Tone in Dialogue and its Prosodic Correlates. In Proceedings of the ESCA Tutorial and Research Workshop (ETRW) on Dialogue and Prosody, Veldhoven, Netherlands, 1-3 September 1999.
53. Eyben, F.; Wöllmer, M.; Schuller, B. Opensmile: The Munich versatile and fast open-source audio feature extractor. In Proceedings of the International Conference on Multimedia (MM '10), Firenze, Italy, 25-29 October 2010; ACM Press: New York, NY, USA, 2010; p. 1459.
54. Eyben, F.; Wollmer, M.; Schuller, B. OpenEAR-Introducing the Munich Open-Source Emotion and Affect Recognition Toolkit. In Proceedings of the IEEE 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, Netherlands, 10-12 September 2009; pp. 1-6.
55. Boersma, P. Praat, a system for doing phonetics by computer. Glot Int. 2002, 5, 341-345.
56. Verbio. Available online: http://www.verbio.com/webverbio3/es/tecnologia/verbio-tts/ 52-verbio-speech-analytics.html (accessed on 2 November 2013).
57. Chuck. Available online at (http://chuck.cs.princeton.edu (accessed on 2 November 2013).
58. Fiebrink, R.; Cook, P. A Meta-Instrument for Interactive, On-the-fly Machine Learning. In Proceedings of the New Interfaces for Musical Expression 9th International Conference, Pittsburgh, PA, USA, 4-6 June 2009. Software Available online: https://code.google.com/p/wekinator/downloads/detail?name=wekinator-2011-06-06.zip&can=2& q= (accessed on 30 June 2011).
59. Discrete Wavelet Transform. Available online: http://www.cs.ucf.edu/mãli/haar (accessed on 2 November 2013).
60. Cheveigné, A.D.; Kawahara, H. YIN, A fundamental frequency estimator for speech and music. J. Acoust. Soc. Am. 2002, 111, 1917-1930.
61. McLeod, P.; Wyvill, G. A Smarter Way to Find Pitch. In Proceedings of International Computer Music Conference (ICMC), Barcelona, Spain, September 2005.
62. Larson, E.; Maddox, R. Real-Time Time-Domain Pitch Tracking Using Wavelets. In Proceedings of the University of Illinois at Urbana Champaign Research Experience for Undergraduates Program, Champaign, IL, USA, 3 August 2005.
63. Callejas, Z.; Lopezcozar, R. Influence of contextual information in emotion annotation for spoken dialogue systems. Speech Commun. 2008, 50, 416-433.
64. Vlasenko, B.; Schuller, B. Combining Frame and Turn-Level Information for Robust Recognition of Emotions Within Speech. In Proceedings of Interspeech, Antwerp, Belgium, 27-31 August 2007.
65. Schuller, B.; Arsic, D. Emotion Recognition in the Noise Applying Large Acoustic Feature Sets. In Proceedings of the Speech Prosody, Dresden, Germany, 2-5 May. 2006. pp. 276-289.
66. Steidl, S. Automatic Classification of Emotion Related User States in Spontaneous Children's Speech. Ph.D. thesis, University of Erlangen-Nürnberg, Germany, November 2009.
67. FAU. Available online: http://www5.cs.fau.de/de/mitarbeiter/steidl-stefan/fau-aibo-emotioncorpus (accessed on 2 November 2013).
68. Holmes, G.; Donkin, A.; Witten, I. WEKA: A Machine Learning Workbench. In Proceedings of the IEEE Australian New Zealand Intelligent Information Systems Conference (ANZIIS '94), Brisbane, Australia, 29 November 1994; pp. 357-361.
69. Bartlett, M.S.; Littlewort, G.; Fasel, I.; Movellan, J.R. Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. In Proceedings of the IEEE 2003 Conference on Computer Vision and Pattern RecognitionWorkshop, Madison, WI, USA, 16-22 June. 2003; p. 53.
70. Pantie, M.; Rothkrantz, L. Automatic analysis of facial expressions: The state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1424-1445.
71. Viola, P.; Jones, M. Robust real-time face detection. Int. J. Comput. Vis. 2004, 57, 137-154.
72. OpenCV. Available online: http://docs.opencv.org/trunk/modules/contrib/doc/facerec (accessed on 2 November 2013).
73. Kobayashi, H.; Hara, F. Facial Interaction Between Animated 3D Face Robot and Human Beings. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, Orlando, FL, USA, 12-15 October 1997; Volume 4, pp. 3732-3737.
74. Padgett, C.; Cottrell, G. Representing Face Images for Emotion Classification. In Advances in Neural Information Processing Systems; The MIT Press: Denver, CO, USA, 2-5 December 1997.
75. Cootes, T.; Edwards, G.; Taylor, C. Active Appearance Models. In Proceedings of the Computer Vision, (ECCV'98), Freiburg, Germany, 7 August 1998; Volume 23. pp. 484-498.
76. Lucey, S.; Matthews, I.; Hu, C. AAM Derived Face Representations for Robust Facial Action Recognition. In Proceedings of the 7th International Conference on in Automatic Face and Gesture Recognition, Southampton, UK, 10-12 April. 2006.
77. Terzopoulos, D.; Waters, K. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans. Pattern Anal. Mach. Intell. 1993, 15, 569-579
78. Kearney, G.; McKenzie, S. Machine interpretation of emotion: Design of a memory-based expert system for interpreting facial expressions in terms of signaled emotions. Cogn. Sci. 1993, 17, 589-622.
79. Bassili, J. Facial motion in the perception of faces and of emotional expression. J. Exp. Psychol.: Hum. Percept. Perform. 1978, 4, 373-379.
80. Ekman, P.; Friesen, W.; Hager, J. Facial Action Coding System: A Technique for the Measurement of Facial Movement; Consulting Psychologists Press: Palo Alto, CA, USA, 1978.
81. A Modular Framework to Detect and Analyze Faces for Audience Measurement Systems. In Proceedings of the 2nd Workshop on Pervasive Advertising In Conjuction with Informatik 2009, Lübeck, Germany, 2 October 2009; pp. 75-87.
82. Izard, C.E. The Maximally Discriminative Facial Movement Coding System; University of Delaware: Newark, DE, USA, 1983.
83. Izard, C.; Dougherty, L.; Hembree, E. A System for Identifying Affect Expressions by Holistic Judgments (Affex); University of Delaware: Newark, DE, USA, 1983.
84. Pelachaud, C.; Badler, N.; Viaud, M. Final Report to NSF of the Standards for Facial Animation Workshop. University of Pennsylvania Institute for Research in Cognitive Science Technical Report No. IRCS-94-21. Available online: http://repository.upenn.edu/ircs reports/167/ (accessed on 2 November 2013).
85. Ekman, P.; Friesen, W. Constants across cultures in the face and emotion. J. Personal. Soc. Psychol. 1971, 17, 124-129.
86. Bruce, V.; Voi, M.E.L. Recognizing faces [and discussion]. Philos. Trans. R. Soc. B: Biol. Sci. 1983, 302, 423-436.
87. Izard, C. Facial expressions and the regulation of emotions. J. Personal. Soc. Psychol. 1990, 58, 487-98.
88. Tong, Y.; Liao, W.; Ji, Q. Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 29, 1683-1699.
89. Littlewort, G.; Whitehill, J.;Wu, T.-F.; Butko, N.; Ruvolo, P.; Movellan, J.; Bartlett, M. The Motion in Emotion-A CERT Based Approach to the FERA Emotion Challenge. In Proceedings of the IEEE Face and Gesture 2011, Santa Barbara, CA, USA, 21-25 March. 2011; pp. 897-902.
90. CERT. Available online: http://mpt4u.com/AFECT (accessed on 2 November 2013).
91. Ernst, A.; Ruf, T.; Kueblbeck, C. A Modular Framework to Detect and Analyze Faces for Audience Measurement Systems. In Proceedings of the 2nd Workshop on Pervasive Advertising at Informatik, Lübeck, Germany, 2 October 2009; pp. 75-87.
92. Küblbeck, C.; Ernst, A. Face detection and tracking in video sequences using the modifiedcensus transformation. Image Vis. Comput. 2006, vol 24, 564-572.
93. SHORE demostration. Available online: http://www.iis.fraunhofer.de/en/bf/bsy/produkte/ shore.html (accessed on 2 November 2013).
94. Wierzbicki, R.; Tschoeppe, C.; Ruf, T.; Garbas, J. EDIS-Emotion-Driven Interactive Systems. In Proceedings of the Semantic Ambient Media Workshop in Conjunction with Pervasive, Newcastle, UK, 2012.
95. Alonso-Martin, F.; Gorostiza, J.F.; Salichs, M.A. Musical Expression in a Social Robot. In Proceedings of the 2012 International IEEE Intelligent Vehicles Symposium. Workshops V Perception in Robotics, Alcalá de Henares, Spain, June 2012.
96. Salichs, M.; Barber, R.; Khamis, A.; Malfaz, M.; Gorostiza, J.; Pacheco, R.; Rivas, R.; Corrales, A.; Delgado, E.; Garcia, D. Maggie: A Robotic Platform for Human-Robot Social Interaction. In Proceedings of the 2006 IEEE Conference on Robotics, Automation and Mechatronics, Bangkok, Thailand, 1-3 June 2006.