Affective state level recognition in naturalistic facial and vocal expressions

IEEE Transactions on Cybernetics

Volumn 44, Issue 3, 2014, Pages 315-318

  Meng, Hongying ^a   Bianchi Berthouze, Nadia ^b  

^a BRUNEL UNIVERSITY   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Affective computing; continuous emotion recognition; dimensional model of affect; HMM; machine learning; naturalistic affective expressions

Indexed keywords

AFFECT; ALGORITHM; AUTOMATED PATTERN RECOGNITION; BIOLOGICAL MODEL; BIOMETRY; COMPUTER SIMULATION; FACIAL EXPRESSION; MACHINE LEARNING; PHOTOGRAPHY; PHYSIOLOGY; PROCEDURES; SPEECH; SPEECH ANALYSIS; STATISTICAL MODEL; VIDEORECORDING;

AFFECT; ALGORITHMS; BIOMETRIC IDENTIFICATION; COMPUTER SIMULATION; FACIAL EXPRESSION; MACHINE LEARNING; MODELS, BIOLOGICAL; MODELS, STATISTICAL; PATTERN RECOGNITION, AUTOMATED; PHOTOGRAPHY; SPEECH; SPEECH PRODUCTION MEASUREMENT; VIDEO RECORDING;

EID: 84894794588     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2013.2253768     Document Type: Article

References (74)

1. R. W. Picard, Affective Computing. Cambridge, MA, USA: The MIT Press, 1997.
2. M. E. Ayadi, M. S. Kamel, and F. Karray, "Survey on speech emotion recognition: Features, classification schemes, and databases," Pattern Recognit., vol. 44, no. 3, pp. 572-587, 2011.
3. B. Schuller, A. Batliner, S. Steidl, and D. Seppi, "Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge," Speech Commun., vol. 53, no. 9-10, pp. 1062-1087, 2011.
4. M. Pantic and L. J. Rothkrantz, "Automatic analysis of facial expressions: The state of the art," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, pp. 1424-1445, Dec. 2000. (Pubitemid 32088469)
5. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
6. H. Meng, B. Romera-Paredes, and N. Bianchi-Berthouze, "Emotion recognition by two view SVM 2K classifier on dynamic facial expression features," in Proc. FG, Mar. 2011, pp. 854-859.
7. P. De Silva, A. Kleinsmith, and N. Bianchi-Berthouze, "Towards unsupervised detection of affective body posture nuances affective computing and intelligent interaction," in Proc. LNCS, vol. 3784. 2005, pp. 32-39. (Pubitemid 43763308)
8. C. Shan, S. Gong, and P. W. McOwan, "Beyond facial expressions: Learning human emotion from body gestures," in Proc. BMVC, 2007.
9. H. Gunes and M. Piccardi, "Automatic temporal segment detection and affect recognition from face and body display," IEEE Trans. Syst. Man Cybern. B, vol. 39, no. 1, pp. 64-84, Feb. 2009.
10. A. Kleinsmith and N. Bianchi-Berthouze, "Affective body expression perception and recognition: A survey," IEEE Trans. Affective Comput., vol. 99, 2012, prePrint.
11. Y. Gao, N. Bianchi-Berthouze, and H. Meng, "What does touch tell us about emotions in touchscreen-based gameplay?" ACM Trans. Comp. Human Interaction, vol. 19, no. 4, article no. 31, Dec. 2012.
12. R. L. Mandryk, K. M. Inkpen, and T. W. Calvert, "Using psychophysiological techniques to measure user experience with entertainment technologies," Behaviour IT, vol. 25, no. 2, pp. 141-158, 2006. (Pubitemid 43065344)
13. L. Devillers, C. Vaudable, and C. Chastagnol, "Real-life emotionrelated states detection in call centers: A cross-corpora study," in Proc. INTERSPEECH, 2010, pp. 2350-2353.
14. M. Nicolaou, H. Gunes, and M. Pantic, "Designing frameworks for automatic affect prediction and classification in dimensional space," in Proc. Workshop Gesture Recognit. (CVPR), 2011, pp. 20-26.
15. P. Lucey, J. F. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, and K. M. Prkachin, "Automatically detecting pain in video through facial action units," IEEE Trans. Syst. Man Cybern. B, vol. 41, no. 3, pp. 664-674, Jun. 2011.
16. B. Schuller, R. Müller, F. Eyben, J. Gast, B. Hörnler, M. Wöllmer, G. Rigoll, A. Höthker, and H. Konosu, "Being bored? recognising natural interest by extensive audiovisual integration for real-life application," Image Vision Comput., vol. 27, no. 12, pp. 1760-1774, Nov. 2009.
17. A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed, "Automatic recognition of non-acted affective postures," IEEE Trans. Syst. Man Cybern. B, vol. 41, no. 4, pp. 1027-1038, Aug. 2011.
18. J. Sanghvi, G. Castellano, I. Leite, A. Pereira, P. W. McOwan, and A. Paiva, "Automatic analysis of affective postures and body motion to detect engagement with a game companion," in Proc. 6th Int. Conf. Human-Robot Interaction, 2011, pp. 305-312.
19. N. Savva, A. Scarinzi, and N. Bianchi-Berthouze, "Continuous recognition of player's affective body expression as dynamic quality of aesthetic experience," IEEE Trans. Comput. Intell. AI Games, vol. 4, no. 3, pp. 199-212, Jun. 2012.
20. M. Thrasher, M. D. Van der Zwaag, N. Bianchi-Berthouze, and J. H. D. M. Westerink, "Mood recognition based on upper body posture and movement features," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 377-386.
21. N. Bianchi-Berthouze, "Understanding the role of body movement in player engagement," Human-Computer Interaction, vol. 28, no. 1, pp. 40-75, 2013.
22. C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, and A. Walton, "Measuring and defining the experience of immersion in games," Int. J. Human-Comput. Stud., vol. 66, no. 9, pp. 641-661, Sep. 2008.
23. N. Bianchi-Berthouze, P. Cairns, and A. L. Cox, "On posture as a modality for expressing and recognizing emotions," in Proc. Emotion HCI, 2008, pp. 74-80.
24. G. N. Yannakakis, J. Hallam, and H. H. Lund, "Entertainment capture through heart rate activity in physical interactive playgrounds," User Modeling User-Adapted Interaction, vol. 18, no. 1-2, pp. 207-243, 2008. (Pubitemid 351185637)
25. P. Cairns, A. Cox, N. Berthouze, S. Dhoparee, and C. Jennett, "Quantifying the experience of immersion in games," in Proc. Cognitive Sci. Games Gameplay workshop Cognitive Sci., 2006.
26. N. Bianchi-Berthouze, "Mining multimedia subjective feedback," J. Intell. Inf. Syst., vol. 19, no. 1, pp. 43-59, Jul. 2002. (Pubitemid 34731842)
27. M. Schröder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. Maat, G. Mckeown, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. Sevin, M. F. Valstar, and M. Wöllmer, "Building autonomous sensitive artificial listeners," IEEE Trans. Affective Comput., vol. 3, no. 2, pp. 165-183, Sep. 2012.
28. S. Steidl, A. Batliner, and B. Schuller, "The hinterland of emotions: Facing the open-microphone challenge," in Proc. 3rd Int. Conf. Affective Comput. Intell. Interaction Workshops., 2009, pp. 1-8.
29. A. Batliner, S. Steidl, B. Schuller, D. Seppi, K. Laskowski, T. Vogt, L. Devillers, L. Vidrascu, N. Amir, L. Kessous, and V. Aharonson, "Combining efforts for improving automatic classification of emotional user states," in Proc. IS-LTC, 2006, pp. 240-245.
30. H. Gunes, B. Schuller, M. Pantic, and R. Cowie, "Emotion representation, analysis and synthesis in continuous space: A survey," in Proc. FG, 2011, pp. 827-834.
31. B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, and M. Pantic, "AVEC 2011 - The first international audio/visual emotion challenge," in Proc. Int. Conf. Affective Comput. Intell. Interaction, Oct. 2011, pp. 415-424.
32. P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, and I. Matthews, "Painful data: The unbc-mcmaster shoulder pain expression archive database," in Proc. FG, 2011, pp. 57-64.
33. K. Prkachin and P. Solomon, "The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain," Pain, vol. 139, pp. 267-274, May 2008.
34. H. Meng and N. Bianchi-Berthouze, "Naturalistic affective expression classification by a multi-stage approach based on hidden markov models," in Proc. Int. Conf. Affective Comput. Intell. Interaction, vol. 6975. 2011, pp. 378-387.
35. G. A. Ramirez, T. Baltrušaitis, and L.-P. Morency, "Modeling latent discriminative dynamic of multi-dimensional affective signals," in Proc. Int. Conf. Affective Comput. Intell. Interaction, vol. 6975. Oct. 2011, pp. 396-406.
36. B. S. Everitt, The Cambridge Dictionary of Statistics. Cambridge, London, U.K.: Cambridge Univ. Press, 2006.
37. M. S. Bartlett, G. Littlewort, M. G. Frank, C. Lainscsek, I. R. Fasel, and J. R. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, 2006.
38. J. F. Cohn, L. I. Reed, Z. Ambadar, J. Xiao, T. Moriyama, and T. Moriyama, "Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior." in Proc. SMC, 2004, pp. 610-616. (Pubitemid 40415926)
39. F. Schwenker, S. Scherer, Y. Magdi, and G. Palm, "The GMM-SVM supervector approach for the recognition of the emotional status from speech," in Proc. ICANN, vol. 5768. 2009, pp. 894-903.
40. F. Eyben, M. Wöllmer, M. Valstar, H. Gunes, B. Schuller, and M. Pantic, "String-based audiovisual fusion of behavioural events for the assessment of dimensional affect," in Proc. FG, 2011, pp. 322-329.
41. M. Nicolaou, H. Gunes, and M. Pantic, "Output-associative RVM regression for dimensional and continuous emotion prediction," in Proc. FG, 2011.
42. M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie, "Abandoning emotion classes - towards continuous emotion recognition with modelling of long-range dependencies," in Proc. INTERSPEECH, 2008, pp. 597-600.
43. F. Eyben, S. Petridis, B. Schuller, G. Tzimiropoulos, and S. Zafeiriou, "Audiovisual classification of vocal outbursts in human conversation using long-short-term memory networks," in Proc. ICASSP, May 2011, pp. 5844-5847.
44. M. A. Nicolaou, H. Gunes, and M. Pantic, "Continuous prediction of spontaneous affect from multiple cues and modalities in valencearousal space," IEEE Trans. Affective Comput., vol. 2, no. 2, pp. 92-105, Apr.-Jun. 2011.
45. M. Wöllmer, A. Metallinou, F. Eyben, B. Schuller, and S. S. Narayanan, "Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling," in Proc. INTERSPEECH, 2010, pp. 2362-2365.
46. I. Cohen, N. Sebe, A. Garg, L. S. Chen, and T. S. Huang, "Facial expression recognition from video sequences: temporal and static modeling," Comp. Vision Image Understanding, vol. 91, no. 1-2, pp. 160-187, 2003.
47. T. L. Nwe, S. W. Foo, and L. C. D. Silva, "Speech emotion recognition using hidden markov models," Speech Commun., vol. 41, no. 4, pp. 603-623, 2003.
48. A. Nogueiras, A. Moreno, A. Bonafonte, and J. B. Mariño, "Speech emotion recognition using hidden markov models," in Proc. INTERSPEECH, 2001, pp. 2679-2682.
49. L. De Silva and P. C. Ng, "Bimodal emotion recognition," in Proc. FG, 2000, pp. 332-335.
50. Z. Zeng, J. Tu, B. Pianfetti, M. Liu, T. Zhang, Z. Zhang, T. Huang, and S. Levinson, "Audio-visual affect recognition through multistream fused HMM for HCI," in Proc. CVPR, vol. 2. 2005, pp. 967-972.
51. T. Kitazoe, S.-I. Kim, Y. Yoshitomi, and T. Ikeda, "Recognition of emotional states using voice, face image and thermal image of face," in Proc. INTERSPEECH, 2000, pp. 653-656.
52. C. M. Lee, S. Yildirim, M. Bulut, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan, "Emotion recognition based on phoneme classes," in Proc. ICSLP, 2004, pp. 889-892.
53. M. Yeasin, B. Bullot, and R. Sharma, "From facial expression to level of interest: A spatio-temporal approach," in Proc. CVPR, 2004, pp. 922-927.
54. Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, pp. 699-714, May 2005. (Pubitemid 40608403)
55. X. Li and Q. Ji, "Active affective state detection and user assistance with dynamic bayesian networks," IEEE Trans. Syst. Man Cybern. A, vol. 35, no. 1, pp. 93-105, Jan. 2005. (Pubitemid 40085388)
56. Y. Tong, W. Liao, and Q. Ji, "Facial action unit recognition by exploiting their dynamic and semantic relationships," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 10, pp. 1683-1699, oct. 2007.
57. D. Jiang, Y. Cui, X. Zhang, P. Fan, I. Ganzalez, and H. Sahli, "Audio visual emotion recognition based on triple stream dynamic bayesian network models," in Proc. Int. Conf. Affective Comput. Intelligent Interaction, vol. 6974. 2011, pp. 609-618.
58. A. M. Rahman, M. I. Tanveer, and M. Yeasin, "A spatio-temporal probabilistic framework for dividing and predicting facial action units," in Proc. Int. Conf. Affective Comput. Intell. Interaction, vol. 6975. 2011, pp. 598-607.
59. Z. Hammal and M. Kunz, "Pain monitoring: A dynamic and contextsensitive system," Pattern Recognit., vol. 45, no. 4, pp. 1265-1280, 2012.
60. G. Mckeown, M. Valstar, R. Cowie, and M. Pantic, "The semaine corpus of emotionally colored character interactions," in IEEE Int. Conf. Multimedia Expo, 2010, pp. 1079-1084.
61. P. Viola and M. J. Jones, "Robust real-time face detection," Int. J. Comp. Vision, vol. 57, no. 2, pp. 137-154, 2004.
62. T. Ojala, M. Pietikäinen, and D. Harwood, "A comparative study of texture measures with classification based on featured distributions," Pattern Recognit., vol. 29, no. 1, pp. 51-59, 1996.
63. T. Ojala, M. Pietikäinen, and T. Mäenpää, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002. (Pubitemid 34835471)
64. P. Ekman and W. V. Friesen, Facial Action Coding System. Mountain View, CA, USA: Consulting Psychologists Press, 1978.
65. T. F. Cootes, G. J. Edwards, and C. J. Taylor, "Active Appearance Models," Eur. Conf. Comp. Vision, vol. 2, pp. 484-498, 1998. (Pubitemid 128093046)
66. J. R. J. Fontaine, K. R. Scherer, E. B. Roesch, and P. C. Ellsworth, "The world of emotions is not two-dimensional," Psychological Sci., vol. 18, no. 12, pp. 1050-1057, 2007.
67. K. M. Prkachin and P. E. Solomon, "The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain," PAIN, vol. 139, no. 2, pp. 267-274, 2008.
68. A. Viterbi, "Error bounds for convolutional codes and an asymptotically optimum decoding algorithm," IEEE Trans. Information Theory, vol. 13, no. 2, pp. 260-269, Apr. 1967.
69. M. Glodek, S. Tschechne, G. Layher, M. Schels, T. Brosch, S. Scherer, M. Kächele, M. Schmidt, H. Neumann, G. Palm, and F. Schwenker, "Multiple classifier systems for the classification of audio-visual emotional states," in Proc. Int. Conf. Affective Comput. Intell. Interaction, vol. 6975. 2011, pp. 359-368.
70. A. Cruz, B. Bhanu, and S. Yang, "A psychologically-inspired matchscore fusion model for video-based facial expression recognition," in Proc. Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 341-350.
71. B. Romera-Paredes, A. Argyriou, N. Berthouze, and M. Pontil, "Exploiting unrelated tasks in multi-task learning," J. Mach. Learn. Res. Proc. Track, vol. 22, pp. 951-959, 2012.
72. W. Liu and S. Chawla, "Class confidence weighted knn algorithms for imbalanced data sets," in Proc. 15th Pacific-Asia Conf. Adv. Knowl. Discovery Data Mining, 2011, pp. 345-356.
73. S. D'Mello and A. Graesser, "The half-life of cognitive-affective states during complex learning," Cognition Emotion, vol. 25, no. 7, pp. 1299-1308, 2011.
74. B. Romera-Paredes, H. Aung, M. Pontil, A. C. de C. Williams, P. Watson, and N. Bianchi-Berthouze, "Transfer learning to account for idiosyncrasy in face and body expressions," in Proc. FG, 2013.