Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities

IEEE Transactions on Cybernetics

Volumn 45, Issue 9, 2015, Pages 1927-1941

  Savran, Arman ^a   Cao, Houwei ^a   Nenkova, Ani ^b   Verma, Ragini ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Acoustic; affective computing; arousal; Bayesian fusion; emotion recognition; facial expressions; lexical; multimodal; particle filter; power; speech; temporal fusion; turn based; valence

Indexed keywords

ACOUSTICS; FACE RECOGNITION; SPEECH;

AFFECTIVE COMPUTING; AROUSAL; BAYESIAN FUSION; EMOTION RECOGNITION; FACIAL EXPRESSIONS; LEXICAL; MULTI-MODAL; PARTICLE FILTER; POWER; VALENCE;

SPEECH RECOGNITION;

AUTOMATED PATTERN RECOGNITION; BAYES THEOREM; EMOTION; FACIAL EXPRESSION; HUMAN; PHYSIOLOGY; PROCEDURES; VIDEORECORDING;

BAYES THEOREM; EMOTIONS; FACIAL EXPRESSION; HUMANS; PATTERN RECOGNITION, AUTOMATED; VIDEO RECORDING;

EID: 84939825651     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2362101     Document Type: Article

References (41)

1. S. D'Mello and J. Kory, "Consistent but modest: A meta-analysis on unimodal and multimodal affect detection accuracies from 30 studies, " in Proc. ACM Int. Conf. Multimodal Interact., New York, NY, USA, 2012, pp. 31-38.
2. B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi, "Multisensor data fusion: A review of the state-of-the-art, " Inf. Fusion, vol. 14, no. 1, pp. 28-44, Jan. 2013.
3. H. Gunes and M. Pantic, "Automatic, dimensional and continuous emotion recognition, " Int. J. Syn. Emot., vol. 1, no. 1, pp. 68-99, Jan. 2010.
4. J. R. Fontaine, K. R. Scherer, E. B. Roesch, and P. Ellsworth, "The world of emotion is not two-dimensional, " Psychol. Sci., vol. 18, pp. 1050-1057, Dec. 2007.
5. R. A. Calvo and S. D'Mello, "Affect detection: An interdisciplinary review of models, methods, and their applications, " IEEE Trans. Affect. Comput., vol. 1, no. 1, pp. 18-37, Jan. 2010.
6. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
7. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey, " Pattern Recognit., vol. 36, no. 1, pp. 259-275, 2003.
8. P. Ekman, W. V. Friesen, and J. C. Hager, Facial Action Coding System. Salt Lake City, UT, USA: A Human Face, 2002.
9. J.-C. Lin, C.-H. Wu, and W.-L. Wei, "Error weighted semi-coupled hidden Markov model for audio-visual emotion recognition, " IEEE Trans. Multimedia, vol. 14, no. 1, pp. 142-156, Feb. 2012.
10. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani, "Robust continuous prediction of human emotions using multiscale dynamic cues, " in Proc. ACM Int. Conf. Multimodal Interact. (ICMI), New York, NY, USA, 2012, pp. 501-508.
11. A. Savran, B. Sankur, and M. T. Bilge, "Regression-based intensity estimation of facial action units, " Image Vis. Comput., vol. 30, no. 10, pp. 774-784, Oct. 2012.
12. B. Schuller, M. Valstar, F. Eyben, R. Cowie, and M. Pantic, "AVEC 2012: The continuous audio/visual emotion challenge, " in Proc. ACM Int. Conf. Multimodal Interact., New York, NY, USA, 2012, pp. 361-362.
13. D. Ververidis and C. Kotropoulos, "Emotional speech recognition: Resources, features, and methods, " Speech Commun., vol. 48, pp. 1162-1181, Sep. 2006.
14. B. Schuller, B. Vlasenko, F. Eyben, G. Rigoll, and A. Wendemuth, "Acoustic emotion recognition: A benchmark comparison of performances, " in Proc. Autom. Speech Recognit. Understanding (ASRU), Merano, Italy, 2009, pp. 552-557.
15. B. Schuller, R. Müller, M. K. Lang, and G. Rigoll, "Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensembles, " in Proc. Interspeech, Lisbon, Portugal, 2005, pp. 805-808.
16. L. Kessous, G. Castellano, and G. Caridakis, "Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis, " J. Multimodal User Interf., vol. 3, nos. 1-2, pp. 33-48, 2010.
17. H. Gunes and M. Piccardi, "Fusing face and body display for bi-modal emotion recognition: Single frame analysis and multi-frame post integration, " in Proc. Affect. Comput. Intell. Interact. (ACII), Beijing, China, 2005, pp. 102-111.
18. H. Gunes and M. Piccardi, "Automatic temporal segment detection and affect recognition from face and body display, " IEEE Trans. Cybern., vol. 39, no. 1, pp. 64-84, Feb. 2009.
19. M. Soleymani, M. Pantic, and T. Pun, "Multimodal emotion recognition in response to videos, " IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 211-223, Apr./Jun. 2012.
20. T. Senechal et al., "Facial action recognition combining heterogeneous features via multikernel learning, " IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 42, no. 4, pp. 993-1005, Aug. 2012.
21. J. Kim, "Robust speech recognition and understanding, " in Bimodal Emotion Recognition Using Speech and Physiological Changes, M. Grimm and K. Kroschel, Eds. Vienna, Austria: I-Tech, 2007, pp. 268-280.
22. A. Savran, R. Gur, and R. Verma, "Automatic detection of emotion valence on faces using consumer depth cameras, " in Proc. IEEE ICCV Workshop Consum. Depth Cameras Comput. Vis., Sydney, NSW, Australia, 2013, pp. 75-82.
23. A. Savran, B. Sankur, and M. T. Bilge, "Comparative evaluation of 3D vs. 2D modality for automatic detection of facial action units, " Pattern Recognit., vol. 45, no. 2, pp. 767-782, 2012.
24. C.-H. Wu and W.-B. Liang, "Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels, " IEEE Trans. Affect. Comput., vol. 2, no. 1, pp. 10-21, Jan./Jun. 2011.
25. A. Kapoor and R. W. Picard, "Multimodal affect recognition in learning environments, " in Proc. ACM Multimedia, Singapore, 2005, pp. 677-682.
26. F. Eyben et al., "String-based audiovisual fusion of behavioral events for the assessment of dimensional affect, " in Proc. IEEE Autom. Face Gesture Recognit. Workshop., Santa Barbara, CA, USA, 2011, pp. 322-329.
27. D. Ozkan, S. Scherer, and L.-P. Morency, "Step-wise emotion recognition using concatenated-HMM, " in Proc. ACM Int. Conf. Multimodal Interact. (ICMI), Santa Monica, CA, USA, 2012, pp. 477-484.
28. M. Wollmer, M. Kaiser, F. Eyben, B. Schuller, and G. Rigoll, "LSTM-modeling of continuous emotions in an audiovisual affect recognition framework, " Image Vis. Comput., vol. 31, no. 2, pp. 153-163, 2013.
29. B. Schuller et al., "AVEC 2011, the audio/visual emotion challenge, " in Proc. Int. Conf. Affect. Comput. Intell. Interact., Berlin, Germany, 2011, pp. 415-424.
30. C. Soladié, H. Salam, C. Pelachaud, N. Stoiber, and R. Séguier, "A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection, " in Proc. ACM Int. Conf. Multimodal Interact. (ICMI), Santa Monica, CA, USA, 2012, pp. 493-500.
31. A. Savran, H. Cao, M. Shah, A. Nenkova, and R. Verma, "Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering, " in Proc. ACM Int. Conf. Multimodal Interact. (ICMI), New York, NY, USA, 2012, pp. 485-492.
32. (2012). AVEC 2012, 2nd International Audio/Visual Emotion Challenge and Workshop [Online]. Available: http://sspnet.eu/avec2012/
33. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, "The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent, " IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 5-17, Jan./Mar. 2012.
34. F. Eyben, M. Wöllmer, and B. Schuller, "openSMILE: The munich versatile and fast open-source audio feature extractor, " in Proc. Int. Conf. Multimedia, New York, NY, USA, 2010, pp. 1459-1462.
35. C. M. Lee and S. S. Narayanan, "Toward detecting emotions in spoken dialogs, " IEEE Trans. Speech Audio Process., vol. 13, no. 2, pp. 293-303, Mar. 2005.
36. B. Schuller, G. Rigoll, and M. Lang, "Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture, " in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2004, pp. 577-580.
37. A. Batliner, K. Fischer, R. Huber, J. Spilker, and E. Nöth, "Desperately seeking emotions or: Actors, wizards, and human beings, " in Proc. ISCA Workshop Speech Emot., New Castle, U.K., 2000, pp. 195-200.
38. H. Cao, A. Savran, R. Verma, and A. Nenkova, "Acoustic and lexical representations for affect prediction in spontaneous conversations, " Comput. Speech Lang., to be published.
39. B. Vlasenko, B. Schuller, K. T. Mengistu, G. Rigoll, and A. Wendemuth, "Balancing spoken content adaptation and unit length in the recognition of emotion and interest, " in Proc. Interspeech, 2008, pp. 805-808.
40. D. A. Dickey and W. A. Fuller, "Distribution of the estimators for autore-gressive time series with a unit root, " J. Amer. Stat. Assoc., vol. 74, no. 1, pp. 427-431, 1979.
41. M. K. Pitt and N. Shephard, Sequential Monte Carlo Methods in Practice, N. de Freitas, A. Doucet, and N. J. Gordon, Eds. New York, NY, USA: Springer, 2001.