Multi-Modal Classifier-Fusion for the Recognition of Emotions

Coverbal Synchrony in Human-Machine Interaction

Volumn , Issue , 2013, Pages 73-98

  Schels, Martin ^a   Glodek, Michael ^b   Meudt, Sascha ^b   Scherer, Stefan ^c   Schmidt, Miriam ^b   Layher, Georg ^b   Tschechne, Stephan ^b   Brosch, Tobias ^b   Hrabal, David ^b   Walter, Steffen ^b   Traue, Harald C ^b   Palm, Günther ^b   Neumann, Heiko ^b   Schwenker, Friedhelm ^b  

^a CNRS   (France)

^b UNIVERSITY OF ULM   (Germany)

^c UNIVERSITY OF GENOA   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84908095321     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (59)

1. Abdel Hady, M. and F. Schwenker. 2010. Combining committee-based semisupervised learning and active learning. Journal of Computer Science and Technology, 25(4):681-698.
2. Argyle, M. 1988. Bodily Communication. Routledge, London (GB).
3. Bayerl, P. and H. Neumann. 2007. A fast biologically inspired algorithm for recurrent motion estimation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 29(2):246-260.
4. Bishop, C. 2006. Pattern Recognition and Machine Learning. Springer, New York (NY).
5. Black, M. and Y. Yacoob. 1997. Recognizing facial expressions in image sequences using local parameterized models of image motion. International Journal of Computer Vision, 25:23-48.
6. Blake, A., B. Bascle, M. Isard, and J. MacCormick. 1998. Statistical models of visual shape and motion. Phil. Trans. Royal Society of London, Series A, 386:1283-1302.
7. Boiten, F., N. Frijda and C. Wientjes. 1994. Emotions and respiratory patterns: Review and critical analysis. International Journal of Psychophysiology, 17(2):103-128.
8. Breiman, L. 1996. Bagging predictors. Journal of Machine Learning, 24(2):123-140.
9. Cannon, W. 1927. The James-Lange theory of emotions: A critical examination and an alternative theory. The American Journal of Psychology, 39(1/4):106-124.
10. Caridakis, G., G. Castellano, L. Kessous, A. Raouzaiou, L. Malatesta, S. Asteriadis and K. Karpouzis. 2007. Multimodal emotion recognition from expressive faces, body gestures and speech. In Artificial Intelligence and Innovations, from Theory to Applications, Vol. 247, pp. 375-388. Springer.
11. Chen, L. and T. Huang. 2000. Emotional expressions in audiovisual human computer interaction. IEEE International Conference on Multimedia and Expo (ICME), New York, 1:423-426.
12. Cootes, T. and C. Taylor. 1992. Active chape models-”smart snakes”. Proceedings of the British Machine Vision Conference (BMVC’92), pp. 266-275.
13. Cootes, T., G. Edwards and C. Taylor. 2001. Acttive appearance models. IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(6):681-685.
14. Dakin, S. and R. Watt. 2009. Biological “bar codes” in human faces. Journal of Vision, 9(4):1-10.
15. Dietrich, C., G. Palm and F. Schwenker. 2003. Decision templates for the classification of bioacoustic time series. Journal of Information Fusion, 4(2):101-109.
16. Dietrich, C., F. Schwenker and G. Palm. 2001. Classification of time series utilizing temporal and decision fusion. Workshop on Multiple Classifier Systems. LNCS 2096, pp. 378-387. Springer.
17. Ekman, P. and W. Friesen. 1978. Facial Action Coding System: Investigator’s Guide. Consulting Psychologists Press, Palo Alto (CA).
18. Eyben, F., M. Wöllmer and B. Schuller, 2012. A multitask approach to continuous five-dimensional affect sensing in natural speech. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(1):6.
19. Fischler, M. and R. Bolles. 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM, 24(6):381-395.
20. Glodek, M., L. Bigalke, M. Schels and F. Schwenker. 2011. Incorporating Uncertainty in a Layered HMM architecture for Human Activity Recognition. Proceedings of the Joint Workshop on Human Gesture and Behavior Understanding (J-HGBU), pp. 33-34, ACM.
21. Glodek, M., G. Layher, F. Schwenker and G. Palm. 2012. Recognizing Human Activities using a Layered Markov Architecture. Proceeding of the International Conference on Artificial Neural Networks (ICANN). LNCS 7552, pp. 677-684. Springer.
22. Glodek, M., M. Schels, G. Palm and F. Schwenker. 2012. Multi-Modal Fusion Based on Classification Using Rejection Option and Markov Fusion Network. Proceedings of the International Conference on Pattern Recognition ICPR, pp. 1084-1087, IEEE.
23. Glodek, M., M. Schels, G. Palm and F. Schwenker. 2012. Multiple Classifier Combination Using Reject Options and Markov Fusion Networks. Proceedings of the International Conference on Multimodal Interaction, pp. 465-472, ACM.
24. Glodek, M., S. Scherer, F. Schwenker. 2011. Conditioned Hidden Markov Model Fusion for Multimodal Classification. Proceedings of the 12th European Conference on Speech Communication and Technology (Interspeech’11), pp. 2269-2272. ISCA.
25. Glodek, M., S. Tschechne, G. Layher, M. Schels, T. Brosch, S. Scherer, M. Kächele, M. Schmidt, H. Neumann, G. Palm and F. Schwenker. 2011. Multiple Classifier Systems for the Classification of Audio-Visual Emotional States. Proceedings of the International Audio/Visual Emotion Challenge (AVEC) and Workshop. LNCS 6975, pp. 359-368, Springer.
26. Haykin, S. 1999. Neural Networks: A Comprehensive Foundation. Prentice Hall, East Lansing (MI).
27. Hermansky, H. 1990. Perceptual linear predictive (PLP) analysis of speech. Journal of the Acoustical Society of America, 87(4):1738-1752.
28. Isard, M. and A. Blake. 1998. CONDENSATION-Conditional density propagation for visual ttracking. International Journal of Computer Vision, 29:5-28.
29. Kelley, J. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1):26-41.
30. Kuncheva, L.I. 2004. Combining Pattern Classifiers: Methods and Algorithms. Wiley & Sons, Hoboken (NJ).
31. Layher, G., S. Tschechne, S. Scherer, T. Brosch, C. Curio and H. Neumann. 2011. Social Signal Processing in Companion Systems-Challenges Ahead. Proceedings of the Workshop Companion-Systeme und Mensch-Companion Interaktion (41st Jahrestagung der Gesellschaft für Informatik).
32. McKeown, G., M. Valstar, R. Cowie and M. Pantic. 2010. The SEMAINE corpus of emotionally coloured character interactions. In Proceedings of the IEEE International Conference Multimedia & Expo (ICME), pp. 1079-1084.
33. Meudt, S., L. Bigalke and F. Schwenker. 2012. ATLAS-An Annotation Tool for HCI Data Utilizing Machine Learning Methods. Proceedings of the 1st International Conference on Affective and Pleasurable Design (APD’12) [Jointly with the 4th International Conference on Applied Human Factors and Ergonomics (AHFE’12)], pp. 5347-5352, CRC Press.
34. Mutch, J. and D. Lowe. 2008. Object class recognition and localization using sparse features with limited receptive fields. International Journal of Computer Vision, 80(1):45-57.
35. Oudeyer, P. 2003. The production and recognition of emotions in speech: Features and algorithms. International Journal of Human-Computer Studies, 59(1-2):157-183.
36. Picard, R. 2003. Affective computing: Challenges. International Journal of Human-Computer Studies, 59(1):55-64.
37. Platt, J. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, pp. 61-74.
38. Poggio, T., U. Knoblich and J. Mutch. 2010. CNS: A GPU-based framework for simulating cortically-organized networks. MIT-CSAIL-TR-2010-013/CBCL-286.
39. Rabiner, L. 1989. A tutorial on hidden markov models and selected applications in speech recognition. IEEE, 77(2):257-286.
40. Rabiner, L. and B. Juang. 1993. Fundamentals of speech recognition. Prentice-Hall Signal Processing Series, East Lansing (MI).
41. Rolls, E. 1994. Brain mechanisms for invariant visual recognition and learning. Behavioural Processes, 33(1-2):113-138.
42. Schölkopf, B., A. Smola, R. Williamson and P. Bartlett. 2000. New support vector algorithms. Neural Computation, 12(5):1207-1245.
43. Schachter, S. 1964. The interaction of cognitive and physiological determinants of emotional state. In L. Berkowitz (ed.), Advances in Experimental Social Psychology (Vol. 1, pp. 49-80). Academic Press.
44. Schels, M. and F. Schwenker. 2010. A multiple classifier system approach for facial expressions in image sequences utilizing gmm supervectors. Proceedings of the 20th International Conference on Pattern Recognition (ICPR’10), pp. 4251-4254. IEEE.
45. Schels, M., M. Glodek, S. Meudt, M. Schmidt, D. Hrabal, R. Böck, S. Walter, F. Schwenker. 2012. Multi-Modal Classifier-Fusion for the Classification of Emotional States in WOZ Scenarios. Proceedings of the 1st International Conference on Affective and Pleasurable Design, pp. 5337-5346. CRC Press.
46. Schels, M., S. Scherer, M. Glodek, H.A. Kestler, G. Palm and F. Schwenker. 2011. On the discovery of events in EEG data utilizing information fusion. Journal on Multimodal User Interfaces, 6(3-4):117-141.
47. Scherer, S., M. Glodek, G. Layher, M. Schels, M. Schmidt, T. Brosch, S. Tschechne, F. Schwenker, H. Neumann, G. Palm. 2012. A generic framework for the inference of user states in human computer interaction: How patterns of low level communicational cues support complex affective states. Journal on Multimodal User Interfaces, 2(1)::1-4.
48. Scherer, S., M. Glodek, F. Schwenker, N. Campbell and G. Palm. 2012. Spotting laughter in natural multiparty conversations: a comparison of automatic online and offline approaches using audiovisual data. ACM Transactions on Interactive Intelligent Systems, 2(1):1-31.
49. Schuller, B., G. Rigoll and M. Lang. 2003. Hidden markov model-based speech emotion recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2:401-404.
50. Schuller, B., M. Valstar, F. Eyben, G. McKeown, R. Cowie and M. Pantic. 2011. The First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011).
51. Schwenker, F. and E. Trentin. 2012. Proceedings of the First IAPR TC3 Workshop on Partially Supervised Learning, LNCS 7081. Springer.
52. Sebe, N., M.S. Lew, Y. Sun, I. Cohen, T. Gevers and T.S. Huang. 2007. Authentic facial expression analysis. Image Vision Comput., 25(12):1856-1863.
53. Serre, T. and M.A. Giese. 2011. Elements for a neural theory of the processing of dynamic faces. In C. Curio, H. Bulthoff and M.A. Giese (eds.), Dynamic Faces. Insights from Experiments and Computation, pp. 187-210. MIT Press.
54. Settles, B. 2009. Active Learning Literature Survey. Department of Computer Sciences, University of Wisconsin-Madison.
55. Thiel, C., F. Schwenker and G. Palm. 2005. Using Dempster-Shafer Theory in MCF Systems to Reject Samples. Proceedings of the 6th International Workshop on Multiple Classifier System, LNCS 3541, pp. 118-127. Springer.
56. Walter, S., S. Scherer, M. Schels, M. Glodek, D. Hrabal, M. Schmidt, R. Bock, K. Limbrecht, H.C. Traue and F. Schwenker. 2011. Multimodal emotion classification in naturalistic user behavior. In Human-Computer Interaction, Part III, HCII 2011, J. Jacko (ed.), LNCS 6763, pp. 603-611. Springer, Berlin (DE).
57. Welch, P. 1967. The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms. IEEE Trans. Audio and Electroacoustics, 15(2):70-73.
58. Zeng, Z., M. Pantic, G. Roisman and T. Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence, 31(1):39-58.
59. Zheng, F., G. Zhang and Z. Song. 2001. Comparison of different implementations of MFCC. Journal of Computer Science and Technology, 16(6):582-589