Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

NeuroImage

Volumn 102, Issue P1, 2014, Pages 162-172

  Verma, Gyanendra K ^a   Tiwary, Uma Shanker ^a  

^a INDIAN INSTITUTE OF INFORMATION TECHNOLOGY   (India)

Author keywords

Discrete wavelet transforms; EEG; Emotion recognition; KNN; Multimodal fusion; Multiresolution; Physiological signals; SVM; Wavelet transforms

Indexed keywords

ARTIFICIAL NEURAL NETWORK; BAYESIAN LEARNING; BLOOD VOLUME; BREATHING PATTERN; COMPUTER MODEL; CONCEPTUAL FRAMEWORK; DEMPSTER SHAFER THEORY; ELECTRODERMAL RESPONSE; ELECTROENCEPHALOGRAM; ELECTROMYOGRAM; ELECTROOCULOGRAM; EMOTION; HUMAN; MAXIMUM ENTROPY MODEL; MEASUREMENT ACCURACY; NONHUMAN; PSYCHOLOGICAL THEORY; RECOGNITION; REVIEW; SKIN TEMPERATURE; SUPPORT VECTOR MACHINE; BIOLOGICAL MODEL; CLASSIFICATION; ELECTROPHYSIOLOGY; PHYSIOLOGY; WAVELET ANALYSIS;

ELECTROPHYSIOLOGICAL PROCESSES; EMOTIONS; HUMANS; MODELS, NEUROLOGICAL; WAVELET ANALYSIS;

EID: 84908272142     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2013.11.007     Document Type: Review

References (76)

1. Aguilar J.F., Garcia J.O., Romero D.G., Rodriguez J.G. A comparative evaluation of fusion strategies for multimodal biometric verification. Int. Conf. on Video-Based Biometrie Person Authentication VBPA 2003, LNCS 2688 2003, 830-837. Guildford Springer -Verlag, Berlin Heidelberg. J. Kittler, M.S. Nixon (Eds.).
2. Atrey P.K., Kankanhalli M.S., Jain R. Information assimilation framework for event detection in multimedia surveillance systems. Springer/ACM Multimed. Syst. J. 2006, 12(3):239-253.
3. Beal M.J., Jojic N., Attias H. Gaphical model for audiovisual object tracking. IEEE Trans. Pattern Anal. Mach. Intell. 2003, 25:828-836.
4. Bengio S., Marcel C., Marcel S., Mariethoz J. Confidence measures for multimodal identity verification. Inf. Fusion 2002, 3(4):267-276.
5. Bhatnagar G., Wu Q.M.J., Raman B. A new fractional random wavelet transform for fingerprint security. IEEE Trans. Syst. Man Cybern. Syst. Hum. 2012, vol. 42(1):262-275.
6. Blattner M., Glinert E. Multimodal integration. IEEE Multimed. 1996, 3(4):14-24.
7. Bolt R. Put-That-There: voice and gesture at the graphics interface. Comput. Graph. 1980, 14(3):262-270.
8. Bredin H., Chollet G. visual speech synchrony measure for talking-face identity verification. IEEE International Conference on Acoustics, Speech and, Signal Processing 2007, vol. 2:233-236.
9. Cacioppo J., Tassinary L. Inferring psychological significance from physiological signals. Am. Psychol. 1990, 45:16-28.
10. Carpenter R. The Logic of Typed Feature Structures 1992, Cambridge University Press.
11. Cees G., Snoek M., Worring M., Smeulders W.M. Early versus late fusion in semantic video analysis. Proc. of the 13th annual ACM int. Conf. on Multimedia 2005, ACM, New York.
12. Cohen I., Sebe N., Garg A., Chen L., Huang T.S. Facial expression recognition from video sequences. temporal and static modeling. CVIU 2003, 91(1-2):160-187.
13. Corradini M., Mehta N., Bernsen J., Martin C. Multimodal input fusion in human-computer interaction. NATO-ASI Conf. on Data Fusion for Situation Monitoring, Incident Detection, Alert, and Response Management 2003.
14. Cortes C., Vapnik V. Support-vector networks. Mach. Learn. 1995, 20.
15. Cowie R., Douglas E., Tsapatsoulis N., Votsis G., Kollias S., Fellenz W., Taylor J. Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 2001, 18:32-80.
16. Smith Craig A., Ellsworth Phoebe C. Patterns of cognitive appraisal in emotion. J. Pers. Soc. Psychol. 1985, 48(4):813-838.
17. Darwin Charles The Expression of the Emotions in Man and Animals 1872/1988.
18. Douglas-Cowie E., Cowie R., Sneddon I., Cox C., Lowry O., McRorie M., Martin J.C., Devillers L., Abrilian S., Batliner A., Amir N., Karpouzis K. The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data. Proc. Second Int'l Conf. Affective Computing and Intelligent Interaction 2007, 488-500.
19. Ekman P. Facial expression and emotion. Am. Psychol. 1993, 48(384):392.
20. Ekman P. Basic emotions. Handbook of Cognition and Emotion 1999, 45-60.
21. Ekman P., Friesen W.V., O'Sullivan M., Chan A., Diacoyanni-Tarlatzis I., Heider K., Krause R., LeCompte W.A., Pitcairn T., Ricci-Bitti P.E. Universals and cultural differences in the judgments of facial expressions of emotion. J. Pers. Soc. Psychol. 1987, 53(4):712-717.
22. Fontaine J., Scherer K.R., Roesch E., Ellsworth P. The world of emotions is not two-dimensional. Psychol. Sci. 2007, 18:1050-1057.
23. Gandetto M., Marchesotti L., Sciutto S., Negroni D., Regazzoni C.S. From multi-sensor surveillance towards smart interactive spaces. IEEE Int. Conf. Multimed. Expo. I 2003, 641-644.
24. Grandjean D., Sander D., Scherer K.R. Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Conscious. Cogn. 2008, 17:484-495.
25. Grimm M., Kroschel K., Narayanan S. The Vera am Mittag German audio-visual emotional speech database. Proc. IEEE Int. Conf. Multimed. Expo. 2008, 865-868.
26. Guironnet M., Pellerin D., Rombaut M. Classification based on low-level feature fusion model. the European Signal Processing Conference. Antalya, Turkey 2005.
27. Healey J.A., Picard R.W. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 2005, 6(2):156-166.
28. Huang Y.S., Suen C.Y. Method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Trans. Pattern. Anal. Mach. Intell. 1995, 17(1):90-94.
29. James W. The Principles of Psychology 1950, vol. 1. Dover Publications.
30. Juslin P.N., Scherer K.R. Vocal expression of affect. The New Handbook of Methods in Nonverbal Behavior Research 2005, 65-135. Oxford Univ. Press.
31. Karray F., Alemzadeh M., Saleh j.A., Nours M. Human-computer interaction: an overview. Int. J. Smart Sensing Intell. Syst. 2008, vol. 1(1).
32. Kim J., Andre E. Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 2008, vol. 30(12):2067-2083.
33. Kittler J., Hatef M., Duin R.P., Matas J. On combining classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 1998, 20(3):226-239.
34. Koelstra S., Muhl C., Soleymani M., Yazdani A., Lee J.S., Ebrahimi T., Pun T., Nijholt A., Patras I. DEAP: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 2012, 3(1):18-31.
35. Lazarus R.S., Kanner A.D., Folkman S. Emotions. A cognitive-phenomenological analysis. Theories of Emotion 1980, 189-217. New York Academic Press.
36. Lewis P.A., Critchley H.D., Rotshtein P., Dolan R.J. Neural correlates of processing valence and arousal in affective words. Cereb. Cortex 2007, 17(3):742-748.
37. Language and the Politics of Emotion. Studies in Emotion and Social Interaction 1990, Cambridge University Press, New York. C.A. Lutz, L. Abu-Lughod (Eds.).
38. Magalhaes J., Ruger S. Information-theoretic semantic multimedia indexing. Int. Conf. on Image and Video Retrieval 2007, 619-626. Amsterdam.
39. McKeown G., Valstar M.F., Cowie R., Pantic M. The SEMAINE corpus of emotionally coloured character interactions. Proc. IEEE Int. Conf. Multimed. Expo. 2010, 1079-1084.
40. Mena J.B., Malpica J. Color image segmentation using the Dempster-Shafer theory of evidence for the fusion of texture. Int. Arch. Photogram. Rem. Sens. Spatial Inform. Sci. 2003, vol. XXXIV:139-144. (Part 3/W8).
41. Meyer G.F., Mulligan J.B., Wuerger S.M. Continuous audiovisual digit recognition using N-best decision fusion. J. Inf. Fusion 2004, 5:91-101.
42. Mihalis A., Gunes H., Pantic M. Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. Affect. Comput. 2011, vol. 2(2).
43. Minsky M. A framework for representing knowledge. The Psychology of Computer Vision 1975, McGraw-Hill, New York.
44. Nefian A.V., Liang L., Pi X., Liu X., Murphye K. Dynamic Bayesian networks for audio-visual speech recognition. EURASIP J. Appl. Signal Process. 2002, 11:1-15.
45. Nesse R.M. Evolutionary explanations of emotions. Hum. Nat. 1990, 1:261-289.
46. Ni J., Ma X., Xu L., Wang J. An image recognition method based on multiple BP neural networks fusion. IEEE Int. Conf. Inf. Acquis. 2004, 323-326.
47. Oliveira A.M., Teixeira M.P., Fonseca I.B., Oliveira M. Joint model-parameter validation of self-estimates of valence and arousal. probing a differential-weighting model of affective intensity. Proc. 22nd Ann. Meeting Int'l Soc. for Psychophysics 2006, 245-250.
48. Pan H., Liang Z., Anastasio T., Huang T. Exploiting the dependencies in information fusion. Proc. Conf. Comput. Vis. Pattern Recognit. 1999, 2:407-412.
49. Pantic M., Valstar M., Rademaker R., Maat L. Web-based database for facial expression analysis. Proc. IEEE Int. Conf. Multimed. Expo. 2005, 317-321.
50. Parrott W. Emotions in Social Psychology 2001, Psychology Press, Philadelphia.
51. Pereira C. Dimensions of emotional meaning in speech. Proc. ISCA Workshop Speech and Emotion 2000, 25-28.
52. Pfleger N. Context based multimodal fusion. ACM Int. Conf. Multimodal Interfaces 2004, 265-272.
53. Pitsikalis V., Katsamanis A., Papandreou G., Maragos P. Adaptive multimodal fusion by uncertainty compensation. Ninth Int. Conf. on Spoken Language Processing, Pittsburgh 2006.
54. Plutchik R. A psycho-evolutionary theory of emotions. Soc. Sci. Inf. 2008, 21(4-5):529-553.
55. Plutchik Robert, Conte R., Hope Circumplex Models of Personality and Emotions 1997, American Psychological Association, Washington, DC.
56. Poh N., Bengio S. How do correlation and variance of base experts affect fusion in biometric authentication tasks?. IEEE Trans. Signal Process. 2005, 53:4384-4396.
57. Rabiner L., Juang B. Fundamentals of Speech Recognition 1993, Prentice Hall, Englewood Cliffs, NJ.
58. Schaaff K. EEG-based Emotion Recognition 2008, Ph.D. thesis, Universitat Karlsruhe.
59. Schaaff K., Schultz T. Towards emotion recognition from electroencephalography signals. 3rd Int. Conf. on Affective Computing and Intelligent Interaction 2009.
60. Schachter S., Singer J.E. Cognitive, social and physiological determinants of emotional state. Psychol. Rev. 1962, 69:379-399.
61. Schuller B. Recognizing affect from linguistic information in 3D continuous space. IEEE Trans. Affect. Comput. 2011, 4(4):192-205.
62. Sebe N., Cohen I., Garg A., Huang T.S. Machine Learning in Computer Vision 2005, Springer, Berlin, NY.
63. Sebea N., Cohenb I., Geversa T., Huangc T.S. Multimodal approaches for emotion recognition: a survey. Proc. of SPIE-IS&T Electronic Imaging 2005, vol. 5670:56-67.
64. Singh R., Vatsa M., Noore A., Singh S.K. Dempster-Shafer theory based finger print classifier fusion with update rule to minimize training time. IEICE Electron. Express 2006, 3(20):429-435.
65. Soleymani M., Lichtenauer J., Pun T., Pantic M. A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 2012, vol. 3(1).
66. Soleymani C., Pantic M., Pun T. Multi-modal emotion recognition in response to videos. IEEE Tran. On Affective Computing 2012, 99(2):211-223.
67. Stickel C., Fink J., Holzinger A. Enhancing universal access - EEG based learnability assessment. Lect. Notes Comput. Sci. 2007, 4556:813-822.
68. Sundberg J., Patel S., Bjo E., Scherer K.R. Interdependencies among voice source parameters in emotional speech. IEEE Trans. Affect. Comput. 2011, vol. 2(3).
69. Tomkins S.S. Affect, imagery, consciousness. The Positive Affects 1962, vol. 1. Springer, New York.
70. Town C. Multi-sensory and multi-modal fusion for sentient computing. Int. J. Comput. Vis. 2007, 71:235-253.
71. Turk M. Multimodal human-computer interaction. Real-time Vision for Human-computer Interaction 2005, Springer, Berlin.
72. Weka machine learning tool - . http://www.cs.waikato.ac.nz/ml/weka/downloading.html.
73. Xie L., Kennedy L., Chang S.F., Divakaran A., Sun H., Lin C.Y. Layered dynamic mixture model for pattern discovery in asynchronous multi-modal streams. IEEE Int. Conf. Acoust. Speech Signal Process. 2005, 2:1053-1056.
74. Xu H., Chua T.S. Fusion of AV features and external information sources for event detection in team sports video. ACM Trans. Multimed. Comput. Commun. Appl. 2006, 2(1):44-67.
75. Zhu Q., Yeh M.C., Cheng K.T. Multimodal fusion using learned text concepts for image categorization. ACM Int. Conf. Multimed. 2006, 211-220.
76. Zou X., Bhanu B. Tracking humans using multimodal fusion. IEEE Conf. Comput. Vis. Pattern Recognit. 2005, 4.