Kernel cross-modal factor analysis for information fusion with application to bimodal emotion recognition

IEEE Transactions on Multimedia

Volumn 14, Issue 3 PART1, 2012, Pages 597-607

  Wang, Yongjin ^a   Guan, Ling ^b   Venetsanopoulos, Anastasios N ^b  

^a HISENSE GROUP   (China)

^b RYERSON UNIVERSITY   (Canada)

Author keywords

Cross modal association; emotion recognition; information fusion; kernel machine technique

Indexed keywords

ALGEBRAIC OPERATIONS; BIMODAL EMOTION RECOGNITION; CROSS-MODAL; EMOTION RECOGNITION; FROBENIUS NORM; KERNEL BASED METHODS; KERNEL CANONICAL CORRELATION ANALYSIS; KERNEL MACHINE; KERNEL MATRICES; KERNEL TRICK; MULTI-MODAL INFORMATION; NON-LINEAR RELATIONSHIPS; OPTIMAL TRANSFORMATION; PROJECTION DIRECTION; STATISTICAL DEPENDENCE; TEMPORAL STRUCTURES; TIME SEGMENTS; TRANSFORMED DOMAIN; VISUAL CHANNELS; VISUAL FEATURE;

FACTOR ANALYSIS; FEATURE EXTRACTION; HIDDEN MARKOV MODELS; INFORMATION FUSION; MATRIX ALGEBRA; MULTIVARIANT ANALYSIS; SPEECH RECOGNITION;

FACE RECOGNITION;

EID: 84861175952     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2012.2189550     Document Type: Article

References (39)

1. L. Guan, Y. Wang, R. Zhang, Y. Tie, A. Bulzacki, and M. T. Ibrahim, "Multimodal information fusion for selected multimedia applications," Int. J. Multimedia Intell. Sec., vol. 1, no. 1, pp. 5-32, 2010.
2. T. Joshi, S. Dey, and D. Samanta, "Multimodal biometrics: state of the art in fusion techniques," Int. J. Biomet., vol. 1, no. 4, pp. 393-417, July 2009.
3. P. K. Atrey,M. A. Hossain, A. El Saddik, and M. S. Kankanhalli, "Multimodal fusion for multimedia analysis: A survey," Multimedia Syst., vol. 16, no. 6, pp. 345-379, 2010.
4. A. Ross and A. K. Jain, "Information fusion in biometrics," Pattern Recognit. Lett., vol. 24, no. 13, pp. 2115-2125, 2003. (Pubitemid 36810733)
5. G. R. G. Lanckriet, M. Deng, N. Cristianini, M. I. Jordan, and W. S. Noble, "Kernel-based data fusion and its application to protein function prediction in yeast," in Proc. Pac. Symp. Biocomput., 2004, pp. 300-311.
6. E. Aldea, C. Fouquier, J. Atif, and I. Bloch, "Kernel fusion for image classification using fuzzy structural information," in Proc. 3rd Int. Conf. Adv. Vis. Comput., 2007, pp. 307-317, LNCS 4842.
7. T. D. Bie, L. Tranchevent, L. Van Oeffelen, and Y.Moreau, "Kernelbased data fusion for gene prioritization," Bioinformatics, vol. 23, no. 13, pp. 125-132, Jul. 2007, Oxford Univ. Press.
8. C. H. Chan, J. Kittler, and M. A. Tahir, "Kernel fusion of multiple histogram descriptors for robust face recognition," Structural, Syuntactic, Statist. Pattern Recognit., pp. 718-727, 2010, LNCS 6218.
9. F. Salimi and M. T. Sadeghi, "Composite kernels for fusing colour information in face verification systems," in Proc. Int. Conf. Digital Image Process., 2009, pp. 407-411.
10. X. Tan and B. Triggs, "Fusing Gabor and LBP feature sets for kernelbased face recognition," in Proc. 3rd Int. Conf. Anal. Model. Faces Gestures, Rio de Janeiro, Brazil, Oct. 2007.
11. I. M. de Diego, A. Serrano, C. Conde, and E. Cabello, "Face verification with a kernel fusion method," Pattern Recognit. Lett., vol. 31, no. 9, pp. 837-844, Jul. 2010.
12. X. Xu, X. Pan, Y. Zhao, and Q. Pu, "Research on kernel-based feature fusion algorithm in multimodal recognition," in Proc. Int. Conf. Inf. Technol. Comput. Sci., 2009, pp. 3-6.
13. Y. Wang and D. Sun, "Feature fusion of palmprint and face based on KFDA," in Proc. IEEE Int. Conf. Signal Process., 2008, pp. 2092-2095.
14. H. Hotelling, "Relations between two sets of variates," Biometrika, vol. 28, pp. 312-377, 1936.
15. Q. Sun, S. Zeng, Y. Liu, P.Heng, and D.Xia, "A new method of feature fusion and its application in image recognition," Pattern Recognit., vol. 36, no. 12, pp. 2437-2448, Dec. 2005. (Pubitemid 41336713)
16. H. Bredin and G. Chollet, "Audio-visual speech synchronymeasure for talking-face identity verification," in Proc. IEEE ICASSP, Honolulu, HI, Apr. 2007, vol. 2, pp. 233-236.
17. N. M. Correa, T. Adali, Y. Li, and V. D. Calhoun, "Canonical correlation analysis for data fusion and group ingerences: examining applications of medical imaging data," IEEE Signal Process. Mag., vol. 27, no. 4, pp. 39-50, 2010.
18. M. E. Sargin, Y. Yemez, E. Erzin, and A. M. Tekalp, "Audiovisual synchronization and fusion using canonical correlation analysis," IEEE Trans. Multimedia, vol. 9, no. 7, pp. 1396-1403, Nov. 2007.
19. C.Fyfe and P. L. Lai, "Kernel and nonlinear canonical correlation analysis," Int. J. Neural Syst., vol. 10, pp. 365-374, 2001.
20. J. Zhao, Y. Fan, and W. Fan, "Fusion of global and local features using KCCA for automatic target recognition," in Proc. Int. Conf. Image Graph., Xi'an, China, 2009, pp. 958-962.
21. X. Xu and Z. Mu, "Feature fusion method based on KCCA for ear and profile face based multimodal recognition," in IEEE Int. Conf. Autom. Logistics, 2007, pp. 620-623. (Pubitemid 351370466)
22. M. B. Blaschko and C. H. Lampert, "Correlational spectral clustering," in Proc. IEEE CVPR, Los Alamitos, CA, 2008, pp. 1-8.
23. D. Li, N. Dimitrova, M. Li, and I. K. Sethi, "Multimedia content processing through cross-modal association," in Proc. ACM Int. Conf. Multimedia, 2003, pp. 604-611.
24. D. Li, C. Taskiran, N. Dimitrova, W. Wang, M. Li, and I. Sethi, "Cross-modal analysis of audio-visual programs for speaker detection," in Proc. IEEE Workshop Multimedia Signal Process., Shanghai, China, 2005, pp. 1-4.
25. K. R. Muller, S. Mika, G. Ratsch, K. Tsuda, and B. Scholkopf, "An introduction to kernel-based learning algorithms," IEEE Trans. Neural Netw, vol. 12, no. 2, pp. 181-201, Mar. 2001. (Pubitemid 32371478)
26. B. Scholkopf,A. Smola, and K. R.Muller, "Nonlinear component analysis as a kernel eigenvalue problem," Neural Computat., vol. 10, no. 5, pp. 1299-1319, 1998. (Pubitemid 128463674)
27. G. Baudat and F. Anouar, "Generalized discriminant analysis using a kernel approach," Neural Comput., vol. 12, no. 10, pp. 2385-2404, 2000.
28. H. Go, K. Kwak, D. Lee, and M. Chun, "Emotion recognition from the facial image and speech signal," in Proc. SICE Annu. Conf., Japan, 2003, vol. 3, pp. 2890-2895.
29. I. Kanluan, M. Grimm, and K. Kroschel, "Audio-visual emotion recognition using an emotion space concept," in Proc. 16th Eur. Signal Process. Conf., Lausanne, Switzerl and , 2008, pp. 1-5.
30. A. Metallinou, S. Lee, and S.Narayanan, "Audio-visual emotion recognition using Gaussianmixturemodels for face and voice," in Proc. 10th IEEE Int. Symp. Multimedia, 2008, pp. 250-257.
31. M. Han, J. H. Hus, and K. T. Song, "A new information fusion method for bimodal robotic emotion recognition," J. Comput., vol. 3, no. 7, pp. 39-47, 2008.
32. M. Song, C. Chen, and M. You, "Audio-visual based emotion recognition using tripled hidden Markov model," in Proc. IEEE ICASSP, Montreal, QC, Canada, 2004, vol. 5, pp. 877-880.
33. Z. Zeng, J. Tu, B. Pianfetti, and T. S. Huang, "Audio-visual affective expression recognition through multi-stream fused HMM," IEEE Trans. Multimedia, vol. 10, no. 4, pp. 570-577, Aug. 2008. (Pubitemid 351711233)
34. Y. Wang and L. Guan, "Recognizing human emotional state from audiovisual signals," IEEE Trans. Multimedia, vol. 10, no. 5, pp. 936-946, Oct. 2008.
35. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, 2008.
36. C. Garcia and G. Tziritas, "Face detection using quantized skin color regions merging and wavelet packet analysis," IEEE Trans. Multimedia, vol. 1, no. 3, pp. 264-277, Sep. 1999.
37. L. R. Rabiner, "A tutorial on hidden markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
38. M. Turk and A. Pentl and , "EigenFaces for recognition," J. Cogn. Neurosci., vol. 13, no. 1, pp. 71-86, 1991.
39. O. Martin, I.Kotsia, B. Macq, and I. Pitas, "The eNTERFACE05 audiovisual emotion database," in Proc. ICDEW, 2006, p. 8.