Convolutional MKL based multimodal emotion recognition and sentiment analysis

Proceedings - IEEE International Conference on Data Mining, ICDM

Volumn 0, Issue , 2016, Pages 439-448

  Poria, Soujanya ^a   Chaturvedi, Iti ^b   Cambria, Erik ^b   Hussain, Amir ^c  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c UNIVERSITY OF STIRLING   (United Kingdom)

Author keywords

Convolutional neural networks; Deep learning; Multimodal sentiment analysis; Multiple kernel learning

Indexed keywords

CLASSIFICATION (OF INFORMATION); DATA MINING; DEEP NEURAL NETWORKS; EMOTION RECOGNITION; MODAL ANALYSIS; SPEECH RECOGNITION;

CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; GOOGLE+; INTERNET CONNECTION; INTERNET WILLS; MULTI-MODAL; MULTIMODAL EMOTION RECOGNITION; MULTIMODAL SENTIMENT ANALYSE; MULTIPLE KERNEL LEARNING; SENTIMENT ANALYSIS;

CONVOLUTION;

EID: 85014552954     PISSN: 15504786     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDM.2016.178     Document Type: Conference Paper

References (36)

1. E. Cambria and B. White, "Jumping NLP curves: A review of natural language processing research, " IEEE Computational Intelligence Magazine, vol. 9, no. 2, pp. 48-57, 2014.
2. E. Cambria, "Affective computing and sentiment analysis, " IEEE Intelligent Systems, vol. 31, no. 2, pp. 102-107, 2016.
3. P. Liu, S. Han, Z. Meng, and Y. Tong, "Facial expression recognition via a boosted deep belief network, " in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014, pp. 1805- 1812.
4. D. Datcu and L. Rothkrantz, "Semantic audio-visual data fusion for automatic emotion recognition, " Euromedia'2008, 2008.
5. V. Rosas, R. Mihalcea, and L.-P. Morency, "Multimodal sentiment analysis of spanish online videos, " IEEE Intelligent Systems, vol. 28, no. 3, pp. 0038-45, 2013.
6. S. Poria, E. Cambria, A. Hussain, and G.-B. Huang, "Towards an intelligent framework for multimodal affective data analysis, " Neural Networks, vol. 63, pp. 104-116, 2015.
7. A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, "Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages, " IEEE Intelligent Systems, vol. 31, no. 6, 2016.
8. S. Poria, E. Cambria, N. Howard, G.-B. Huang, and A. Hussain, "Fusing audio, visual and textual clues for sentiment analysis from multimodal content, " Neurocomputing, vol. 174, pp. 50-59, 2016.
9. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks, " in Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097-1105.
10. Z. Zuo, B. Shuai, G. Wang, X. Liu, X. Wang, B. Wang, and Y. Chen, "Convolutional recurrent neural networks: Learning spatial dependencies for image representation, " in 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops, Boston, MA, USA, June 7-12, 2015, 2015, pp. 18-26.
11. N. Subrahmanya and Y. Shin, "Sparse multiple kernel learning for signal processing applications, " Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 32, no. 5, pp. 788-798, 2010.
12. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, "Large-scale video classification with convolutional neural networks, " in CVPR, 2014.
13. S. Poria, E. Cambria, D. Hazarika, and P. Vij, "A deeper look into sarcastic tweets using deep convolutional neural networks, " in COLING, 2016.
14. S. Poria, E. Cambria, and A. Gelbukh, "Aspect extraction for opinion mining with a deep convolutional neural network, " Knowledge-Based Systems, vol. 108, 2016.
15. P. Ekman and W. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Stanford University, Palo Alto, 1977.
16. L. C. De Silva, T. Miyasato, and R. Nakatsu, "Facial emotion recognition using multi-modal information, " in Information, Communications and Signal Processing, 1997. ICICS., Proceedings of 1997 International Conference on, vol. 1. IEEE, 1997, pp. 397-401.
17. D. Datcu and L. J. Rothkrantz, "Emotion recognition using bimodal data fusion, " in Proceedings of the 12th International Conference on Computer Systems and Technologies. ACM, 2011, pp. 122-128.
18. B. Schuller, "Recognizing affect from linguistic information in 3d continuous space, " Affective Computing, IEEE Transactions on, vol. 2, no. 4, pp. 192-205, 2011.
19. V. Rozgic, S. Ananthakrishnan, S. Saleem, R. Kumar, and R. Prasad, "Speech language & multimedia technol., raytheon bbn technol., cambridge, ma, usa, " in Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific. IEEE, 2012, pp. 1-4.
20. A. Metallinou, S. Lee, and S. Narayanan, "Audio-visual emotion recognition using Gaussian mixture models for face and voice, " in Multimedia, 2008. ISM 2008. Tenth IEEE International Symposium on. IEEE, 2008, pp. 250-257.
21. F. Eyben, M. Wollmer, A. Graves, B. Schuller, E. Douglas-Cowie, and R. Cowie, "On-line emotion recognition in a 3-d activation-valence-Time continuum using acoustic and linguistic cues, " Journal on Multimodal User Interfaces, vol. 3, no. 1-2, pp. 7-19, 2010.
22. C.-H. Wu and W.-B. Liang, "Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels, " Affective Computing, IEEE Transactions on, vol. 2, no. 1, pp. 10-21, 2011.
23. M. Hu and B. Liu, "Mining and summarizing customer reviews, " in Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2004, pp. 168-177.
24. V. Ṕerez-Rosas, R. Mihalcea, and L.-P. Morency, "Utterance-level multimodal sentiment analysis, " in Proceedings of ACL 2013, 2013, pp. 973-982.
25. L.-P. Morency, R. Mihalcea, and P. Doshi, "Towards multimodal sentiment analysis: Harvesting opinions from the web, " in Proceedings of the 13th international conference on multimodal interfaces. ACM, 2011, pp. 169-176.
26. C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, "Iemocap: Interactive emotional dyadic motion capture database, " Language resources and evaluation, vol. 42, no. 4, pp. 335-359, 2008.
27. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, " in Proceedings of the 26th Annual International Conference on Machine Learning, ser. ICML '09, 2009, pp. 609-616.
28. S. Poria, E. Cambria, and A. Gelbukh, "Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis, " in EMNLP, 2015, pp. 2539-2544.
29. I. Chaturvedi, Y.-S. Ong, I. Tsang, R.Welsch, and E. Cambria, "Learning word dependencies in text by means of a deep recurrent belief network, " Knowledge-Based Systems, vol. 108, pp. 144-154, 2016.
30. D. Cristinacce and T. Cootes, "Automatic feature localisation with constrained local models, " Pattern Recogn., vol. 41, no. 10, pp. 3054- 3067, 2008.
31. F. Eyben, M. Wollmer, and B. Schuller, "Opensmile: The munich versatile and fast open-source audio feature extractor, " in Proceedings of the international conference on Multimedia. ACM, 2010, pp. 1459- 1462.
32. T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space, " arXiv preprint arXiv:1301.3781, 2013.
33. K. Toutanova, D. Klein, C. D. Manning, and Y. Singer, "Featurerich part-of-speech tagging with a cyclic dependency network, " in IN PROCEEDINGS OF HLT-NAACL, 2003, pp. 252-259.
34. L. Oneto, F. Bisio, E. Cambria, and D. Anguita, "Statistical learning theory and ELM for big social data analysis, " IEEE Computational Intelligence Magazine, vol. 11, no. 3, pp. 45-55, 2016.
35. A. Jain, S. Vishwanathan, and M. Varma, "Spf-gmkl: generalized multiple kernel learning with a million kernels, " in Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2012, pp. 750-758.
36. V. Rozgic, S. Ananthakrishnan, S. Saleem, R. Kumar, and R. Prasad, "Ensemble of svm trees for multimodal emotion recognition, " in Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific. IEEE, 2012, pp. 1-4.