Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis

Neurocomputing

Volumn 261, Issue , 2017, Pages 217-230

  Poria, Soujanya ^a   Peng, Haiyun ^b   Hussain, Amir ^a   Howard, Newton ^c   Cambria, Erik ^b  

^a UNIVERSITY OF STIRLING   (United Kingdom)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Classification; Convolutional neural network; Deep learning; ELM; Emotion; MKL; Multimodal sentiment analysis; Sentiment; SVM

Indexed keywords

CLASSIFICATION (OF INFORMATION); CONVOLUTION; CONVOLUTIONAL NEURAL NETWORKS; DEEP LEARNING; DEEP NEURAL NETWORKS; SENTIMENT ANALYSIS; SUPPORT VECTOR MACHINES;

DECISION LEVEL FUSION; EMOTION; EMOTION RECOGNITION; ENSEMBLE APPLICATIONS; INTERNET CONNECTION; MULTI-MODAL; MULTIPLE KERNEL LEARNING; SENTIMENT;

MODAL ANALYSIS;

ACCURACY; ARTICLE; ARTIFICIAL NEURAL NETWORK; CAMERA; COGNITION; COMPUTER; DATA ANALYSIS; EMOTION; HUMAN; INTERNET; KERNEL METHOD; PRIORITY JOURNAL; RECOGNITION; SMARTPHONE;

EID: 85012926540     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2016.09.117     Document Type: Article

References (100)

1. Cambria, E., Affective computing and sentiment analysis. IEEE Intel. Syst. 31:2 (2016), 102–107.
2. Morency, L.-P., Mihalcea, R., Doshi, P., Towards multimodal sentiment analysis: Harvesting opinions from the web. Proceedings of the 13th International Conference on Multimodal Interfaces, 2011, ACM, 169–176.
3. Cambria, E., Howard, N., Hsu, J., Hussain, A., Sentic blending: Scalable multimodal fusion for continuous interpretation of semantics and sentics. Proceedings of the IEEE SSCI, Singapore, 2013, 108–117.
4. Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., Morency, L.-P., Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intell. Syst. 28:3 (2013), 46–53.
5. Rosas, V., Mihalcea, R., Morency, L.-P., Multimodal sentiment analysis of spanish online videos. IEEE Intel. Syst. 28:3 (2013), 0038–45.
6. Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S., Iemocap: Interactive emotional dyadic motion capture database. Lang. Resources Eval. 42:4 (2008), 335–359.
7. Baltrusaitis, T., Robinson, P., Morency, L., 3d constrained local model for rigid and non-rigid facial tracking. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, IEEE, 2610–2617.
8. Qi, H., Wang, X., Iyengar, S.S., Chakrabarty, K., Multisensor data fusion in distributed sensor networks using mobile agents. Proceedings of 5th International Conference on Information Fusion, 2001, 11–16.
9. Poria, S., Chaturvedi, I., Cambria, E., Hussain, A., Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of the ICDM, Barcelona, 2016, 439–448.
10. Poria, S., Cambria, E., Bajpai, R., Hussain, A., A review of affective computing: From unimodal analysis to multimodal fusion. Inf. Fusion 37 (2017), 98–125.
11. Cambria, E., Wang, H., White, B., Guest editorial: Big social data analysis. Knowl.-Based Syst. 69 (2014), 1–2.
12. Pang, B., Lee, L., Vaithyanathan, S., Thumbs up?: Sentiment classification using machine learning techniques. Proceedings of the EMNLP, 2002, ACL, 79–86.
13. Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C., Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of EMNLP, 1631, 2013, 1642–1654.
14. Yu, H., Hatzivassiloglou, V., Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. Proceedings of the EMNLP, 2003, ACL, 129–136.
15. Melville, P., Gryc, W., Lawrence, R.D., Sentiment analysis of blogs by combining lexical knowledge with text classification. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2009, ACM, 1275–1284.
16. Zadeh, A., Zellers, R., Pincus, E., Morency, L.-P., Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intel. Syst. 31:6 (2016), 82–88.
17. Hu, X., Tang, J., Gao, H., Liu, H., Unsupervised sentiment analysis with emotional signals. Proceedings of the WWW, 2013, 607–618.
18. Gangemi, A., Presutti, V., Reforgiato Recupero, D., Frame-based detection of opinion holders and topics: A model and a tool. IEEE Comput. Intel. Mag. 9:1 (2014), 20–30, 10.1109/MCI.2013.2291688.
19. Cambria, E., Hussain, A., Sentic Computing: A Common-Sense-Based Framework for Concept-Level Sentiment Analysis. 2015, Springer, Cham, Switzerland.
20. Cambria, E., Poria, S., Bajpai, R., Schuller, B., SenticNet 4: A semantic resource for sentiment analysis based on conceptual primitives. Proceedings of the COLING, 2016, 2666–2677.
21. Qiu, G., Liu, B., Bu, J., Chen, C., Expanding domain sentiment lexicon through double propagation. Proceedings of the IJCAI, vol.9, 2009, 1199–1204.
22. Kanayama, H., Nasukawa, T., Fully automatic lexicon expansion for domain-oriented sentiment analysis. Proceedings of the EMNLP, 2006, ACL, 355–363.
23. Blitzer, J., Dredze, M., Pereira, F., et al. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. Proceedings of the ACL 2007, vol.7, 2007, 440–447.
24. Pan, S.J., Ni, X., Sun, J.-T., Yang, Q., Chen, Z., Cross-domain sentiment classification via spectral feature alignment. Proceedings of the WWW, 2010, ACM, 751–760.
25. Bollegala, D., Weir, D., Carroll, J., Cross-domain sentiment classification using a sentiment sensitive thesaurus. IEEE Trans. Knowl. Data Eng. 25:8 (2013), 1719–1731.
26. Strapparava, C., Valitutti, A., Wordnet affect: an affective extension of wordnet. Proceedings of the LREC, 4, 2004, 1083–1086.
27. Alm, C.O., Roth, D., Sproat, R., Emotions from text: machine learning for text-based emotion prediction. Proceedings of the EMNLP, 2005, Association for Computational Linguistics, 579–586.
28. Cambria, E., Livingstone, A., Hussain, A., The hourglass of emotions. Cognitive Behavioural Systems, 2012, Springer, 144–157.
29. Mishne, G., Experiments with mood classification in blog posts. Proceedings of ACM SIGIR 2005 Workshop on Stylistic Analysis of Text for Information Access, 19, 2005.
30. Oneto, L., Bisio, F., Cambria, E., Anguita, D., Statistical learning theory and ELM for big social data analysis. IEEE Comput. Intel. Mag. 11:3 (2016), 45–55.
31. Yang, C., Lin, K.H.-Y., Chen, H.-H., Building emotion lexicon from weblog corpora. Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, 2007, Association for Computational Linguistics, 133–136.
32. Chaumartin, F.-R., Upar7: A knowledge-based system for headline sentiment tagging. Proceedings of the 4th International Workshop on Semantic Evaluations, 2007, Association for Computational Linguistics, 422–425.
33. Poria, S., Cambria, E., Gelbukh, A., Aspect extraction for opinion mining with a deep convolutional neural network. Knowl.-Based Syst. 108 (2016), 42–49.
34. Li, X., Xie, H., Chen, L., Wang, J., Deng, X., News impact on stock price return via sentiment analysis. Knowl.-Based Syst. 69 (2014), 14–23.
35. Chikersal, P., Poria, S., Cambria, E., Gelbukh, A., Siong, C.E., Modelling public sentiment in twitter: using linguistic patterns to enhance supervised learning. Computational Linguistics and Intelligent Text Processing, 2015, Springer, 49–65.
36. S. Poria, A. Gelbukh, B. Agarwal, E. Cambria, N. Howard, Common sense knowledge based personality recognition from text, in Advances in Soft Computing and Its Applications, Springer, pp. 484–496.
37. Ekman, P., Universal facial expressions of emotion. Culture and Personality: Contemporary Readings/Chicago, 1974.
38. Matsumoto, D., More evidence for the universality of a contempt expression. Motiv. Emotion 16:4 (1992), 363–368.
39. Lanitis, A., Taylor, C.J., Cootes, T.F., A unified approach to coding and interpreting face images. Proceedings of the Fifth International Conference on Computer Vision, 1995, IEEE, 368–373.
40. Datcu, D., Rothkrantz, L., Semantic audio-visual data fusion for automatic emotion recognition. Proceedings of the Euromedia, 2008.
41. Kenji, M., Recognition of facial expression from optical flow. IEICE Trans. Inform. Syst. 74:10 (1991), 3474–3483.
42. Ueki, N., Morishima, S., Yamada, H., Harashima, H., Expression analysis/synthesis system based on emotion space constructed by multilayered neural network. Syst. Comput. Japan 25:13 (1994), 95–107.
43. Chen, L.S.-H., Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction, 2000 Ph.D. thesis.
44. Xu, M., Ni, B., Tang, J., Yan, S., Image re-emotionalizing. The Era of Interactive Media, 2013, Springer, 3–14.
45. Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S., Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91:1 (2003), 160–187.
46. Mansoorizadeh, M., Charkari, N.M., Multimodal information fusion application to human emotion recognition from face and speech. Multim. Tools Appl. 49:2 (2010), 277–297.
47. Rosenblum, M., Yacoob, Y., Davis, L.S., Human expression recognition from motion using a radial basis function network architecture. IEEE Trans. Neural Netw. 7:5 (1996), 1121–1138.
48. Otsuka, T., Ohya, J., A study of transformation of facial expressions based on expression recognition from temproal image sequences. Technical Report, 1997, Institute of Electronic, Information, and Communications Engineers (IEICE).
49. Yacoob, Y., Davis, L.S., Recognizing human facial expressions from long image sequences using optical flow. IEEE Trans. Pattern Anal. Mach. Intell. 18:6 (1996), 636–642.
50. Essa, I.A., Pentland, A.P., Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19:7 (1997), 757–763.
51. Murray, I.R., Arnott, J.L., Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. J. Acoust. Soc. Am. 93:2 (1993), 1097–1108.
52. Cowie, R., Douglas-Cowie, E., Automatic statistical analysis of the signal and prosodic signs of emotion in speech. Proceedings of the Fourth International Conference on Spoken Language, 3, 1996, IEEE, 1989–1992.
53. Dellaert, F., Polzin, T., Waibel, A., Recognizing emotion in speech. Proceedings of the Fourth International Conference on Spoken Language, 1996. ICSLP 96, 3, 1996, IEEE, 1970–1973.
54. Johnstone, T., Emotional speech elicited using computer games. Proceedings of the Fourth International Conference on Spoken Language, 1996. ICSLP 96, 3, 1996, IEEE, 1985–1988.
55. Navas, E., Hernaez, I., Luengo, I., An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional tts. IEEE Trans. Audio, Speech, Lang. Process. 14:4 (2006), 1117–1127.
56. De Silva, L.C., Miyasato, T., Nakatsu, R., Facial emotion recognition using multi-modal information. Proceedings of 1997 International Conference on Information, Communications and Signal Processing, 1, 1997, IEEE, 397–401.
57. Chen, L.S., Huang, T.S., Miyasato, T., Nakatsu, R., Multimodal human emotion/expression recognition. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, 1998, IEEE, 366–371.
58. Wang, Y., Guan, L., Recognizing human emotional state from audiovisual signals*. IEEE Trans. Multim 10:5 (2008), 936–946.
59. Datcu, D., Rothkrantz, L.J., Emotion recognition using bimodal data fusion. Proceedings of the 12th International Conference on Computer Systems and Technologies, 2011, ACM, 122–128.
60. Kessous, L., Castellano, G., Caridakis, G., Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. J. Multimodal User Interf. 3:1–2 (2010), 33–48.
61. Schuller, B., Recognizing affect from linguistic information in 3d continuous space. IEEE Trans. Affective Comput. 2:4 (2011), 192–205.
62. Rashid, M., Abu-Bakar, S., Mokji, M., Human emotion recognition from videos using spatio-temporal and audio features. Vis. Comput. 29:12 (2013), 1269–1275.
63. Glodek, M., Reuter, S., Schels, M., Dietmayer, K., Schwenker, F., Kalman filter based classifier fusion for affective state recognition. Multiple Classifier Systems, 2013, Springer, 85–94.
64. Hommel, S., Rabie, A., Handmann, U., Attention and emotion based adaption of dialog systems. Intelligent Systems: Models and Applications, 2013, Springer, 215–235.
65. Rozgic, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Prasad, R., Speech language & multimedia technol., raytheon bbn technol., cambridge, ma, usa. Proceedings of the 2012 Asia-Pacific Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012, IEEE, 1–4.
66. Metallinou, A., Lee, S., Narayanan, S., Audio-visual emotion recognition using gaussian mixture models for face and voice. Proceedings of the Tenth IEEE International Symposium on Multimedia, 2008, IEEE, 250–257.
67. Eyben, F., Wöllmer, M., Graves, A., Schuller, B., Douglas-Cowie, E., Cowie, R., On-line emotion recognition in a 3-d activation-valence-time continuum using acoustic and linguistic cues. J. Multimodal User Interf. 3:1–2 (2010), 7–19.
68. Wu, C.-H., Liang, W.-B., Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels. IEEE Trans.Affect. Comput. 2:1 (2011), 10–21.
69. Bucak, S., Jin, R., Jain, A., Multiple kernel learning for visual object recognition: a review. IEEE Trans. Pattern Anal. Mach. Intell. 36:7 (2014), 1354–1369.
70. Hinrichs, C., Singh, V., Xu, G., Johnson, S., Mkl for robust multi-modality ad classification. Med. Image Comput. Comput. Assist. Interv. 5762 (2009), 786–794.
71. Zhang, Z., Li, Z.-N., Drew, M., Adamkl: A novel biconvex multiple kernel learning approach. Proceedings of the 2010 20th International Conference on Pattern Recognition (ICPR), 2010, 2126–2129.
72. Wang, S., Jiang, S., Huang, Q., Tian, Q., Multiple kernel learning with high order kernels. Proceedings of the International Conference on Pattern Recognition, 2010, 2138–2141.
73. Subrahmanya, N., Shin, Y., Sparse multiple kernel learning for signal processing applications. IEEE Trans. Pattern Anal. Mach. Intell. 32:5 (2010), 788–798.
74. Wawer, A., Mining opinion attributes from texts using multiple kernel learning. Proceedings of 2011 IEEE 11th International Conference on Data Mining Workshops (ICDMW), 2011, 123–128.
75. Yang, J., Tian, Y., Duan, L.-Y., Huang, T., Gao, W., Group-sensitive multiple kernel learning for object recognition. IEEE Trans. Image Process. 21:5 (2012), 2838–2852.
76. Nilufar, S., Ray, N., Zhang, H., Object detection with dog scale-space: A multiple kernel learning approach. IEEE Trans. Image Process. 21:8 (2012), 3744–3756.
77. Vahdat, A., Cannons, K., Mori, G., Oh, S., Kim, I., Compositional models for video event detection: a multiple kernel learning latent variable approach. Proceedings of 2013 IEEE International Conference on Computer Vision (ICCV), 2013, 1185–1192.
78. Niaz, U., Merialdo, B., Fusion methods for multi-modal indexing of web data. Proceedings of the 2013 14th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), 2013, 1–4.
79. Xu, X., Tsang, I., Xu, D., Soft margin multiple kernel learning. IEEE Trans. Neural Netw. Learn. Syst. 24:5 (2013), 749–761.
80. Ni, B., Li, T., Moulin, P., Beta process multiple kernel learning. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, 963–970.
81. Liu, F., Zhou, L., Shen, C., Yin, J., Multiple kernel learning in the primal for multimodal alzheimers disease classification. IEEE J. Biomedical Health Inform. 18:3 (2014), 984–990.
82. Xia, H., Hoi, S., Jin, R., Zhao, P., Online multiple kernel similarity learning for visual search. IEEE Trans. Pattern Anal. Mach. Intel. 36:3 (2014), 536–549.
83. Saragih, J.M., Lucey, S., Cohn, J.F., Face alignment through subspace constrained mean-shifts. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, 2009, IEEE, 1034–1041.
84. Eyben, F., Wöllmer, M., Schuller, B., Opensmile: the munich versatile and fast open-source audio feature extractor. Proceedings of the International Conference on Multimedia, 2010, ACM, 1459–1462.
85. Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S., Cnn features off-the-shelf: an astounding baseline for recognition. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2014, IEEE, 512–519.
86. Kalchbrenner, N., Grefenstette, E., Blunsom, P., A convolutional neural network for modelling sentences. CoRR, abs/1404.2188, 2014.
87. T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, (2013) arXiv preprint arXiv: 1301.3781.
88. Poria, S., Cambria, E., Gelbukh, A., Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. Proceedings of EMNLP, 2015, 2539–2544.
89. Cambria, E., Fu, J., Bisio, F., Poria, S., AffectiveSpace 2: Enabling affective intuition for concept-level sentiment analysis. Proceedings of the AAAI, Austin, 2015, 508–514.
90. Rajagopal, D., Cambria, E., Olsher, D., Kwok, K., A graph-based approach to commonsense concept extraction and semantic similarity detection. Proceedings of the WWW, Rio De Janeiro, 2013, 565–570.
91. Huang, G.-B., Cambria, E., Toh, K.-A., Widrow, B., Xu, Z., New trends of learning in computational intelligence. IEEE Comput. Intel. Mag. 10:2 (2015), 16–17.
92. Jain, A., Vishwanathan, S., Varma, M., Spf-gmkl: generalized multiple kernel learning with a million kernels. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012, ACM, 750–758.
93. Rozgic, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Prasad, R., Ensemble of svm trees for multimodal emotion recognition. Proceedings of the 2012 Asia-Pacific Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012, IEEE, 1–4.
94. Poria, S., Cambria, E., Gelbukh, A., Bisio, F., Hussain, A., Sentiment data flow analysis by means of dynamic linguistic patterns. IEEE Comput. Intel. Mag. 10:4 (2015), 26–36.
95. Huang, G.-B., Wang, D.H., Lan, Y., Extreme learning machines: a survey. Int. J. Mach. Learn. Cybern. 2:2 (2011), 107–122.
96. Ridella, S., Rovetta, S., Zunino, R., Circular backpropagation networks for classification. IEEE Trans. Neural Netw. 8:1 (1997), 84–97.
97. Huang, G.-B., Chen, L., Siew, C.-K., Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw. 17:4 (2006), 879–892.
98. Huang, G.-B., An insight into extreme learning machines: Random neurons, random features and kernels. Cognitive Comput., 2014, 10.1007/s12559-014-9255-2.
99. Huang, G.-B., Zhou, H., Ding, X., Zhang, R., Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man, Cybern. Part B: Cybern. 42:2 (2012), 513–529.
100. Liu, X., Wang, L., Huang, G.-B., Zhang, J., Yin, J., Multiple kernel extreme learning machine. Neurocomputing 149, Part A (2015), 253–264.