Fusing audio, visual and textual clues for sentiment analysis from multimodal content

Neurocomputing

Volumn 174, Issue , 2016, Pages 50-59

  Poria, Soujanya ^a   Cambria, Erik ^b   Howard, Newton ^c   Huang, Guang Bin ^d   Hussain, Amir ^a  

^a UNIVERSITY OF STIRLING   (United Kingdom)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^d NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

Big social data analysis; Multimodal fusion; Multimodal sentiment analysis; Opinion mining; Sentic computing

Indexed keywords

INFORMATION ANALYSIS; SOCIAL NETWORKING (ONLINE);

COMPARATIVE EXPERIMENTS; MULTI-MODAL FUSION; OPINION MINING; SENTIC COMPUTING; SENTIMENT ANALYSIS; SOCIAL MEDIA PLATFORMS; SOURCES OF INFORMATIONS; STATE-OF-THE-ART SYSTEM;

DATA MINING;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BAYESIAN LEARNING; CLASSIFIER; CONTROLLED STUDY; DATA PROCESSING; EXTREME LEARNING MACHINE; FACIAL EXPRESSION; INFORMATION PROCESSING; INTERMETHOD COMPARISON; MULTIMODAL SENTIMENT ANALYSIS; PRIORITY JOURNAL; SPEECH; SUPPORT VECTOR MACHINE;

EID: 84941006193     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2015.01.095     Document Type: Article

References (59)

1. L.-P. Morency, R. Mihalcea, P. Doshi, Towards multimodal sentiment analysis: harvesting opinions from the web, In: Proceedings of the 13th International Conference on Multimodal Interfaces, ACM, Alicante, Spain, 2011, pp. 169-176.
2. E. Cambria, N. Howard, J. Hsu, A. Hussain, Sentic blending: scalable multimodal fusion for continuous interpretation of semantics and sentics, In: IEEE SSCI, Singapore, 2013, pp. 108-117.
3. Huang G.-B., Zhu Q.-Y., Siew C.-K. Extreme learning machine. theory and applications. Neurocomputing 2006, 70(1):489-501.
4. Huang G.-B., Cambria E., Toh K.-A., Widrow B., Xu Z. New trends of learning in computational intelligence. IEEE Comput. Intelli. Mag. 2015, 10(2):16-17.
5. Huang G., Huang G.-B., Song S., You K. Trends in extreme learning machines. a review. Neural Netw. 2015, 61:32-48.
6. Decherchi S., Gastaldo P., Zunino R., Cambria E., Redi J. Circular-ELM for the reduced-reference assessment of perceived image quality. Neurocomputing 2013, 102:78-89.
7. Cambria E., Gastaldo P., Bisio F., Zunino R. An ELM-based model for affective analogical reasoning. Neurocomputing 2015, 149:443-455.
8. Principi E., Squartini S., Cambria E., Piazza F. Acoustic template-matching for automatic emergency state detection. an ELM based algorithm. Neurocomputing 2015, 149:426-434.
9. Poria S., Cambria E., Hussain A., Huang G.-B. Towards an intelligent framework for multimodal affective data analysis. Neural Netw. 2015, 63:104-116.
10. H. Qi, X. Wang, S.S. Iyengar, K. Chakrabarty, Multisensor data fusion in distributed sensor networks using mobile agents, In: Proceedings of 5th International Conference on Information Fusion, 2001, pp. 11-16.
11. Ekman, Paul, Friesen, Wallace V, O'Sullivan, Maureen, Chan, Anthony, Diacoyanni-Tarlatzis, Irene, Heider, Karl, Krause, Rainer, LeCompte, William Ayhan and Pitcairn, Tom and Ricci-Bitti, Pio E and others, Universals and cultural differences in the judgments of facial expressions of emotion. J. person. soc. psychol. 53 (4) (1987) 712-717.
12. Matsumoto D. More evidence for the universality of a contempt expression. Motiv. Emot. 1992, 16(4):363-368.
13. Ekman P., Friesen W. Facial Action Coding System. A Technique for the Measurement of Facial Movement 1978, Consulting Psychologists Press, Palo Alto.
14. W.V. Friesen, P. Ekman, Emfacs-7: Emotional Facial Action Coding System, Unpublished manuscript, University of California at San Francisco 2.
15. A. Lanitis, C.J. Taylor, T.F. Cootes, A unified approach to coding and interpreting face images, In: Fifth International Conference on Computer Vision, 1995. Proceedings, IEEE, Cambridge, Massachusetts, USA, 1995, pp. 368-373.
16. D. Datcu, L. Rothkrantz, Semantic audio-visual data fusion for automatic emotion recognition, In: Euromedia, Citeseer, 2008.
17. Kenji M. Recognition of facial expression from optical flow. IEICE Trans. Inf. Syst. 1991, 74(10):3474-3483.
18. Ueki N., Morishima S., Yamada H., Harashima H. Expression analysis/synthesis system based on emotion space constructed by multilayered neural network. Syst. Comput. Jpn. 1994, 25(13):95-107.
19. L.S.-H. Chen, Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction (Ph.D. thesis), Citeseer, 2000.
20. Murray I.R., Arnott J.L. Toward the simulation of emotion in synthetic speech. a review of the literature on human vocal emotion. J. Acoust. Soc. Am. 1993, 93(2):1097-1108.
21. R. Cowie, E. Douglas-Cowie, Automatic statistical analysis of the signal and prosodic signs of emotion in speech, In: Fourth International Conference on Spoken Language, 1996. ICSLP 96, Proceedings, vol. 3, IEEE, Philadelphia, PA, USA, 1996, pp. 1989-1992.
22. F. Dellaert, T. Polzin, A. Waibel, Recognizing emotion in speech, In: Fourth International Conference on Spoken Language, 1996, ICSLP 96, Proceedings, vol. 3, IEEE, Philadelphia, PA, USA, 1996, pp. 1970-1973.
23. T. Johnstone, Emotional speech elicited using computer games, In: Fourth International Conference on Spoken Language, 1996, ICSLP 96, Proceedings, vol. 3, IEEE, Philadelphia, PA, USA, 1996, pp. 1985-1988.
24. Navas E., Hernáez I., Luengo I. An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional tts. IEEE Trans. Audio Speech Lang. Process. 2006, 14(4):1117-1127.
25. H. Atassi, A. Esposito, A speaker independent approach to the classification of emotional vocal expressions, In: ICTAI, 2008, pp. 147-150.
26. F. Burkhardt, A. Paeschke, M. Rolfes, W. Sendlmeier, B. Weiss, A database of German emotional speech, In: Interspeech, 2005, pp. 1517-1520.
27. P. Pudil, F. Ferri, J. Novovicova, J. Kittler, Floating search methods for feature selection with nonmonotonic criterion functions, In: IAPR, 1994, pp. 279-283.
28. K.R. Scherer, Adding the affective dimension: a new look in speech analysis and synthesis, In: ICSLP, 1996, pp. 1808-1811.
29. G. Caridakis, G. Castellano, L. Kessous, A. Raouzaiou, L. Malatesta, S. Asteriadis, K. Karpouzis, Multimodal emotion recognition from expressive faces, body gestures and speech, In: Artificial Intelligence and Innovations 2007: From Theory to Applications, 2007, pp. 375-388.
30. J. Wiebe, Learning subjective adjectives from corpora, In: AAAI/IAAI, 2000, pp. 735-740.
31. P.D. Turney, Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews, In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, 2002, pp. 417-424.
32. E. Cambria, D. Olsher, D. Rajagopal, SenticNet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis, In: AAAI, Quebec City, 2014, pp. 1515-1521.
33. T. Wilson, J. Wiebe, P. Hoffmann, Recognizing contextual polarity in phrase-level sentiment analysis, In: HLT/EMNLP, Vancouver, BC, Canada, 2005, pp. 347-354.
34. E. Riloff, J. Wiebe, Learning extraction patterns for subjective expressions, In: Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2003, pp. 105-112.
35. B. Pang, L. Lee, A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts, In: ACL, Barcelona, 2004, pp. 271-278.
36. C. Strapparava, A. Valitutti, Wordnet affect: an affective extension of wordnet., In: LREC, vol. 4, 2004, pp. 1083-1086.
37. C.O. Alm, D. Roth, R. Sproat, Emotions from text: machine learning for text-based emotion prediction, In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2005, pp. 579-586.
38. G. Mishne, Experiments with mood classification in blog posts, In: Proceedings of ACM SIGIR 2005 Workshop on Stylistic Analysis of Text for Information Access, vol. 19, 2005.
39. R. Xia, C. Zong, X. Hu, E. Cambria, Feature ensemble plus sample selection: domain adaptation for sentiment classification (extended abstract), In: IJCAI, Buenos Aires, 2015, pp. 4229-4233.
40. C. Yang, K.H.-Y. Lin, H.-H. Chen, Building emotion lexicon from weblog corpora, In: Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, Association for Computational Linguistics, 2007, pp. 133-136.
41. F.-R. Chaumartin, Upar7: a knowledge-based system for headline sentiment tagging, In: Proceedings of the 4th International Workshop on Semantic Evaluations, Association for Computational Linguistics, 2007, pp. 422-425.
42. A. Esuli, F. Sebastiani, Sentiwordnet: a publicly available lexical resource for opinion mining, In: Proceedings of LREC, vol. 6, 2006, pp. 417-422.
43. K.H.-Y. Lin, C. Yang, H.-H. Chen, What emotions do news articles trigger in their readers?, In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Glasgow, UK, 2007, pp. 733-734.
44. A. Pak, P. Paroubek, Twitter as a corpus for sentiment analysis and opinion mining, In: LREC, 2010, pp. 1320-1326.
45. M. Hu, B. Liu, Mining and summarizing customer reviews, In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, Seattle, Washington, USA, 2004, pp. 168-177.
46. Balahur A., Hermida J.M., Montoyo A. Building and exploiting emotinet, a knowledge base for emotion detection based on the appraisal theory model. IEEE Trans. Affect. Comput. 2012, 3(1):88-101.
47. C. Shan, S. Gong, P.W. McOwan, Beyond facial expressions: learning human emotion from body gestures, In: BMVC, 2007, pp. 1-10.
48. Mansoorizadeh M., Charkari N.M. Multimodal information fusion application to human emotion recognition from face and speech. Multimed. Tools Appl. 2010, 49(2):277-297.
49. Zeng Z., Tu J., Liu M., Huang T.S., Pianfetti B., Roth D., Levinson S. Audio-visual affect recognition. IEEE Trans. Multimed. 2007, 9(2):424-428.
50. Gunes H., Piccardi M. Bi-modal emotion recognition from expressive face and body gestures. J. Netw. Comput. Appl. 2007, 30(4):1334-1345.
51. Pun T., Alecu T.I., Chanel G., Kronegg J., Voloshynovskiy S. Brain-computer interaction research at the computer vision and multimedia laboratory, University of Geneva. IEEE Trans. Neural Syst. Rehabil. Eng. 2006, 14(2):210-213.
52. F. Wallhoff, Facial Expressions and Emotion Database, Technische Universität München.
53. M. Pantic, M. Valstar, R. Rademaker, L. Maat, Web-based database for facial expression analysis, In: IEEE International Conference on Multimedia and Expo, 2005. ICME 2005, IEEE, Amsterdam, The Netherlands, 2005, pp. 5-9.
54. M. Paleari, B. Huet, Toward emotion indexing of multimedia excerpts, In: International Workshop on Content-Based Multimedia Indexing, 2008, CBMI 2008, IEEE, London, UK, 2008, pp. 425-432.
55. Poria S., Gelbukh A., Cambria E., Hussain A., Huang G.-B. EmoSenticSpace. a novel framework for affective common-sense reasoning. Knowl. Based Syst. 2014, 69:108-123.
56. J.M. Saragih, S. Lucey, J.F. Cohn, Face alignment through subspace constrained mean-shifts, In: 2009 IEEE 12th International Conference on Computer Vision, IEEE, Kyoto, Japan, 2009, pp. 1034-1041.
57. F. Eyben, M. Wollmer, B. Schuller, Openear-introducing the munich open-source emotion and affect recognition toolkit, In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009, IEEE, Amsterdam, Netherlands, 2009, pp. 1-6.
58. E. Cambria, A. Hussain, C. Havasi, C. Eckl. Sentic computing: exploitation of common sense for the development of emotion-sensitive systems. In: LNCS, vol. 5967, Springer, pp. 148-156, 2010.
59. Cambria E., Hussain A. Sentic Computing. A Common-Sense-Based Framework for Concept-Level Sentiment Analysis 2015, Springer, Cham, Switzerland.