Towards an intelligent framework for multimodal affective data analysis

Neural Networks

Volumn 63, Issue , 2015, Pages 104-116

  Poria, Soujanya ^a   Cambria, Erik ^b   Hussain, Amir ^a   Huang, Guang Bin ^c  

^a UNIVERSITY OF STIRLING   (United Kingdom)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

Affective computing; Emotion analysis; Facial expressions; Multimodal; Multimodal sentiment analysis; Speech; Text

Indexed keywords

DATA MINING; HUMAN COMPUTER INTERACTION; INFORMATION RETRIEVAL; SEMANTICS; SENTIMENT ANALYSIS; SOCIAL NETWORKING (ONLINE); SPEECH;

AFFECTIVE COMPUTING; EMOTION ANALYSIS; FACIAL EXPRESSIONS; MULTI-MODAL; TEXT;

MODAL ANALYSIS;

AFFECT; ARTICLE; AUDIO RECORDING; AUTOMATED PATTERN RECOGNITION; CLASSIFIER; DATA ANALYSIS; DATA EXTRACTION; DATA PROCESSING; EMOTION; FACIAL EXPRESSION; HUMAN; KNOWLEDGE BASE; LEARNING ALGORITHM; MACHINE LEARNING; MEASUREMENT ACCURACY; MULTIMEDIA; MULTIMODAL DATA ANALYSIS; SEMANTICS; SUPPORT VECTOR MACHINE; VIDEORECORDING; WRITTEN LANGUAGE; ALGORITHM; ARTIFICIAL INTELLIGENCE; BIOMETRY; INFORMATION RETRIEVAL; PROCEDURES;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BIOMETRIC IDENTIFICATION; HUMANS; INFORMATION STORAGE AND RETRIEVAL;

EID: 84917739875     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2014.10.005     Document Type: Article

References (93)

1. Alm, C., Roth, D., & Sproat, R. (2005a). Emotions from text: Machine learning for text-based emotion prediction. In HLT and EMNLP. (pp. 579-586). Vancouver, British Columbia, Canada.
2. Alm, C., Roth, D., & Sproat, R. (2005b). Emotions from text: Machine learning for text-based emotion prediction. in: Proceedings of the conference on empirical methods in natural language processing (pp 347-354) Vancouver, Canada.
3. Arora P., Bakliwal A., Varma V. Hindi subjective Lexicon generation using WordNet graph traversal. International Journal of Computational Linguistics and Applications 2012, 3(1):25-39.
4. Atassi, H., & Esposito, A. (2008). A speaker independent approach to the classification of emotional vocal expressions (pp. 147-152).
5. Balahur A., Hermida J.M., Montoyo A. Building and exploiting emotinet, a knowledge base for emotion detection based on theappraisal theory model. IEEE Transactions on Affective Computing 2012, 3(1).
6. Bazzanella C. Emotions, language and context. Emotion in dialogic interaction. Advances in the complex 2004, 59-76. Benjamins, Amsterdam, Philadelphia. E. Weigand (Ed.).
7. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W., & Weiss, B. (2005). A database of german emotional speech. In Interspeech (pp. 1517-1520).
8. Cambria E., Howard N., Hsu J., Hussain A. Sentic blending: Scalable multimodal fusion for the continuous interpretation of semantics and sentics. IEEE SSCI 2013, 108-117.
9. Cambria E., Hussain A., Havasi C., Eckl C. Common sense computing: from the society of mind to digital intuition and beyond. LNCS 2009, Vol. 5707:252-259. Springer.
10. Cambria E., Hussain A., Havasi C., Eckl C. Sentic computing: Exploitation of common sense for the development of emotion-sensitive systems. LNCS 2010, Vol. 5967:148-156. Springer.
11. Cambria E., Hussain A., Havasi C., Eckl C. SenticSpace: Visualizing opinions and sentiments in a multi-dimensional vector space. LNAI 2010, Vol. 6279:385-393. Springer.
12. Cambria E., Mazzocco T., Hussain A., Eckl C. Sentic medoids: Organizing affective common sense knowledge in a multi-dimensional vector space. LNCS 2011, Vol. 6677:601-610. Springer.
13. Cambria E., Olsher D., Rajagopal D. SenticNet 3: A common and common-sense knowledge base for cognition-driven sentiment analysis. AAAI 2014, 1515-1521.
14. Caridakis G., Castellano G., Kessous L., Raouzaiou A., Malatesta L., Asteriadis S., et al. Multimodal emotion recognition from expressive faces, body gestures and speech. Artificial intelligence and innovations 2007: From theory to applications 2007, 375-388. Springer, US.
15. Chaumartin F. UPAR7: A knowledge-based system for headline sentiment tagging. SemEval-2007 2007, 422-425. ACL, Prague.
16. Chen L.S.H. Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction 2000, diss., University of Illinois.
17. Chiu, C.C., Chang, Y.L., & Lai, Y.J. (1994). The analysis and recognition of human vocal emotions. In Proc. international computer symposium (pp. 83-88).
18. Cohen I., Sebe N., Garg A., Chen Lawrence S., Huang Thomas S. Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and Image Understanding 2003, 91(1):160-187.
19. Cortes C., Vapnik V. Support-vector networks. Machine Learning 1995, 20(3):273-297.
20. Cowie, R., & Douglas-Cowie, E. (1996). Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In Proc. international conf. on spoken language processing (pp. 1989-1992).
21. Datcu, D., & Rothkrantz, L. (2008). Semantic audio-visual data fusion for automatic emotion recognition. In Euromedia'08 (pp. 1-6).
22. Datcu D., Rothkrantz L. Multimodal recognition of emotions in car environments. DCI&I 2009 2009.
23. Dellaert, F., Polzin, T., & Waibel, A. (1996). Recognizing emotion in speech. In Proc. international conf. on spoken language processing (pp. 1970-1973).
24. Dobrišek S., Gajšek R., Mihelič F., Pavešić N., Štruc V. Towards efficient multi-modal emotion recognition. International Journal of Advanced Robotic Systems 2013, 10(53).
25. Ekman P. Universal facial expressions of emotions. California mental health research digest, Vol. 8 1970, 151-158.
26. Ekman P., Friesen W.V. Facial action coding system: Investigator's guide 1978, Consulting Psychologists Press.
27. Essa I.A., Pentland A.P. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997, 19:757-763.
28. Esuli A., Sebastiani F. SentiWordNet: A publicly available lexical resource for opinion mining. Proceedings of the 5th conference on language resources and evaluation 2006, LREC.
29. Friesen, W., & Ekman, P. (1983). EMFACS-7: Emotional facial action coding system. Unpublished manual, University of California, California.
30. Gunes, H., & Piccardi, M. (2006). A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In Proc. 18th int'l conf. pattern recognition, ICPR'06, Vol. 1 (pp. 1148-1153).
31. Gunes H., Piccardi M. Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications 2007, 30(4):1334-1345.
32. Gunes H., Piccardi M. Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) 2009, 39(1):64-84.
33. Havasi, C., Speer, R., & Pustejovsky, J. (2009). Automatically suggesting semantic structure for a generative Lexicon ontology. In Generative Lexicon.
34. Howard N. Brain language: The fundamental code unit. The Brain Sciences Journal 2012, 1(1):4-45. 999.
35. Howard N., Cambria E. Intention awareness: Improving upon situation awareness in human-centric environments. Human-Centric Computing and Information Sciences 2013, 3(9).
36. Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In SIGKDD, Seattle.
37. Huang G.-B., Zhu Q., Siew C. Extreme learning machine: theory and applications. Neurocomputing 2006, 70(1):489-501.
38. Jimenez Vargas S., Gelbukh A. SC spectra: A linear-time soft cardinality approximation for text comparison. Lecture Notes in Artificial Intelligence 2011, 7095:213-224.
39. Jimenez Vargas S., Gelbukh A. Baselines for natural language processing tasks based on soft cardinality spectra. International Journal of Applied and Computational Mathematics 2012, 11(2):180-199.
40. Johnstone, T. (1996). Emotional speech elicited using computer games. In Proc. international conf. on spoken language processing (pp. 1985-1988).
41. Kapoor A., Burleson W., Picard R. Automatic prediction of frustration. International Journal of Human-Computer Studies 2007, 65:724-736.
42. Kuncheva L. Combining pattern classifiers: Methods and algorithms 2004, Wiley & Sons.
43. Lanitis, A., Taylor, C. J., & Cootes, T. F. (1995). A unified approach to coding and interpreting face images. In Fifth international conference on computer vision, proceedings (pp. 368-373).
44. Lapponi, E., Read, J., & Ovrelid, L. (2012). Representing and resolving negation for sentiment analysis. In ICDM SENTIRE, (pp. 687-692) Brussels.
45. Lin, K.H.-Y., Yang, C., & Chen, H.H. (2007). What emotions news articles trigger in their readers? In Proceedings of SIGIR (pp. 733-734).
46. Lucey P., Cohn J.F., Kanade T., Saragih J., Ambadar Z., Matthews I. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW) 2010, IEEE.
47. Maas, A., Daly, R., Pham, P., Huang, D., Ng, A., & Potts, C. (2011). Learning word vectors for sentiment analysis. In Proceedings of the association for computational linguistics, ACL 2011, Portland.
48. Mansoorizadeh M., Charkari N.M. Multimodal information fusion application to human emotion recognition from face and speech. Multimedia Tools and Applications 2010, 49(2):277-297.
49. Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006). The eNTERFACE'05 audio-visual emotion database. In Proceedings of the first IEEE workshop on multimedia database management.
50. Mase K. Recognition of facial expression from optical flow. IEICE Transactions 1991, E74(10):3474-3483.
51. Matsumoto D. More evidence for the universality of a contempt expression. Motivation and Emotion 1992, 16(4):363-368.
52. McKay Cory, Ichiro Fujinaga, Philippe Depalle jAudio: A feature extraction library. Proceedings of the International Conference on Music Information Retrieval 2005, 600-603.
53. Melville, Prem, Gryc, Wojciech, & Lawrence, Richard D. (2009). Sentiment analysis of blogs by combining lexical knowledge with text classification. In Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining.
54. Mishne, G. (2005), Experiments with mood classification in blog posts. In Proceedings of the 1st workshop on stylistic analysis of text forinformation access, Style 2005, Brazil.
55. Murray I.R., Arnott J.L. Toward the simulation of emotion in synthetic speech: a review of the literature of human vocal emotion. The Journal of the Acoustical Society of America 1993, 93(2):1097-1108.
56. Navas E., Hernez L. An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional TTS. IEEE Transactions on Audio, Speech and Language Processing 2006, 14:1117-1127.
57. Ortony A., Terence J. What's basic about basic emotions?. Psychological Review 1990, 97(3):315.
58. Otsuka T., Ohya J. A study of transformation of facial expressions based on expression recognition from temproal image sequences, Tech. rep. 1997, Institute of Electronic, Information, and Communications Engineers (IEICE).
59. Pak, A., & Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of LREC.
60. Paleari, M., & Huet, B. (2008). Toward emotion indexing of multimedia excerpts. In CBMI, London.
61. Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd meeting of the association for computational linguistics, Barcelona.
62. Pantic, M., Valstar, M. F., Rademaker, R., & Maat, L. (2005). Web-based database for facial expression analysis. In Proc. 13th ACM int'l conf. multimedia, Multimedia'05 (pp. 317-321).
63. Picard R. Affective computing 1997, The MIT Press, Boston.
64. Poria S., Cambria E., Winterstein G., Huang G.-B. Sentic patterns: Dependency-based rules for concept-level sentiment analysis. Knowledge-Based Systems 2014, 69:45-63.
65. Poria S., Gelbukh A., Hussain A., Das D., Bandyopadhyay S. Enhanced SenticNet with affective labels for concept-based opinion mining. IEEE Intelligent Systems 2013, 28(2):31-38.
66. Pudil P., Ferri F., Novovicova J., Kittler J. Floating search method for feature selection with non monotonic criterion functions. Pattern Recognition 1994, 2:279-283.
67. Pun T., Alecu T., Chanel G., Kronegg J., Voloshynovskiy S. Brain-computer interaction research at the computer vision and multimedia laboratory. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2006, 14(2):210-213.
68. Qi, H., & Wang, X.L. (2001). Multisensor data fusion in distributed sensor. Networks using mobile agents.
69. Riloff, E., & Wiebe, J. (2003). Learning extraction patterns for subjective expressions. In Conference on empirical methods in natural language processing, EMNLP-03 (pp. 105-112).
70. Rosenblum M., Yacoob Y., Davis L. Human expression recognition from motion using a radial basis function network architecture. IEEE Transactions on Neural Networks 1996, 7:1121-1138.
71. Sato, J., & Morishima, S. (1996). Emotion modeling in speech production using emotion space. In Proc. IEEE int. workshop on robot and human communication (pp. 472-477).
72. Scherer, K. R. (1996). Adding the affective dimension: a new look in speech analysis and synthesis. In Proc. international conf. on spoken language processing(pp. 1808-1811).
73. Schuller, B., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009). Acoustic emotion recognition: A benchmark comparison of performances. In IEEE workshop on automatic speech recognition & understanding, ASRU'09(pp. 552-557).
74. Shan, C., Gong, S., & McOwan, P. (2007). Beyond facial expressions: Learning human emotion from body gestures. In BMVC, Warwick.
75. Sidorov G., Miranda-Jiménez S., Viveros-Jiménez F., Gelbukh A., Castro-Sánchez N., Velásquez F., et al. Empirical study of machine learning based approach for opinion mining in tweets. Lecture Notes in Artificial Intelligence 2013, 7629:1-14.
76. Sidorov G., Velasquez F., Stamatatos E., Gelbukh A., Chanona-Hernández L. Syntactic dependency-based N-grams as classification features. Lecture Notes in Artificial Intelligence 2013, 7630:1-11.
77. Sidorov G., Velasquez F., Stamatatos E., Gelbukh A., Chanona-Hernández L. Syntactic dependency-based N-grams: More evidence of usefulness in classification. Lecture Notes in Computer Science 2013, 7816:13-24.
78. Speer R., Havasi C. ConceptNet 5: A large semantic network for relational knowledge. Theory and applications of natural language processing 2012, Springer, (Chapter 6). E. Hovy, M. Johnson, G. Hirst (Eds.).
79. Strapparava, C., & Mihalcea, R. (2007). Semeval-2007 task 14: Affective text. In Proceedings of the 4th international workshop on the semantic evaluations, SemEval 2007, Prague.
80. Strapparava, C., & Mihalcea, R. (2008). Learning to identify emotions in text. In Proceedings of the ACM conference on applied computing, ACM-SAC 2008, Fortaleza.
81. Strapparava, C., & Valitutti, A. (2004). WordNet-affect: an affective extension of WordNet. In Proceedings of the 4th international conference on language resources and evaluation, Lisbon.
82. Turney, P. (2002). Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th annual meeting of the association for computational linguistics, ACL 2002, (pp. 417-424) Philadelphia.
83. Tzanetakis G. Music genre classification of audio signal. IEEE Transactions on Speech and Audio Processing 2002, 10(5).
84. Ueki N., Morishima S., Yamada H., Harashima H. Expression analysis/synthesis system based on emotion space constructed by multilayered neural network. Systems and Computers in Japan 1994, 25(13):95-103.
85. Valstar, Michel François, Jiang, Bihan, Mehu, Marc, Pantic, Maja, & Scherer, Klaus (2011). The first facial expression recognition and analysis challenge. In 2011 IEEE international conference on automatic face & gesture recognition and workshops, FG 2011 (pp. 921-926).
86. Wawer A. Extracting emotive patterns for languages with rich morphology. International Journal of Computational Linguistics and Applications 2012, 3(1):11-24.
87. Wiebe, J. (2010). Learning subjective adjectives from corpora. In Proceedings of the American association for artificial intelligence, AAAI 2000, (pp. 735-740) Austin, Texas, 2000.
88. Wilson, T., Wiebe, J., & Hoffmann, P. (2005). Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of human language technologies conference/conference on empirical methods in natural language processing, HLT/EMNLP 2005, Vancouver.
89. Xia R., Zong C.Q., Hu X.L., Cambria E. Feature ensemble plus sample selection: A comprehensive approach to domain adaptation for sentiment classification. IEEE Intelligent Systems 2013, 28(3):10-18.
90. Yacoob Y., Davis L. Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 1996, 18:636-642.
91. Yongjin W., Ling G., Venetsanopoulos A.N. Kernel cross-modal factor analysis for information fusion with application to bimodal emotion recognition. IEEE Transactions on Multimedia 2012, 14(3):597-607.
92. Zeng Z., Tu J., Liu M., Huang T., Pianfetti B., Roth D., et al. Audio-visual affect recognition. IEEE Transactions on Multimedia 2007, 9(2):424-428.
93. Zhibing, X., & Ling, G. (2013). Multimodal information fusion of audiovisual emotion recognition using novel information theoretic tools. In 2013 IEEE international conference on multimedia and expo, ICME, Vol. 1, no. 6 (pp. 15-19).