A review of affective computing: From unimodal analysis to multimodal fusion

Information Fusion

Volumn 37, Issue , 2017, Pages 98-125

  Poria, Soujanya ^a   Cambria, Erik ^c   Bajpai, Rajiv ^b   Hussain, Amir ^a  

^a UNIVERSITY OF STIRLING   (United Kingdom)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

Affective computing; Audio, visual and text information fusion; Multimodal affect analysis; Multimodal fusion; Sentiment analysis

Indexed keywords

ARTIFICIAL INTELLIGENCE; SENTIMENT ANALYSIS; SOCIAL NETWORKING (ONLINE); SOCIAL SCIENCES COMPUTING;

AFFECT ANALYSIS; AFFECTIVE COMPUTING; FUSION TECHNIQUES; INTERDISCIPLINARY RESEARCH; LITERATURE REVIEWS; MULTI-MODAL FUSION; MULTIMODAL ANALYSIS; NATURAL LANGUAGE PROCESSING;

MODAL ANALYSIS;

EID: 85011844403     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2017.02.003     Document Type: Article

References (300)

1. [1] Balazs, J., Velásquez, J., Opinion mining and information fusion: a survey. Inf. Fusion, 27(95-110), 2016.
2. [2] Sun, S., Luo, C., Chen, J., A review of natural language processing techniques for opinion mining systems. Inf. Fusion, 36(10-25), 2017.
3. [3] Cambria, E., Wang, H., White, B., Guest editorial: big social data analysis. Knowl.-Based Syst. 69 (2014), 1–2.
4. [4] Rosas, V., Mihalcea, R., Morency, L.-P., Multimodal sentiment analysis of spanish online videos. IEEE Intell. Syst. 28:3 (2013), 38–45.
5. [5] Qi, H., Wang, X., Iyengar, S.S., Chakrabarty, K., Multisensor data fusion in distributed sensor networks using mobile agents. Proceedings of 5th International Conference on Information Fusion, 2001, 11–16.
6. [6] Morency, L.-P., Mihalcea, R., Doshi, P., Towards multimodal sentiment analysis: harvesting opinions from the web. Proceedings of the 13th international conference on multimodal interfaces, 2011, ACM, 169–176.
7. [7] Shimojo, S., Shams, L., Sensory modalities are not separate modalities: plasticity and interactions. Curr.Opin.Neurobiol. 11:4 (2001), 505–509.
8. [8] D'mello, S.K., Kory, J., A review and meta-analysis of multimodal affect detection systems. ACM Comput. Surv. 47:3 (2015), 43–79.
9. [9] Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S., A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Trans. pattern Anal.Mach.Intell. 31:1 (2009), 39–58.
10. [10] Darwin, C., The Expression of the Emotions in Man and Animals. 1872, John Murray, London.
11. [11] Ekman, P., Keltner, D., Universal facial expressions of emotion. California Mental Health Res. Digest 8:4 (1970), 151–158.
12. [12] Parrott, W.G., Emotions in Social Psychology: Essential Readings. 2001, Psychology Press.
13. [13] Dalgleish, T., Power, M.J., Handbook of Cognition and Emotion. 1999, Wiley Online Library.
14. [14] Prinz, J.J., Gut Reactions: A Perceptual Theory of Emotion. 2004, Oxford University Press.
15. [15] Russell, J.A., Core affect and the psychological construction of emotion. Psychol.Rev., 110(1), 2003, 145.
16. [16] Osgood, C.E., The nature and measurement of meaning. Psychol.Bull. 49:3 (1952), 197–237.
17. [17] Russell, J.A., Affective space is bipolar. J.Personality Social Psychol. 37:3 (1979), 345–356.
18. [18] Whissell, C., The dictionary of affect in language. Emotion, 4(113-131), 1989, 94.
19. [19] Plutchik, R., Emotion: A Psychoevolutionary Synthesis. 1980, Harpercollins College Division.
20. [20] Freitas, A., Castro, E., Facial expression: the effect of the smile in the treatment of depression. empirical study with Portuguese subjects. Emotional Expression, 2009, 127–140.
21. [21] Mehrabian, A., Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr. Psychol. 14:4 (1996), 261–292.
22. [22] Fontaine, J.R., Scherer, K.R., Roesch, E.B., Ellsworth, P.C., The world of emotions is not two-dimensional. Psychol.Sci. 18:12 (2007), 1050–1057.
23. [23] Cochrane, T., Eight dimensions for the emotions. Social Sci. Inf. 48:3 (2009), 379–420.
24. [24] Cambria, E., Livingstone, A., Hussain, A., The hourglass of emotions. Esposito, A., Vinciarelli, A., Hoffmann, R., Muller, V., (eds.) Cognitive Behavioral Systems Lecture Notes in Computer Science, 7403, 2012, Springer, Berlin Heidelberg, 144–157.
25. [25] Cambria, E., Fu, J., Bisio, F., Poria, S., AffectiveSpace 2: enabling affective intuition for concept-level sentiment analysis. Proceedings of AAAI, 2015, 508–514 Austin.
26. [26] Pérez-Rosas, V., Mihalcea, R., Morency, L.-P., Utterance-level multimodal sentiment analysis. ACL (1), 2013, 973–982.
27. [27] Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., Morency, L.-P., Youtube movie reviews: Sentiment analysis in an audio-visual context. Intell. Syst. IEEE 28:3 (2013), 46–53.
28. [28] Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., Mcrorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., et al. The humaine database: addressing the collection and annotation of naturalistic and induced emotional data. Affective Computing Intell.Interact., 2007, Springer, 488–500.
29. [29] Douglas-Cowie, E., Cowie, R., Schroder, M., A new emotion database: considerations, sources and scope. ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000, 39–44.
30. [30] McKeown, G., Valstar, M., Cowie, R., Pantic, M., Schroder, M., The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. Affective Comput. IEEE Trans. 3:1 (2012), 5–17.
31. [31] Douglas-Cowie, E., Cowie, R., Cox, C., Amier, N., Heylen, D., The sensitive artificial listner: an induction technique for generating emotionally coloured conversation. LREC Workshop Corpora Res. Emot. Affect, 2008.
32. [32] Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S., Iemocap: interactive emotional dyadic motion capture database. Lang.Resour.Eval. 42:4 (2008), 335–359.
33. [33] Martin, O., Kotsia, I., Macq, B., Pitas, I., The enterface’05 audio-visual emotion database. Data Engineering Workshops, 2006. Proceedings. 22nd International Conference on, 2006, IEEE 8–8.
34. [34] E. Paul, W. Friesen, Facial action coding system investigator's guide, 1978.
35. [35] Izard, C.E., Dougherty, L.M., Hembree, E.A., A System for Identifying Affect Expressions by Holistic Judgments (AFFEX). 1983, Instructional Resources Center, University of Delaware.
36. [36] Kring, A.M., Sloan, D., The facial expression coding system (faces): a users guide. Unpublished manuscript, 1991.
37. [37] Ekman, P., Friesen, W.V., Hager, J.C., Facs investigator's guide. A Human Face, 2002.
38. [38] P. Ekman, E. Rosenberg, J. Hager, Facial action coding system affect interpretation dictionary (facsaid), 1998.
39. [39] Ekman, P., Rosenberg, E.L., What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS). 1997, Oxford University Press, USA.
40. [40] Matsumoto, D., More evidence for the universality of a contempt expression. Motivation Emotion 16:4 (1992), 363–368.
41. [41] Rinn, W.E., The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychol.Bull., 95(1), 1984, 52.
42. [42] Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J., Measuring facial expressions by computer image analysis. Psychophysiology 36:02 (1999), 253–263.
43. [43] Breidt, M., Wallraven, C., Cunningham, D.W., Bulthoff, H., Facial animation based on 3d scans and motion capture. Siggraph’03 Sketches and Applications, 2003.
44. [44] Parke, F.I., Waters, K., Computer facial animation. 2008, CRC Press.
45. [45] Tao, H., Chen, H.H., Wu, W., Huang, T.S., Compression of mpeg-4 facial animation parameters for transmission of talking heads. Circuits Syst. Video Technol. IEEE Trans. 9:2 (1999), 264–276.
46. [46] Yacoob, Y., Davis, L., Computing spatio-temporal representations of human faces. Computer Vision and Pattern Recognition, 1994, IEEE, 70–75.
47. [47] Black, M.J., Yacoob, Y., Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. Computer Vision, 1995. Proceedings., Fifth International Conference on, 1995, IEEE, 374–381.
48. [48] Zhang, Z., Feature-based facial expression recognition: Sensitivity analysis and experiments with a multilayer perceptron. Int.J.Pattern Recognit.Artif. Intell. 13:06 (1999), 893–911.
49. [49] Haro, A., Flickner, M., Essa, I., Detecting and tracking eyes by using their physiological properties, dynamics, and appearance. Computer Vision and Pattern Recognition, 1, 2000, IEEE, 163–168.
50. [50] Jones, M.J., Poggio, T., Multidimensional morphable models. Computer Vision, 1998. Sixth International Conference on, 1998, IEEE, 683–688.
51. [51] Cootes, T.F., Edwards, G.J., Taylor, C.J., et al. Active appearance models. IEEE Trans.Pattern Anal.Mach.Intell. 23:6 (2001), 681–685.
52. [52] Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J., Classifying facial actions. Pattern Anal. Mach. Intell. IEEE Trans. 21:10 (1999), 974–989.
53. [53] Tian, Y.-l., Kanade, T., Cohn, J.F., Recognizing action units for facial expression analysis. Pattern Anal. Mach. Intell. IEEE Trans. 23:2 (2001), 97–115.
54. [54] Fasel, B., Luettin, J., Recognition of asymmetric facial action unit activities and intensities. Pattern Recognition, 2000. Proceedings. 15th International Conference on, 1, 2000, IEEE, 1100–1103.
55. [55] Lyons, M.J., Budynek, J., Akamatsu, S., Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21:12 (1999), 1357–1362.
56. [56] Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., Movellan, J., Dynamics of facial expression extracted automatically from video. Image Vision Comput. 24:6 (2006), 615–625.
57. [57] Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S., Facial expression recognition from video sequences: temporal and static modeling. Comput. Vision Image Underst. 91:1 (2003), 160–187.
58. [58] Wang, Y., Ai, H., Wu, B., Huang, C., Real time facial expression recognition with adaboost. Proceedings of the 17th International Conference on, 3, 2004, IEEE, 926–929.
59. [59] Lanitis, A., Taylor, C.J., Cootes, T.F., Automatic face identification system using flexible appearance models. Image Vision Comput. 13:5 (1995), 393–401.
60. [60] Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J., Active shape models-their training and application. Comput.Vision Image Underst. 61:1 (1995), 38–59.
61. [61] Blanz, V., Vetter, T., A morphable model for the synthesis of 3d faces. Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999, ACM Press/Addison-Wesley Publishing Co., 187–194.
62. [62] Ohta, H., Saji, H., Nakatani, H., Recognition of facial expressions using muscle-based feature models. Pattern Recognition, 1998. Proceedings. Fourteenth International Conference on, 2, 1998, IEEE, 1379–1381.
63. [63] Cohen, I., Sebe, N., Gozman, F., Cirelo, M.C., Huang, T.S., Learning bayesian network classifiers for facial expression recognition both labeled and unlabeled data. Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, 1, 2003, IEEE, I–595.
64. [64] Kimura, S., Yachida, M., Facial expression recognition and its degree estimation. Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on, 1997, IEEE, 295–300.
65. [65] Verma, R., Davatzikos, C., Loughead, J., Indersmitten, T., Hu, R., Kohler, C., Gur, R.E., Gur, R.C., Quantification of facial expressions using high-dimensional shape transformations. J.Neurosci.Methods 141:1 (2005), 61–73.
66. [66] Baltrušaitis, T., Robinson, P., Morency, L.-P., 3d constrained local model for rigid and non-rigid facial tracking. Computer Vision and Pattern Recognition, 2012, IEEE, 2610–2617.
67. [67] Morency, L.-P., Whitehill, J., Movellan, J., Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. Automatic Face & Gesture Recognition, 2008. FG’08. 8th IEEE International Conference on, 2008, IEEE, 1–8.
68. [68] Yeasin, M., Bullot, B., Sharma, R., From facial expression to level of interest: a spatio-temporal approach. Computer Vision and Pattern Recognition, 2, 2004, IEEE, II–922.
69. [69] Lien, J.J.-J., Kanade, T., Cohn, J.F., Li, C.-C., Detection, tracking, and classification of action units in facial expression. Robo. Auton. Syst. 31:3 (2000), 131–146.
70. [70] Chang, Y., Hu, C., Turk, M., Probabilistic expression analysis on manifolds. Computer Vision and Pattern Recognition, 2, 2004, IEEE, II–520.
71. [71] Kring, A.M., Sloan, D.M., The facial expression coding system (faces): development, validation, and utility. Psychol.Assess., 19(2), 2007, 210.
72. [72] Davatzikos, C., Measuring biological shape using geometry-based shape transformations. Image Vision Comput. 19:1 (2001), 63–74.
73. [73] Wen, Z., Huang, T.S., Capturing subtle facial motions in 3d face tracking. Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, 2003, IEEE, 1343–1350.
74. [74] Pantic, M., Rothkrantz, L.J., Expert system for automatic analysis of facial expressions. Image Vision Comput. 18:11 (2000), 881–905.
75. [75] Pantic, M., Rothkrantz, L.J., Automatic analysis of facial expressions: The state of the art. Pattern Anal. Mach. Intell. IEEE Trans. 22:12 (2000), 1424–1445.
76. [76] Fasel, B., Luettin, J., Automatic facial expression analysis: a survey. Pattern Recognit. 36:1 (2003), 259–275.
77. [77] De Meijer, M., The contribution of general features of body movement to the attribution of emotions. J. Nonverbal Behav. 13:4 (1989), 247–268.
78. [78] Kapur, A., Kapur, A., Virji-Babul, N., Tzanetakis, G., Driessen, P.F., Gesture-based affective computing on motion capture data. Affective Computing and Intelligent Interaction, 2005, Springer, 1–7.
79. [79] Piana, S., Staglianò, A., Camurri, A., Odone, F., A set of full-body movement features for emotion recognition to help children affected by autism spectrum condition. IDGEI International Workshop, 2013.
80. [80] Piana, S., Stagliano, A., Odone, F., Verri, A., Camurri, A., Real-time automatic emotion recognition from body gestures. arXiv preprint arXiv:1402.5047, 2014.
81. [81] Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., Karpouzis, K., Multimodal emotion recognition from expressive faces, body gestures and speech. Artificial intelligence and innovations 2007: From theory to applications, 2007, Springer, 375–388.
82. [82] Balomenos, T., Raouzaiou, A., Ioannou, S., Drosopoulos, A., Karpouzis, K., Kollias, S., Emotion analysis in man-machine interaction systems. Machine learning for multimodal interaction, 2004, Springer, 318–328.
83. [83] Hinton, G.E., Osindero, S., Teh, Y.-W., A fast learning algorithm for deep belief nets. Neural Comput. 18:7 (2006), 1527–1554.
84. [84] Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012, 1097–1105.
85. [85] LeCun, Y., Kavukcuoglu, K., Farabet, C., et al. Convolutional networks and applications in vision. ISCAS, 2010, 253–256.
86. [86] Kavukcuoglu, K., Sermanet, P., Boureau, Y.-L., Gregor, K., Mathieu, M., Cun, Y.L., Learning convolutional feature hierarchies for visual recognition. Advances in neural information processing systems, 2010, 1090–1098.
87. [87] Hamel, P., Eck, D., Learning features from music audio with deep belief networks. ISMIR, 2010, Utrecht, The Netherlands, 339–344.
88. [88] Chaturvedi, I., Ong, Y.-S., Tsang, I., Welsch, R., Cambria, E., Learning word dependencies in text by means of a deep recurrent belief network. Knowl.-Based Syst. 108 (2016), 144–154.
89. [89] Hinton, G., A practical guide to training restricted boltzmann machines. Momentum, 9(1), 2010, 926.
90. [90] Xu, C., Cetintas, S., Lee, K.-C., Li, L.-J., Visual sentiment prediction with deep convolutional neural networks. arXiv preprint arXiv:1411.5731, 2014.
91. [91] You, Q., Luo, J., Jin, H., Yang, J., Robust image sentiment analysis using progressively trained and domain transferred deep networks. arXiv preprint arXiv:1509.06041, 2015.
92. [92] Xu, B., Fu, Y., Jiang, Y.-G., Li, B., Sigal, L., Heterogeneous knowledge transfer in video emotion recognition, attribution and summarization. arXiv preprint arXiv:1511.04798, 2015.
93. [93] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M., Learning spatiotemporal features with 3d convolutional networks. arXiv preprint arXiv:1412.0767, 2014.
94. [94] Poria, S., Chaturvedi, I., Cambria, E., Hussain, A., Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings of ICDM, 2016, Barcelona.
95. [95] Wu, C.-H., Yeh, J.-F., Chuang, Z.-J., Emotion perception and recognition from speech. Affective Information Processing, 2009, Springer, 93–110.
96. [96] Morrison, D., Wang, R., De Silva, L.C., Ensemble methods for spoken emotion recognition in call-centres. Speech Commun. 49:2 (2007), 98–112.
97. [97] Wu, C.-H., Liang, W.-B., Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels. IEEE Trans. Affective Comput. 2:1 (2011), 10–21.
98. [98] Murray, I.R., Arnott, J.L., Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. J. Acoust. Soc. Am. 93:2 (1993), 1097–1108.
99. [99] Luengo, I., Navas, E., Hernáez, I., Sánchez, J., Automatic emotion recognition using prosodic parameters. Interspeech, 2005, 493–496.
100. [100] Koolagudi, S.G., Kumar, N., Rao, K.S., Speech emotion recognition using segmental level prosodic analysis. Devices and Communications (ICDeCom), 2011 International Conference on, 2011, IEEE, 1–5.
101. [101] Västfjäll, D., Kleiner, M., Emotion in product sound design. Proceedings of Journées Design Sonore, 2002.
102. [102] Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E., How to find trouble in communication. Speech Commun. 40:1 (2003), 117–143.
103. [103] Lee, C.M., Narayanan, S.S., Toward detecting emotions in spoken dialogs. IEEE Trans.Speech Audio Process. 13:2 (2005), 293–303.
104. [104] Hirschberg, J., Benus, S., Brenier, J.M., Enos, F., Friedman, S., Gilman, S., Girand, C., Graciarena, M., Kathol, A., Michaelis, L., et al. Distinguishing deceptive from non-deceptive speech. INTERSPEECH, 2005, 1833–1836.
105. [105] Devillers, L., Vidrascu, L., Lamel, L., Challenges in real-life emotion annotation and machine learning based detection. Neural Netw. 18:4 (2005), 407–422.
106. [106] Vogt, T., André, E., Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on, 2005, IEEE, 474–477.
107. [107] Eyben, F., Wöllmer, M., Schuller, B., Openear—introducing the munich open-source emotion and affect recognition toolkit. 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, 2009, IEEE, 1–6.
108. [108] Levenson, R.W., Human emotion: a functional view. Nat.Emotion 1 (1994), 123–126.
109. [109] Datcu, D., Rothkrantz, L., Semantic audio-visual data fusion for automatic emotion recognition. Euromedia’2008, 2008.
110. [110] Dellaert, F., Polzin, T., Waibel, A., Recognizing emotion in speech. Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on, 3, 1996, IEEE, 1970–1973.
111. [111] Johnstone, T., Emotional speech elicited using computer games. Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on, 3, 1996, IEEE, 1985–1988.
112. [112] Chen, L.S.-H., Joint Processing of Audio-Visual Information for the Recognition of Emotional Expressions in Human-Computer Interaction, 2000, Citeseer Ph.D. thesis.
113. [113] Navas, E., Hernaez, I., Luengo, I., An objective and subjective study of the role of semantics and prosodic features in building corpora for emotional tts. Audio Speech Lang. Process. IEEE Transac. 14:4 (2006), 1117–1127.
114. [114] El Ayadi, M., Kamel, M.S., Karray, F., Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit. 44:3 (2011), 572–587.
115. [115] Atassi, H., Esposito, A., A speaker independent approach to the classification of emotional vocal expressions. Tools with Artificial Intelligence, 2008. ICTAI’08. 20th IEEE International Conference on, 2, 2008, IEEE, 147–152.
116. [116] Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W.F., Weiss, B., A database of german emotional speech. Interspeech, 5, 2005, 1517–1520.
117. [117] Pudil, P., Ferri, F., Novovicova, J., Kittler, J., Floating search methods for feature selection with nonmonotonic criterion functions. Pattern Recognition, 1994. Vol. 2-Conference B: Computer Vision & Image Processing., Proceedings of the 12th IAPR International. Conference on, 2, 1994, IEEE, 279–283.
118. [118] Scherer, K.R., Adding the affective dimension: a new look in speech analysis and synthesis. ICSLP, 1996.
119. [119] Huang, Z., Dong, M., Mao, Q., Zhan, Y., Speech emotion recognition using cnn. Proceedings of the ACM International Conference on Multimedia, 2014, ACM, 801–804.
120. [120] Graves, A., Fernández, S., Schmidhuber, J., Bidirectional lstm networks for improved phoneme classification and recognition. Artificial Neural Networks: Formal Models and Their Applications–ICANN 2005, 2005, Springer, 799–804.
121. [121] Eyben, F., Wöllmer, M., Graves, A., Schuller, B., Douglas-Cowie, E., Cowie, R., On-line emotion recognition in a 3-d activation-valence-time continuum using acoustic and linguistic cues. J. Multimodal User Interfaces 3:1-2 (2010), 7–19.
122. [122] Anand, N., Verma, P., Convoluted feelings convolutional and recurrent nets for detecting emotion from audio data. Technical Report, 2015, Stanford University.
123. [123] Han, K., Yu, D., Tashev, I., Speech emotion recognition using deep neural network and extreme learning machine. Interspeech, 2014, 223–227.
124. [124] Tajadura-Jiménez, A., Västfjäll, D., Auditory-induced emotion: a neglected channel for communication in human-computer interaction. Affect and Emotion in Human-Computer Interaction, 2008, Springer, 63–74.
125. [125] Vogt, T., André, E., Wagner, J., Automatic recognition of emotions from speech: a review of the literature and recommendations for practical realisation. Affect and emotion in human-computer interaction, 2008, Springer, 75–91.
126. [126] Strapparava, C., Valitutti, A., Wordnet affect: an affective extension of wordnet. LREC, 4, 2004, 1083–1086.
127. [127] Alm, C.O., Roth, D., Sproat, R., Emotions from text: machine learning for text-based emotion prediction. Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, 2005, Association for Computational Linguistics, 579–586.
128. [128] Mishne, G., et al. Experiments with mood classification in blog posts. Proceedings of ACM SIGIR 2005 workshop on stylistic analysis of text for information access, 19, 2005, Citeseer, 321–327.
129. [129] Oneto, L., Bisio, F., Cambria, E., Anguita, D., Statistical learning theory and ELM for big social data analysis. IEEE Comput. Intell. Mag. 11:3 (2016), 45–55.
130. [130] Yang, C., Lin, K.H.-Y., Chen, H.-H., Building emotion lexicon from weblog corpora. Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, 2007, Association for Computational Linguistics, 133–136.
131. [131] Chaumartin, F.-R., Upar7: a knowledge-based system for headline sentiment tagging. Proceedings of the 4th International Workshop on Semantic Evaluations, 2007, Association for Computational Linguistics, 422–425.
132. [132] Esuli, A., Sebastiani, F., Sentiwordnet: a publicly available lexical resource for opinion mining. Proceedings of LREC, 6, 2006, Citeseer, 417–422.
133. [133] Lin, K.H.-Y., Yang, C., Chen, H.-H., What emotions do news articles trigger in their readers?. Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, 2007, ACM, 733–734.
134. [134] Hu, M., Liu, B., Mining and summarizing customer reviews. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, ACM, 168–177.
135. [135] Cambria, E., Affective computing and sentiment analysis. IEEE Intell. Syst. 31:2 (2016), 102–107.
136. [136] Pang, B., Lee, L., Vaithyanathan, S., Thumbs up?: sentiment classification using machine learning techniques. Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, 2002, Association for Computational Linguistics, 79–86.
137. [137] Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C., Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the conference on empirical methods in natural language processing (EMNLP), 1631, 2013, Citeseer, 1642.
138. [138] Yu, H., Hatzivassiloglou, V., Towards answering opinion questions: separating facts from opinions and identifying the polarity of opinion sentences. Proceedings of the 2003 conference on Empirical methods in natural language processing, 2003, Association for Computational Linguistics, 129–136.
139. [139] Melville, P., Gryc, W., Lawrence, R.D., Sentiment analysis of blogs by combining lexical knowledge with text classification. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, 2009, ACM, 1275–1284.
140. [140] Turney, P.D., Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. Proceedings of the 40th annual meeting on association for computational linguistics, 2002, Association for Computational Linguistics, 417–424.
141. [141] Hu, X., Tang, J., Gao, H., Liu, H., Unsupervised sentiment analysis with emotional signals. Proceedings of the 22nd international conference on World Wide Web, 2013, International World Wide Web Conferences Steering Committee, 607–618.
142. [142] Gangemi, A., Presutti, V., Reforgiato Recupero, D., Frame-based detection of opinion holders and topics: a model and a tool. Comput. Intell. Mag. IEEE 9:1 (2014), 20–30, 10.1109/MCI.2013.2291688.
143. [143] Cambria, E., Hussain, A., Sentic Computing: A Common-Sense-Based Framework for Concept-Level Sentiment Analysis. 2015, Springer, Cham, Switzerland.
144. [144] Kanayama, H., Nasukawa, T., Fully automatic lexicon expansion for domain-oriented sentiment analysis. Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, 2006, Association for Computational Linguistics, 355–363.
145. [145] Blitzer, J., Dredze, M., Pereira, F., et al. Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification. ACL, 7, 2007, 440–447.
146. [146] Pan, S.J., Ni, X., Sun, J.-T., Yang, Q., Chen, Z., Cross-domain sentiment classification via spectral feature alignment. Proceedings of the 19th international conference on World wide web, 2010, ACM, 751–760.
147. [147] Bollegala, D., Weir, D., Carroll, J., Cross-domain sentiment classification using a sentiment sensitive thesaurus. Knowl. Data Eng. IEEE Trans. 25:8 (2013), 1719–1731.
148. [148] Cambria, E., Poria, S., Bajpai, R., Schuller, B., SenticNet 4: a semantic resource for sentiment analysis based on conceptual primitives. Proceedings of COLING, 2016, 2666–2677.
149. [149] Wu, H.-H., Tsai, A.C.-R., Tsai, R.T.-H., Hsu, J.Y.-j., Sentiment value propagation for an integral sentiment dictionary based on commonsense knowledge. Technologies and Applications of Artificial Intelligence (TAAI), 2011 International Conference on, 2011, IEEE, 75–81.
150. [150] Chenlo, J.M., Losada, D.E., An empirical study of sentence features for subjectivity and polarity classification. Inf. Sci. 280 (2014), 275–288.
151. [151] Shah, R., Yu, Y., Verma, A., Tang, S., Shaikh, A., Zimmermann, R., Leveraging multimodal information for event summarization and concept-level sentiment analysis. Knowl.-Based Syst. 108 (2016), 102–109.
152. [152] Gezici, G., Dehkharghani, R., Yanikoglu, B., Tapucu, D., Saygin, Y., Su-sentilab: a classification system for sentiment analysis in twitter. International Workshop on Semantic Evaluation, 2013, 471–477.
153. [153] Poria, S., Cambria, E., Hazarika, D., Vij, P., A deeper look into sarcastic tweets using deep convolutional neural networks. Proceedings of COLING, 2016, 1601–1612.
154. [154] Bravo-Marquez, F., Mendoza, M., Poblete, B., Meta-level sentiment models for big social data analysis. Knowl.-Based Syst. 69 (2014), 86–99.
155. [155] Jaiswal, M., Tabibu, S., Bajpai, R., The truth and nothing but the truth: multimodal analysis for deception detection. ICDM, 2016.
156. [156] Xie, H., Li, X., Wang, T., Lau, R., Wong, T.-L., Chen, L., Wang, F.-L., Li, Q., Incorporating sentiment into tag-based user profiles and resource profiles for personalized search in folksonomy. Inf. Process. Manage. 52 (2016), 61–72.
157. [157] Scharl, A., Hubmann-Haidvogel, A., Jones, A., Fischl, D., Kamolov, R., Weichselbraun, A., Rafelsberger, W., Analyzing the public discourse on works of fiction – detection and visualization of emotion in online coverage about hbo's game of thrones. Inf. Process. Manage. 52:1 (2016), 129–138.
158. [158] Egger, M., Schoder, D., Consumer-oriented tech mining: Integrating the consumer perspective into organizational technology intelligence – the case of autonomous driving. Hawaii International Conference on System Sciences, 2017.
159. [159] Poria, S., Cambria, E., Winterstein, G., Huang, G.-B., Sentic patterns: dependency-based rules for concept-level sentiment analysis. Knowl.-Based Syst. 69 (2014), 45–63.
160. [160] Cambria, E., Hussain, A., Havasi, C., Eckl, C., Common sense computing: from the society of mind to digital intuition and beyond. Fierrez, J., Ortega, J., Esposito, A., Drygajlo, A., Faundez-Zanuy, M., (eds.) Biometric ID Management and Multimodal Communication Lecture Notes in Computer Science, 5707, 2009, Springer, Berlin Heidelberg, 252–259.
161. [161] Hatzivassiloglou, V., McKeown, K.R., Predicting the semantic orientation of adjectives. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, 1997, Association for Computational Linguistics, 174–181.
162. [162] Jia, L., Yu, C., Meng, W., The effect of negation on sentiment analysis and retrieval effectiveness. Proceedings of the 18th ACM conference on Information and knowledge management, 2009, ACM, 1827–1830.
163. [163] Reyes, A., Rosso, P., On the difficulty of automatically detecting irony: beyond a simple case of negation. Knowl. Inf. Syst. 40:3 (2014), 595–614.
164. [164] Chawla, K., Ramteke, A., Iitb-sentiment-analysts: Participation in sentiment analysis in twitter semeval 2013 task. Second Joint Conference on Lexical and Computational Semantics (* SEM), 2, 2013, Citeseer, 495–500.
165. [165] Polanyi, L., Culy, C., Van Den Berg, M., Thione, G.L., Ahn, D., Sentential structure and discourse parsing. Proceedings of the 2004 ACL Workshop on Discourse Annotation, 2004, Association for Computational Linguistics, 80–87.
166. [166] Wolf, F., Gibson, E., Representing discourse coherence: a corpus-based study. Comput. Ling. 31:2 (2005), 249–287.
167. [167] Wellner, B., Pustejovsky, J., Havasi, C., Rumshisky, A., Sauri, R., Classification of discourse coherence relations: an exploratory study using multiple knowledge sources. Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, 2009, Association for Computational Linguistics, 117–125.
168. [168] Ramesh, B.P., Yu, H., Identifying discourse connectives in biomedical text. AMIA Annual Symposium Proceedings, 2010, 2010, American Medical Informatics Association, 657.
169. [169] Liu, B., Sentiment analysis and opinion mining. Synth. Lect. Human Lang. Technol. 5:1 (2012), 1–167.
170. [170] Moilanen, K., Pulman, S., Sentiment composition. Proceedings of the Recent Advances in Natural Language Processing International Conference, 2007, 378–382.
171. [171] Ding, X., Liu, B., Yu, P.S., A holistic lexicon-based approach to opinion mining. Proceedings of the 2008 International Conference on Web Search and Data Mining, 2008, ACM, 231–240.
172. [172] Poria, S., Cambria, E., Gelbukh, A., Aspect extraction for opinion mining with a deep convolutional neural network. Knowl.-Based Syst. 108 (2016), 42–49.
173. [173] Ng, H.T., Goh, W.B., Low, K.L., Feature selection, perceptron learning, and a usability case study for text categorization. ACM SIGIR Forum, 31, 1997, ACM, 67–73.
174. [174] Kim, Y.-H., Hahn, S.-Y., Zhang, B.-T., Text filtering by boosting naive bayes classifiers. Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 2000, ACM, 168–175.
175. [175] Jordan, A., On discriminative vs. generative classifiers: Aa comparison of logistic regression and naive bayes. Adv.Neural Inf.Process.Syst., 14, 2002, 841.
176. [176] Li, Y., McLean, D., Bandar, Z.A., O'shea, J.D., Crockett, K., Sentence similarity based on semantic nets and corpus statistics. Knowl. Data Eng. IEEE Trans. 18:8 (2006), 1138–1150.
177. [177] Phan, X.-H., Nguyen, L.-M., Horiguchi, S., Learning to classify short and sparse text & web with hidden topics from large-scale data collections. Proceedings of the 17th international conference on World Wide Web, 2008, ACM, 91–100.
178. [178] Sahlgren, M., Cöster, R., Using bag-of-concepts to improve the performance of support vector machines in text categorization. Proceedings of the 20th international conference on Computational Linguistics, 2004, Association for Computational Linguistics, 487.
179. [179] Wang, F., Wang, Z., Li, Z., Wen, J.-R., Concept-based short text classification and ranking. Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, 2014, ACM, 1069–1078.
180. [180] Zhang, Y., Liu, B., Semantic text classification of emergent disease reports. Knowledge Discovery in Databases: PKDD 2007, 2007, Springer, 629–637.
181. [181] Wu, C.-E., Tsai, R.T.-H., Using relation selection to improve value propagation in a conceptnet-based sentiment dictionary. Knowl.-Based Syst. 69 (2014), 100–107.
182. [182] Cambria, E., White, B., Jumping NLP curves: a review of natural language processing research. IEEE Comput. Intell. Mag. 9:2 (2014), 48–57.
183. [183] Wilson, T., Wiebe, J., Hoffmann, P., Recognizing contextual polarity in phrase-level sentiment analysis. Proceedings of the conference on human language technology and empirical methods in natural language processing, 2005, Association for Computational Linguistics, 347–354.
184. [184] Asher, N., Benamara, F., Mathieu, Y.Y., Appraisal of opinion expressions in discourse. Lingvisticæ Investigationes 32:2 (2009), 279–292.
185. [185] Narayanan, R., Liu, B., Choudhary, A., Sentiment analysis of conditional sentences. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, 2009, Association for Computational Linguistics, 180–189.
186. [186] Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P., Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12 (2011), 2493–2537.
187. [187] Kalchbrenner, N., Grefenstette, E., Blunsom, P., A convolutional neural network for modelling sentences. CoRR, abs/1404.2188, 2014.
188. [188] Glorot, X., Bordes, A., Bengio, Y., Domain adaptation for large-scale sentiment classification: a deep learning approach. In Proceedings of the Twenty-eight International Conference on Machine Learning, ICML, 2011, 513–520.
189. [189] Poria, S., Cambria, E., Gelbukh, A., Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. Proceedings of EMNLP, 2015, 2539–2544.
190. [190] Kim, Y., Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.
191. [191] Mikolov, T., Chen, K., Corrado, G., Dean, J., Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
192. [192] Chuang, Z.-J., Wu, C.-H., Multi-modal emotion recognition from speech and text. Comput. Ling. Chinese Lang.Process. 9:2 (2004), 45–62.
193. [193] Forbes-Riley, K., Litman, D.J., Predicting emotion in spoken dialogue from multiple knowledge sources. HLT-NAACL, 2004, Citeseer, 201–208.
194. [194] Litman, D.J., Forbes-Riley, K., Predicting student emotions in computer-human tutoring dialogues. Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, 2004, Association for Computational Linguistics, 351.
195. [195] Rigoll, G., Müller, R., Schuller, B., Speech emotion recognition exploiting acoustic and linguistic information sources. Proc. SPECOM, Patras, Greece, 2005, 61–67.
196. [196] Litman, D.J., Forbes-Riley, K., Recognizing student emotions and attitudes on the basis of utterances in spoken tutoring dialogues with both human and computer tutors. Speech Commun. 48:5 (2006), 559–590.
197. [197] Seppi, D., Batliner, A., Schuller, B., Steidl, S., Vogt, T., Wagner, J., Devillers, L., Vidrascu, L., Amir, N., Aharonson, V., Patterns, prototypes, performance: classifying emotional user states. INTERSPEECH, 2008, 601–604.
198. [198] Schuller, B., Recognizing affect from linguistic information in 3d continuous space. Affective Comput. IEEE Trans. 2:4 (2011), 192–205.
199. [199] Rozgic, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Prasad, R., Speech language & multimedia technol., raytheon bbn technol., cambridge, ma, usa. Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, 2012, IEEE, 1–4.
200. [200] Savran, A., Cao, H., Shah, M., Nenkova, A., Verma, R., Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering. Proceedings of the 14th ACM international conference on Multimodal interaction, 2012, ACM, 485–492.
201. [201] Sarkar, C., Bhatia, S., Agarwal, A., Li, J., Feature analysis for computational personality recognition using youtube personality data set. Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, 2014, ACM, 11–14.
202. [202] Alam, F., Riccardi, G., Predicting personality traits using multimodal information. Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, 2014, ACM, 15–18.
203. [203] Ellis, J.G., Jou, B., Chang, S.-F., Why we watch the news: a dataset for exploring sentiment in broadcast video news. Proceedings of the 16th International Conference on Multimodal Interaction, 2014, ACM, 104–111.
204. [204] Siddiquie, B., Chisholm, D., Divakaran, A., Exploiting multimodal affect and semantics to identify politically persuasive web videos. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015, ACM, 203–210.
205. [205] Poria, S., Cambria, E., Hussain, A., Huang, G.-B., Towards an intelligent framework for multimodal affective data analysis. Neural Netw. 63 (2015), 104–116.
206. [206] Cai, G., Xia, B., Convolutional neural networks for multimedia sentiment analysis. National CCF Conference on Natural Language Processing and Chinese Computing, 2015, Springer, 159–167.
207. [207] Ji, R., Cao, D., Lin, D., Cross-modality sentiment analysis for social multimedia. Multimedia Big Data (BigMM), 2015 IEEE International Conference on, 2015, IEEE, 28–31.
208. [208] Yamasaki, T., Fukushima, Y., Furuta, R., Sun, L., Aizawa, K., Bollegala, D., Prediction of user ratings of oral presentations using label relations. Proceedings of the 1st International Workshop on Affect & Sentiment in Multimedia, 2015, ACM, 33–38.
209. [209] Monkaresi, H., Hussain, M.S., Calvo, R.A., Classification of affects using head movement, skin color features and physiological signals. Systems, Man, and Cybernetics (SMC), 2012 IEEE International Conference on, 2012, IEEE, 2664–2669.
210. [210] Wang, S., Zhu, Y., Wu, G., Ji, Q., Hybrid video emotional tagging using users’ eeg and video content. Multimedia tools Appl. 72:2 (2014), 1257–1283.
211. [211] Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S., Analysis of emotion recognition using facial expressions, speech and multimodal information. Proceedings of the 6th international conference on Multimodal interfaces, 2004, ACM, 205–211.
212. [212] Chen, C.-Y., Huang, Y.-K., Cook, P., Visual/acoustic emotion recognition. 2005 IEEE International Conference on Multimedia and Expo, 2005, IEEE, 1468–1471.
213. [213] Gunes, H., Piccardi, M., Fusing face and body display for bi-modal emotion recognition: single frame analysis and multi-frame post integration. Affective Computing and Intelligent Interaction, 2005, Springer, 102–111.
214. [214] Hoch, S., Althoff, F., McGlaun, G., Rigoll, G., Bimodal fusion of emotional data in an automotive environment. Acoustics, Speech, and Signal Processing, 2005. Proceedings.(ICASSP’05). IEEE International Conference on, 2, 2005, IEEE, ii–1085.
215. [215] Kapoor, A., Picard, R.W., Multimodal affect recognition in learning environments. Proceedings of the 13th annual ACM international conference on Multimedia, 2005, ACM, 677–682.
216. [216] Kim, J., Bimodal Emotion Recognition using Speech and Physiological Changes. 2007, INTECH Open Access Publisher.
217. [217] Wang, Y., Guan, L., Recognizing human emotional state from audiovisual signals*. Multimedia IEEE Trans. 10:5 (2008), 936–946.
218. [218] Zeng, Z., Tu, J., Liu, M., Huang, T.S., Multi-stream confidence analysis for audio-visual affect recognition. Affective Computing and Intelligent Interaction, 2005, Springer, 964–971.
219. [219] Gunes, H., Piccardi, M., Affect recognition from face and body: early fusion vs. late fusion. 2005 IEEE international conference on systems, man and cybernetics, 4, 2005, IEEE, 3437–3443.
220. [220] Pal, P., Iyer, A.N., Yantorno, R.E., Emotion detection from infant facial expressions and cries. Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on, 2, 2006, IEEE II–II.
221. [221] Sebe, N., Cohen, I., Gevers, T., Huang, T.S., Emotion recognition based on joint visual and audio cues. Pattern Recognition, 2006. ICPR 2006. 18th International Conference on, 1, 2006, IEEE, 1136–1139.
222. [222] Zeng, Z., Hu, Y., Fu, Y., Huang, T.S., Roisman, G.I., Wen, Z., Audio-visual emotion recognition in adult attachment interview. Proceedings of the 8th international conference on Multimodal interfaces, 2006, ACM, 139–145.
223. [223] Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Raouzaiou, A., Karpouzis, K., Modeling naturalistic affective states via facial and vocal expressions recognition. Proceedings of the 8th international conference on Multimodal interfaces, 2006, ACM, 146–154.
224. [224] D'mello, S., Graesser, A., Mind and body: dialogue and posture for affect detection in learning environments. Frontiers Artif. Intell.Appl., 158, 2007, 161.
225. [225] Gong, S., Shan, C., Xiang, T., Visual inference of human emotion and behaviour. Proceedings of the 9th international conference on Multimodal interfaces, 2007, ACM, 22–29.
226. [226] Han, M.-J., Hsu, J.-H., Song, K.-T., Chang, F.-Y., A new information fusion method for svm-based robotic audio-visual emotion recognition. Systems, Man and Cybernetics, 2007. ISIC. IEEE International Conference on, 2007, IEEE, 2656–2661.
227. [227] Jong-Tae, J., Sang-Wook, S., Kwang-Eun, K., Kwee-Bo, S., Emotion recognition method based on multimodal sensor fusion algorithm. ISIS, Sokcho-City, 2007.
228. [228] Karpouzis, K., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A., Malatesta, L., Kollias, S., Modeling naturalistic affective states via facial, vocal, and bodily expressions recognition. Artifical intelligence for human computing, 2007, Springer, 91–112.
229. [229] Schuller, B., Müeller, R., Höernler, B., Höethker, A., Konosu, H., Rigoll, G., Audiovisual recognition of spontaneous interest within conversations. Proceedings of the 9th international conference on Multimodal interfaces, 2007, ACM, 30–37.
230. [230] Shan, C., Gong, S., McOwan, P.W., Beyond facial expressions: learning human emotion from body gestures. BMVC, 2007, 1–10.
231. [231] Zeng, Z., Tu, J., Liu, M., Huang, T.S., Pianfetti, B., Roth, D., Levinson, S., Audio-visual affect recognition. Multimedia IEEE Trans. 9:2 (2007), 424–428.
232. [232] Haq, S., Jackson, P.J., Edge, J., Audio-visual feature selection and reduction for emotion classification. Proc. Int. Conf. on Auditory-Visual Speech Processing (AVSP’08), Tangalooma, Australia, 2008.
233. [233] Kanluan, I., Grimm, M., Kroschel, K., Audio-visual emotion recognition using an emotion space concept. Signal Processing Conference, 2008 16th European, 2008, IEEE, 1–5.
234. [234] Metallinou, A., Lee, S., Narayanan, S., Audio-visual emotion recognition using gaussian mixture models for face and voice. Multimedia, 2008. ISM 2008. Tenth IEEE International Symposium on, 2008, IEEE, 250–257.
235. [235] Wimmer, M., Schuller, B., Arsic, D., Rigoll, G., Radig, B., Low-level fusion of audio, video feature for multi-modal emotion recognition. VISAPP (2), 2008, 145–151.
236. [236] Bailenson, J.N., Pontikakis, E.D., Mauss, I.B., Gross, J.J., Jabon, M.E., Hutcherson, C.A., Nass, C., John, O., Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int.J.Human-Comput.Stud. 66:5 (2008), 303–317.
237. [237] Castellano, G., Kessous, L., Caridakis, G., Emotion recognition through multiple modalities: face, body gesture, speech. Affect and emotion in human-computer interaction, 2008, Springer, 92–103.
238. [238] Chetty, G., Wagner, M., Goecke, R., A multilevel fusion approach for audiovisual emotion recognition. AVSP, 2008, 115–120.
239. [239] Emerich, S., Lupu, E., Apatean, A., Emotions recognition by speech and facial expressions analysis. Proceedings of the 17th European Signal Processing Conference (EUSIPCO’09), 2009, 1617–1621.
240. [240] Gunes, H., Piccardi, M., Automatic temporal segment detection and affect recognition from face and body display. Syst. Man, Cybern. Part B 39:1 (2009), 64–84.
241. [241] Haq, S., Jackson, P.J., Edge, J., Speaker-dependent audio-visual emotion recognition. AVSP, 2009, 53–58.
242. [242] Khalili, Z., Moradi, M.H., Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of eeg. 2009 International Joint Conference on Neural Networks, 2009, IEEE, 1571–1575.
243. [243] Paleari, M., Benmokhtar, R., Huet, B., Evidence theory-based multimodal emotion recognition. International Conference on Multimedia Modeling, 2009, Springer, 435–446.
244. [244] Rabie, A., Wrede, B., Vogt, T., Hanheide, M., Evaluation and discussion of multi-modal emotion recognition. Computer and Electrical Engineering, 2009. ICCEE’09. Second International Conference on, 1, 2009, IEEE, 598–602.
245. [245] D'Mello, S.K., Graesser, A., Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User-Adapted Interact. 20:2 (2010), 147–187.
246. [246] Dy, M.L.I.C., Espinosa, I.V.L., Go, P.P.V., Mendez, C.M.M., Cu, J.W., Multimodal emotion recognition using a spontaneous filipino emotion database. Human-Centric Computing (HumanCom), 2010 3rd International Conference on, 2010, IEEE, 1–5.
247. [247] Gajsek, R., Štruc, V., Mihelic, F., Multi-modal emotion recognition using canonical correlations and acoustic features. Proceedings of the 2010 20th International Conference on Pattern Recognition, 2010, IEEE Computer Society, 4133–4136.
248. [248] Kessous, L., Castellano, G., Caridakis, G., Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. J. Multimodal User Interfaces 3:1-2 (2010), 33–48.
249. [249] Kim, J., Lingenfelser, F., Ensemble approaches to parametric decision fusion for bimodal emotion recognition. BIOSIGNALS, 2010, 460–463.
250. [250] Mansoorizadeh, M., Charkari, N.M., Multimodal information fusion application to human emotion recognition from face and speech. Multimedia Tools Appl. 49:2 (2010), 277–297.
251. [251] Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., Narayanan, S.S., et al. Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling. INTERSPEECH, 2010, 2362–2365.
252. [252] Glodek, M., Tschechne, S., Layher, G., Schels, M., Brosch, T., Scherer, S., Kächele, M., Schmidt, M., Neumann, H., Palm, G., et al. Multiple classifier systems for the classification of audio-visual emotional states. Affective Computing and Intelligent Interaction, 2011, Springer, 359–368.
253. [253] Banda, N., Robinson, P., Noise analysis in audio-visual emotion recognition. International Conference on Multimodal Interaction, Alicante, Spain, 2011, Citeseer, 1–4.
254. [254] Chanel, G., Rebetez, C., Bétrancourt, M., Pun, T., Emotion assessment from physiological signals for adaptation of game difficulty. Syst. Man Cybern. Part A 41:6 (2011), 1052–1063.
255. [255] Cueva, D.R., Gonçalves, R.A., Cozman, F., Pereira-Barretto, M.R., Crawling to improve multimodal emotion detection. Advances in Soft Computing, 2011, Springer, 343–350.
256. [256] Datcu, D., Rothkrantz, L.J., Emotion recognition using bimodal data fusion. Proceedings of the 12th International Conference on Computer Systems and Technologies, 2011, ACM, 122–128.
257. [257] Jiang, D., Cui, Y., Zhang, X., Fan, P., Ganzalez, I., Sahli, H., Audio visual emotion recognition based on triple-stream dynamic bayesian network models. Affective Computing and Intelligent Interaction, 2011, Springer, 609–618.
258. [258] Lingenfelser, F., Wagner, J., André, E., A systematic discussion of fusion techniques for multi-modal affect recognition tasks. Proceedings of the 13th international conference on multimodal interfaces, 2011, ACM, 19–26.
259. [259] Nicolaou, M.A., Gunes, H., Pantic, M., Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. Affective Comput. IEEE Trans. 2:2 (2011), 92–105.
260. [260] Vu, H.A., Yamazaki, Y., Dong, F., Hirota, K., Emotion recognition based on human gesture and speech information using rt middleware. Fuzzy Systems (FUZZ), 2011 IEEE International Conference on, 2011, IEEE, 787–791.
261. [261] Wagner, J., Andre, E., Lingenfelser, F., Kim, J., Exploring fusion methods for multimodal emotion recognition with missing data. Affective Comput. IEEE Trans. 2:4 (2011), 206–218.
262. [262] Walter, S., Scherer, S., Schels, M., Glodek, M., Hrabal, D., Schmidt, M., Böck, R., Limbrecht, K., Traue, H.C., Schwenker, F., Multimodal emotion classification in naturalistic user behavior. Human-Computer Interaction. Towards Mobile and Intelligent Interaction Environments, 2011, Springer, 603–611.
263. [263] Hussain, M., Monkaresi, H., Calvo, R.A., Combining classifiers in multimodal affect detection. Proceedings of the Tenth Australasian Data Mining Conference-Volume 134, 2012, Australian Computer Society, Inc., 103–108.
264. [264] Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I., Deap: a database for emotion analysis; using physiological signals. Affective Comput. IEEE Trans. 3:1 (2012), 18–31.
265. [265] Lin, J.-C., Wu, C.-H., Wei, W.-L., Error weighted semi-coupled hidden markov model for audio-visual emotion recognition. Multimedia, IEEE Trans. 14:1 (2012), 142–156.
266. [266] Lu, K., Jia, Y., Audio-visual emotion recognition with boosted coupled hmm. Pattern Recognition (ICPR), 2012 21st International Conference on, 2012, IEEE, 1148–1151.
267. [267] Metallinou, A., Wöllmer, M., Katsamanis, A., Eyben, F., Schuller, B., Narayanan, S., Context-sensitive learning for enhanced audiovisual emotion classification. Affective Comput. IEEE Trans. 3:2 (2012), 184–198.
268. [268] Park, J.-S., Jang, G.-J., Seo, Y.-H., Music-aided affective interaction between human and service robot. EURASIP J. Audio, Speech, Music Process. 2012:1 (2012), 1–13.
269. [269] Rashid, M., Abu-Bakar, S., Mokji, M., Human emotion recognition from videos using spatio-temporal and audio features. Visual Comput. 29:12 (2013), 1269–1275.
270. [270] Soleymani, M., Pantic, M., Pun, T., Multimodal emotion recognition in response to videos. Affective Comput. IEEE Trans. 3:2 (2012), 211–223.
271. [271] Tu, B., Yu, F., Bimodal emotion recognition based on speech signals and facial expression. Foundations of Intelligent Systems, 2012, Springer, 691–696.
272. [272] Baltrusaitis, T., Banda, N., Robinson, P., Dimensional affect recognition using continuous conditional random fields. Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on, 2013, IEEE, 1–8.
273. [273] Dobrišek, S., Gajšek, R., Mihelič, F., Pavešić, N., Štruc, V., Towards efficient multi-modal emotion recognition. Int. J. Adv. Robotic Sy., 10(53), 2013.
274. [274] Glodek, M., Reuter, S., Schels, M., Dietmayer, K., Schwenker, F., Kalman filter based classifier fusion for affective state recognition. Multiple Classifier Systems, 2013, Springer, 85–94.
275. [275] Hommel, S., Rabie, A., Handmann, U., Attention and emotion based adaption of dialog systems. Intelligent Systems: Models and Applications, 2013, Springer, 215–235.
276. [276] Krell, G., Glodek, M., Panning, A., Siegert, I., Michaelis, B., Wendemuth, A., Schwenker, F., Fusion of fragmentary classifier decisions for affective state recognition. Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction, 2013, Springer, 116–130.
277. [277] Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., Rigoll, G., Lstm-modeling of continuous emotions in an audiovisual affect recognition framework. Image Vision Comput. 31:2 (2013), 153–163.
278. [278] Chen, L., Yoon, S.-Y., Leong, C.W., Martin, M., Ma, M., An initial analysis of structured video interviews by using multimodal emotion detection. Proceedings of the 2014 workshop on Emotion Representation and Modelling in Human-Computer-Interaction-Systems, 2014, ACM, 1–6.
279. [279] Poria, S., Cambria, E., Howard, N., Huang, G.-B., Hussain, A., Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174 (2016), 50–59.
280. [280] Song, M., Bu, J., Chen, C., Li, N., Audio-visual based emotion recognition-a new approach. Computer Vision and Pattern Recognition, 2, 2004, IEEE, II–1020.
281. [281] Zeng, Z., Hu, Y., Liu, M., Fu, Y., Huang, T.S., Training combination strategy of multi-stream fused hidden markov model for audio-visual affect recognition. Proceedings of the 14th annual ACM international conference on Multimedia, 2006, ACM, 65–68.
282. [282] Petridis, S., Pantic, M., Audiovisual discrimination between laughter and speech. Acoustics, Speech and Signal Processing, 2008. ICASSP 2008. IEEE International Conference on, 2008, IEEE, 5117–5120.
283. [283] Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S., Multimodal fusion for multimedia analysis: a survey. Multimedia Syst. 16:6 (2010), 345–379.
284. [284] Corradini, A., Mehta, M., Bernsen, N.O., Martin, J., Abrilian, S., Multimodal input fusion in human-computer interaction. NATO Sci. Ser.s Sub Ser. III Comput.Syst. Sci., 198, 2005, 223.
285. [285] Iyengar, G., Nock, H.J., Neti, C., Audio-visual synchrony for detection of monologues in video archives. Multimedia and Expo, 2003. ICME’03. Proceedings. 2003 International Conference on, 1, 2003, IEEE, I–329.
286. [286] Adams, W., Iyengar, G., Lin, C.-Y., Naphade, M.R., Neti, C., Nock, H.J., Smith, J.R., Semantic indexing of multimedia content using visual, audio, and text cues. EURASIP J. Adv. Signal Process. 2003:2 (2003), 1–16.
287. [287] Nefian, A.V., Liang, L., Pi, X., Liu, X., Murphy, K., Dynamic bayesian networks for audio-visual speech recognition. EURASIP J. Adv. Signal Process. 2002:11 (2002), 1–15.
288. [288] Nickel, K., Gehrig, T., Stiefelhagen, R., McDonough, J., A joint particle filter for audio-visual speaker tracking. Proceedings of the 7th international conference on Multimodal interfaces, 2005, ACM, 61–68.
289. [289] Potamitis, I., Chen, H., Tremoulis, G., Tracking of multiple moving speakers with multiple microphone arrays. Speech Audio Process. IEEE Trans. 12:5 (2004), 520–529.
290. [290] Gunes, H., Pantic, M., Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners. International conference on intelligent virtual agents, 2010, 371–377.
291. [291] Valstar, M.F., Almaev, T., Girard, J.M., McKeown, G., Mehu, M., Yin, L., Pantic, M., Cohn, J.F., Fera 2015-second facial expression recognition and analysis challenge. Automatic Face and Gesture Recognition, 6, 2015, 1–8.
292. [292] Nicolaou, M.A., Gunes, H., Pantic, M., Automatic segmentation of spontaneous data using dimensional labels from multiple coders. Proceedings of LREC Int'l Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, 2010, 43–48.
293. [293] Chang, K.-h., Fisher, D., Canny, J., Ammon: a speech analysis library for analyzing affect, stress, and mental health on mobile phones. Proc. PhoneSense, 2011, 2011.
294. [294] Zhang, S., Li, L., Zhao, Z., Audio-visual emotion recognition based on facial expression and affective speech. Multimedia and Signal Processing, 2012, Springer, 46–52.
295. [295] Eyben, F., Wöllmer, M., Valstar, M.F., Gunes, H., Schuller, B., Pantic, M., String-based audiovisual fusion of behavioural events for the assessment of dimensional affect. Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, 2011, IEEE, 322–329.
296. [296] Rahman, T., Busso, C., A personalized emotion recognition system using an unsupervised feature adaptation scheme. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012, IEEE, 5117–5120.
297. [297] Jin, Q., Li, C., Chen, S., Wu, H., Speech emotion recognition with acoustic and lexical features. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, IEEE, 4749–4753.
298. [298] Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Prasad, R., Ensemble of svm trees for multimodal emotion recognition. Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, 2012, IEEE, 1–4.
299. [299] DeVault, D., Artstein, R., Benn, G., Dey, T., Fast, E., Gainer, A., Georgila, K., Gratch, J., Hartholt, A., Lhommet, M., et al. Simsensei kiosk: a virtual human interviewer for healthcare decision support. Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, 2014, 1061–1068.
300. [300] Hoque, M.E., Picard, R.W., Acted vs. natural frustration and delight: many people smile in natural frustration. Automatic Face & Gesture Recognition and Workshops, 2011, IEEE, 354–359.