Unimodal and multimodal human perceptionof naturalistic non-basic affective statesduring human-computer interactions

IEEE Transactions on Affective Computing

Volumn 4, Issue 4, 2013, Pages 452-465

  Dmello, Sidney K ^a   Dowell, Nia ^b   Graesser, Art ^b  

^a UNIVERSITY OF NOTRE DAME   (United States)

^b University of Memphis   (United States)

Author keywords

Emotion perception; multimodal affect detection; naturalistic expressions; non basic emotions; superadditivity

Indexed keywords

REGRESSION ANALYSIS;

AFFECT DETECTION; AFFECTIVE STATE; HUMAN OBSERVERS; INHIBITORY EFFECT; LOGISTIC REGRESSION MODELS; NATURALISTIC EXPRESSIONS; NON-BASIC EMOTIONS; SUPERADDITIVITY;

BEHAVIORAL RESEARCH;

EID: 84897043319     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/T-AFFC.2013.19     Document Type: Article

References (66)

1. R.A. Calvo and S.K. DMello,"Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications," IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 18-37, Jan.-June 2010, DOI: 10.1109/T-AFFC.2010.1
2. Z. Zeng et al.,"A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009
3. M. Pantic and L. Rothkrantz,"Toward an Affect-Sensitive Multimodal Human-Computer Interaction," Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sept. 2003, DOI: 10.1109/jproc.2003.817122 (Pubitemid 40890819)
4. M.F. Valstar et al.,"Meta-Analysis of the First Facial Expression Recognition Challenge," IEEE Trans. Systems, Man, and Cybernetics. Part B, Cybernetics, vol. 42, no. 4, pp. 966-979, Aug. 2012
5. R. Picard, Affective Computing. MIT Press, 1997
6. H. Dudley,"Remaking Speech," The J. Acoustical Soc. of Am., vol. 11, p. 165, 1939
7. M. Glodek et al.,"Multiple Classifier Systems for the Classification of Audio-Visual Emotional States," Proc. Fourth Intl. Conf. Affective Computing and Intelligent Interaction (ACII 11), pp. 359-368, 2011
8. J. Lin et al.,"Error Weighted Semi-Coupled Hidden Markov Model for Audio-Visual Emotion Recognition," IEEE Trans. Multimedia, vol. 14, no. 1, pp. 142-156, Feb. 2012
9. A. Metallinou et al.,"Context-Sensitive Learning for Enhanced Audiovisual Emotion Classification," IEEE Trans. Affective Computing, vol. 3, no. 2, pp. 184-198, Apr.-June 2012
10. K. Karpouzis et al.,"Modeling Naturalistic Affective States via Facial, Vocal, and Bodily Expressions Recognition," Artificial Intelligence for Human Computing, T. Huang, ed., pp. 91-112, Springer-Verlag, 2007
11. M. Paleari et al.,"Evidence Theory-Based Multimodal Emotion Recognition," Proc. 15th Intl Multimedia Modeling Conf. (MMM 09), pp. 435-446, 2009
12. B. Schuller,"Recognizing Affect from Linguistic Information in 3D Continuous Space," IEEE Trans. Affective Computing, vol. 2, no. 4, pp. 192-205, Oct.-Dec. 2011
13. C. Wu and W. Liang,"Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels," IEEE Trans. Affective Computing, vol. 2, no. 1, pp. 10-21, Jan.-Mar. 2011
14. K. Forbes-Riley and D.J. Litman,"Benefits and Challenges of Real-Time Uncertainty Detection and Adaptation in a Spoken Dialogue Computer Tutor," Speech Comm., vol. 53, no. 9/10, pp. 1115-1136, 2011, DOI: 10.1016/j.specom.2011.02.006
15. O. AlZoubi et al.,"Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals," IEEE Trans. Affective Computing, vol. 3, no. 3, pp. 298-310, July-Sept. 2012
16. Z. Khalali and M. Moradi,"Emotion Recognition System Using Brain and Peripheral Signals: Using Correlation Dimension to Improve the Results of EEG," Proc. Intl Joint Conf. Neural Networks, pp. 1571-1575, 2009
17. G. Chanel et al.,"Emotion Assessment from Physiological Signals for Adaptation of Game Difficulty," IEEE Trans. Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 41, no. 6, pp. 1052-1063, Nov. 2011
18. M. Soleymani et al.,"Multi-Modal Emotion Recognition in Response to Videos," IEEE Trans. Affective Computing, vol. 3, no. 2, pp. 211-223, Apr.-June 2012
19. L. Kessous et al.,"Multimodal Emotion Recognition in Speech-Based Interaction Using Facial Expression, Body Gesture and Acoustic Analysis," J. Multimodal User Interfaces, vol. 3, no. 1, pp. 33-48, 2010
20. G. Castellano et al.,"Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech," Affect and Emotion in Human-Computer Interaction, C. Peter and R. Beale, eds., pp. 92-103, Springer, 2008
21. J. Wagner et al.,"Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data," IEEE Trans. Affective Computing, vol. 2, no. 4, pp. 206-218, Oct.-Dec. 2011
22. M. Nicolaou et al.,"Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence & Arousal Space," IEEE Trans. Affective Computing, vol. 2, no. 2, pp. 92-105, Apr.-June 2011
23. S. DMello and A. Graesser,"Multimodal Semi-Automated Affect Detection from Conversational Cues, Gross Body Language, and Facial Features," User Modeling and User-Adapted Interaction, vol. 20, no. 2, pp. 147-187, 2010
24. N. Holmes and C. Spence,"Multisensory Integration: Space, Time and Superadditivity," Current Biology, vol. 15, no. 18, pp. R762-R764, 2005, DOI: 10.1016/j.cub.2005.08.058 (Pubitemid 41323732)
25. S. DMello, J. Kory, and L.-P. Morency,"Consistent but Modest: Comparing Multimodal and Unimodal Affect Detection Accuracies from 30 Studies," Proc. 14th ACM Intl Conf. Multimodal Interaction, pp. 31-38, 2012
26. J.A. Russell et al.,"Facial and Vocal Expressions of Emotion," Ann. Rev. of Psychology, vol. 54, pp. 329-349, 2003 (Pubitemid 37417497)
27. R.A. Adolphs,"Neural Systems for Recognizing Emotions," Current Opinion in Neurobiology, vol. 12, pp. 169-177, 2002
28. M. Mehu and K. Scherer,"A Psycho-Ethological Approach to Social Signal Processing," Cognitive Processing, vol. 13, no. 2, pp. 397-414, 2012
29. A. Heberlein and A. Atkinson,"Neuroscientific Evidence for Simulation and Shared Substrates in Emotion Recognition: Beyond Faces," Emotion Rev., vol. 1, no. 2, pp. 162-177, 2009
30. S.D. Preston et al.,"The Neural Substrates of Cognitive Empathy," Social Neuroscience, vol. 2, no. 3/4, pp. 254-275, 2007, DOI: 10.1080/ 17470910701376902 (Pubitemid 47190216)
31. J. Russell,"Core Affect and the Psychological Construction of Emotion," Psychological Rev., vol. 110, pp. 145-172, 2003 (Pubitemid 37170695)
32. L. Barrett,"Are Emotions Natural Kinds? Perspectives on Psychological Science, vol. 1, pp. 28-58, 2006
33. L. Camras and J. Shutter,"Emotional Facial Expressions in Infancy," Emotion Rev., vol. 2, no. 2, pp. 120-129, 2010
34. A. Graesser et al.,"Detection of Emotions During Learning with AutoTutor," Proc. 28th Ann. Conf. Cognitive Science Society, pp. 285-290, 2006
35. S. Afzal and P. Robinson,"Natural Affect Data: Collection and Annotation," New Perspectives on Affect and Learning Technologies, R. Calvo and S. DMello, eds., pp. 44-70, Springer, 2011
36. B. de Gelder and J. Van den Stock,"Real Faces, Real Emotions: Perceiving Facial Expressions in Naturalistic Contexts of Voices, Bodies and Scenes," Oxford Handbook of Face Perception, A.J. Calder et al., eds., pp. 535-550, Oxford Univ. Press, 2011
37. B. de Gelder et al.,"Beyond the Face: Exploring Rapid Influences of Context on Face Processing," Progress in Brain Research, vol. 155, pp. 37-48, 2005, DOI: 10.1016/s0079-6132(06)55003-4
38. E. Douglas-Cowie et al.,"Multimodal Databases of Everyday Emotion: Facing up to Complexity," Proc. Ninth European Conf. Speech Comm. and Technology (Interspeech 05), pp. 813-816, 2005 (Pubitemid 43908186)
39. A. Graesser et al.,"AutoTutor: An Intelligent Tutoring System with Mixed-Initiative Dialogue," IEEE Trans. Education, vol. 48, no. 4, pp. 612-618, Nov. 2005, DOI: 10.1109/TE.2005.856149 (Pubitemid 41723893)
40. S. DMello,"A Selective Meta-Analysis on the Relative Incidence of Discrete Affective States during Learning with Technology," J. Educational Psychology, vol. 105, pp. 1082-1099, 2013
41. S. DMello and A. Graesser,"Language and Discourse Are Powerful Signals of Student Emotions during Tutoring," IEEE Trans. Learning Technologies, vol. 5, no. 4, pp. 304-317, Jan. 2012
42. A. Graesser et al.,"Emotions and Learning with AutoTutor," Proc. 13th Intl Conf. Artificial Intelligence in Education, pp. 569-571, 2007
43. E. Rosenberg and P. Ekman,"Coherence between Expressive and Experiential Systems in Emotion," Cognition & Emotion, vol. 8, no. 3, pp. 201-229, 1994
44. S. Craig et al.,"Emote Aloud during Learning with AutoTutor: Applying the Facial Action Coding System to Cognitive-Affective States During Learning," Cognition & Emotion, vol. 22, no. 5, pp. 777-788, 2008 (Pubitemid 351983084)
45. S. DMello and A. Graesser,"The Half-Life of Cognitive-Affective States during Complex Learning," Cognition & Emotion, vol. 25, no. 7, pp. 1299-1308, 2011
46. S. Craig et al.,"Affect and Learning: An Exploratory Look into the Role of Affect in Learning," J. Educational Media, vol. 29, pp. 241-250, 2004, DOI: 10.1080/1358165042000283101 (Pubitemid 39473944)
47. A. Graesser and S. DMello,"Emotions during the Learning of Difficult Material," Psychology of Learning and Motivation, B. Ross, ed., vol. 57, pp. 183-226, Elsevier, 2012
48. J.C. Pinheiro and D.M. Bates, Mixed-Effects Models in S and SPLUS. Springer Verlag, 2000
49. D.M. Bates and M. Maechler,"lme4: Linear Mixed-Effects Models Using S4 Classes,"Book lme4: Linear Mixed-Effects Models Using S4 Classes, Series lme4: Linear Mixed-Effects Models Using S4 Classes, ed., 2010
50. B. McDaniel et al.,"Facial Features for Affective State Detection in Learning Environments," Proc. 29th Ann. Meeting of the Cognitive Science Society, pp. 467-472, 2007
51. D. Matsumoto et al.,"Cross-Cultural Judgments of Spontaneous Facial Expressions of Emotion," J. Nonverbal Behavior, vol. 33, no. 4, pp. 213-238, 2009, DOI: 10.1007/s10919-009-0071-4
52. H. Aviezer et al.,"Angry, Disgusted, or Afraid? Studies on the Malleability of Emotion Perception," Psychological Science, vol. 19, no. 7, pp. 724-732, 2008
53. G. Stemmler et al.,"Constraints for Emotion Specificity in Fear and Anger: The Context Counts," Psychophysiology, vol. 38, no. 2, pp. 275-291, 2001 (Pubitemid 32267770)
54. P.J. Naab and J.A. Russell,"Judgments of Emotion from Spontaneous Facial Expressions of New Guineans," Emotion, vol. 7, no. 4, pp. 736-744, 2007, DOI: 10.1037/1528-3542.7a.736
55. K.A. Lindquist et al.,"Language and the Perception of Emotion," Emotion, vol. 6, no. 1, pp. 125-138, 2006, DOI: 10.1037/1528-3542.6.1.125 (Pubitemid 43829902)
56. L.F. Barrett and E.A. Kensinger,"Context Is Routinely Encoded during Emotion Perception," Psychological Science, vol. 21, no. 4, pp. 595-599, 2010, DOI: 10.1177/0956797610363547
57. T. Masuda et al.,"Placing the Face in Context: Cultural Differences in the Perception of Facial Emotion," J. Personality and Social Psychology, vol. 94, no. 3, pp. 365-381, 2008, DOI: 10.1037/0022-3514.94.3.365
58. J.M. Carroll and J.A. Russell,"Do Facial Expressions Signal Specific Emotions? Judging Emotion from the Face in Context," J. Personality and Social Psychology, vol. 70, no. 2, pp. 205-218, 1996 (Pubitemid 126420669)
59. L.F. Barrett,"Variety Is the Spice of Life: A Psychological Construction Approach to Understanding Variability in Emotion," Cognition & Emotion, vol. 23, no. 7, pp. 1284-1306, 2009, DOI: 10.1080/ 02699930902985894
60. J.A. Coan,"Emergent Ghosts of the Emotion Machine," Emotion Rev., vol. 2, no. 3, pp. 274-285, 2010
61. J.A. Russell,"Forced-Choice Response Format in the Study of Facial Expression," Motivation and Emotion, vol. 17, no. 1, pp. 41-51, 1993
62. M.G. Frank and J. Stennett,"The Forced-Choice Paradigm and the Perception of Facial Expressions of Emotion," J. Personality and Social Psychology, vol. 80, no. 1, pp. 75-85, 2001 (Pubitemid 33594868)
63. P.A. Pour et al.,"The Impact of System Feedback on Learners Affective and Physiological States," Proc. 10th Intl Conf. Intelligent Tutoring Systems, pp. 264-273, 2010
64. J.L. Tracy and R.W. Robins,"The Automaticity of Emotion Recognition," Emotion, vol. 8, no. 1, pp. 81-95, 2008, DOI: 10.1037/1528-3542.8.1.81 (Pubitemid 351274179)
65. N. Ambady and R. Rosenthal,"Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: A Meta-Analysis," Psychological Bull., vol. 111, no. 2, p. 256, 1992
66. K. Kelly and J. Metcalfe,"Metacognition of Emotional Face Recognition," Emotion, vol. 11, no. 4, pp. 896-906, 2011.