The additive value of multimodal features for predicting engagement, frustration, and learning during tutoring

ICMI 2014 - Proceedings of the 2014 International Conference on Multimodal Interaction

Volumn , Issue , 2014, Pages 42-49

  Grafsgaard, Joseph F ^a   Wiggins, Joseph B ^a   Vail, Alexandria K ^a   Boyer, Kristy Elizabeth ^a   Wiebe, Eric N ^b   Lester, James C ^a  

^a North Carolina State University   (United States)

^b NORTH CAROLINA STATE UNIVERSITY   (United States)

Author keywords

Affect; Engagement; Facial expression; Frustration; Gesture; Multimodal; Posture; Tutorial dialogue

Indexed keywords

COMPUTER PROGRAMMING; INTERACTIVE COMPUTER SYSTEMS;

AFFECT; ENGAGEMENT; FACIAL EXPRESSIONS; FRUSTRATION; GESTURE; MULTI-MODAL; POSTURE; TUTORIAL DIALOGUE;

STUDENTS;

EID: 84944313264     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2663204.2663264     Document Type: Conference Paper

References (25)

1. Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K. and Christopherson, R.M. 2009. Emotion Sensors Go To School. 14th International Conference on Artificial Intelligence in Education, 17-24.
2. Cooper, D.G., Muldner, K., Arroyo, I., Woolf, B.P. and Burleson, W. 2010. Ranking Feature Sets for Emotion Models used in Classroom Based Intelligent Tutoring Systems. Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization, 135-146.
3. D'Mello, S.K. and Graesser, A.C. 2010. Multimodal Semi-automated Affect Detection From Conversational Cues, Gross Body Language, and Facial Features. User Modeling and User-Adapted Interaction. 20, 2, 147-187.
4. D'Mello, S.K. and Kory, J. 2012. Consistent but Modest: A Meta-Analysis on Unimodal and Multimodal Affect Detection Accuracies from 30 Studies. Proceedings of the 14th ACM International Conference on Multimodal Interaction, 31-38.
5. D'Mello, S.K., Lehman, B., Pekrun, R. and Graesser, A.C. 2014. Confusion Can Be Beneficial for Learning. Learning & Instruction. 29, 153-170.
6. Ekman, P., Friesen, W. V. and Hager, J.C. 2002. Facial Action Coding System. A Human Face.
7. Grafsgaard, J.F., Fulton, R.M., Boyer, K.E., Wiebe, E.N. and Lester, J.C. 2012. Multimodal Analysis of the Implicit Affective Channel in Computer-Mediated Textual Communication. Proceedings of the 14th ACM International Conference on Multimodal Interaction, 145-152.
8. Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N. and Lester, J.C. 2013. Automatically Recognizing Facial Expression: Predicting Engagement and Frustration. Proceedings of the 6th International Conference on Educational Data Mining, 43-50.
9. Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N. and Lester, J.C. 2013. Automatically Recognizing Facial Indicators of Frustration: A Learning-Centric Analysis. Proceedings of the 5th International Conference on Affective Computing and Intelligent Interaction, 159-165.
10. Harrigan, J.A. and O'Connell, D.M. 1996. How Do You Look When Feeling Anxious? Facial Displays of Anxiety. Personality and Individual Differences. 21, 2, 205-212.
11. Hart, S.G. and Staveland, L.E. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Human Mental Workload. P.A. Hancock and N. Meshkati, eds. Elsevier Science. 139-183.
12. Joshi, J., Goecke, R., Parker, G. and Breakspear, M. 2013. Can Body Expressions Contribute to Automatic Depression Analysis? Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1-7.
13. Kapoor, A. and Picard, R.W. 2005. Multimodal Affect Recognition in Learning Environments. Proceedings of the 13th Annual ACM International Conference on Multimedia, 677-682.
14. Littlewort, G., Bartlett, M.S., Salamanca, L.P. and Reilly, J. 2011. Automated Measurement of Children's Facial Expressions during Problem Solving Tasks. Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, 30-35.
15. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J.R. and Bartlett, M.S. 2011. The Computer Expression Recognition Toolbox (CERT). Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, 298-305.
16. Marx, J.D. and Cummings, K. 2007. Normalized Change. American Journal of Physics. 75, 1, 87-91.
17. O'Brien, H.L. and Toms, E.G. 2010. The Development and Evaluation of a Survey to Measure User Engagement. Journal of the American Society for Information Science and Technology. 61, 1, 50-69.
18. Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A. and Morency, L.-P. 2013. Automatic Behavior Descriptors for Psychological Disorder Analysis. Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1-8.
19. Symonds, M.R.E. and Moussalli, A. 2010. A Brief Guide to Model Selection, Multimodel Inference and Model Averaging in Behavioural Ecology using Akaike's Information Criterion. Behavioral Ecology and Sociobiology. 65, 1, 13-21.
20. Vail, A.K. and Boyer, K.E. 2014. Adapting to Personality Over Time: Examining the Effectiveness of Dialogue Policy Progressions in Task-Oriented Interaction. Proceedings of the 15th Annual SIGDIAL Meeting on Discourse and Dialogue, 41-50.
21. Vail, A.K. and Boyer, K.E. 2014. Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes. Proceedings of the 12th International Conference on Intelligent Tutoring Systems, 199-209.
22. VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A. and Rosé, C.P. 2007. When Are Tutorial Dialogues More Effective Than Reading? Cognitive Science. 31, 1, 3-62.
23. Wiebe, E.N., Lamb, A., Hardy, M. and Sharek, D. 2014. Measuring Engagement in Video Game-based Environments: Investigation of the User Engagement Scale. Computers in Human Behavior. 32, 123-132.
24. Woolf, B.P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D.G. and Picard, R.W. 2009. Affect-Aware Tutors: Recognising and Responding to Student Affect. International Journal of Learning Technology. 4, 3-4, 129-164.
25. Wu, T., Butko, N.J., Ruvolo, P., Whitehill, J., Bartlett, M.S. and Movellan, J.R. 2012. Multi-Layer Architectures for Facial Action Unit Recognition. IEEE Transactions on Systems, Man and Cybernetics, Part B: Cybernetics. 42, 4, 1027-1038.