Automatic detection of nonverbal behavior predicts learning in dyadic interactions

IEEE Transactions on Affective Computing

Volumn 5, Issue 2, 2014, Pages 112-125

  Won, Andrea Stevenson ^a   Bailenson, Jeremy N ^a   Janssen, Joris H ^b  

^a STANFORD UNIVERSITY   (United States)

^b Sense Observation Systems   (Netherlands)

Author keywords

collaborative learning; gesture recognition; machine learning; Natural data set

Indexed keywords

ARTIFICIAL INTELLIGENCE; GESTURE RECOGNITION; HARDWARE; LEARNING ALGORITHMS;

AUTOMATIC DETECTION; COLLABORATIVE LEARNING; DATA SET; DYADIC INTERACTION; LEARNING INTERACTIONS; LEARNING PERFORMANCE; NONVERBAL BEHAVIOR; PSYCHOLOGICAL STATE;

LEARNING SYSTEMS;

EID: 84905692452     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAFFC.2014.2329304     Document Type: Article

References (65)

1. A. M. Kessell and B. Tversky, "Gestures for thinking and explaining," in Proc. Cogn. Sci. Soc. Meetings, 2005, p. 2498.
2. W. Roth, "Gestures: Their role in teaching and learning," Edu. Res., vol. 71, no. 3, pp. 365-392, 2011.
3. S. Mota and R. W. Picard, "Automated posture analysis for detecting learner's interest level," in Proc. IEEE Conf. Comp. Vision Pattern Recog. Workshop, 2003, vol. 5, p. 49.
4. T. Dragon, I. Arroyo, B. Woolf, W. Burleson, R. el Kaliouby, and H. Eydgahi, "Viewing student affect and learning through classroom observation and physical sensors," in Intelligent Tutoring System. Berlin, Germany: Springer, 2008, pp. 29-39.
5. S. D'Mello, T. Jackson, S. Craig, B. Morgan, P. Chipman, H. White, N. Person, B. Kort, R. el Kaliouby, R. W. Picard, and A. Graesser, "AutoTutor detects and responds to learners affective and cognitive states," in Proc. Workshop Emotional. Cogn. Issues Intell. Tutoring Syst. Conjunction 9th Int. Conf. Intell. Tutoring Syst., 2008, pp. 31-43.
6. S. K. D'Mello and A. Graesser, "Automatic detection of learner's affect from gross body language," Appl. Artif. Intell., vol. 23, no. 2, pp. 123-150, 2009.
7. M. LaFrance and M. Broadbent, "Group rapport: Posture sharing as a nonverbal indicator," Group Organization Stud., vol. 1, pp. 328-333, 1976.
8. F. J. Bernieri, "Coordinated movement and rapport in teacher student interactions," J. Nonverbal Behav., vol. 12, pp 120-138, 1988.
9. J. Zhou, "The effects of reciprocal imitation on teacher-student relationships and student learning outcomes," Mind, Brain, Edu., vol. 6, no. 2, pp. 66-73, 2012.
10. T. O. Meservy, M. L. Jensen, J. Kruse, J. K. Burgoon, and F. Jay, "Detecting deception through automatic, unobtrusive analysis of nonverbal behavior," IEEE Intell. Syst., vol. 20, no. 5, pp. 36-42, Sep./Oct. 2008.
11. K. Karpouzis, G. Caridakis, L. Kessous, N. Amir, A. Raouzaiou, L. Malatesta, and S. Kollias, "Modeling naturalistic affective states via facial, vocal and bodily expressions recognition," in Proc. 8th Int. Conf. Artif. Intell. Human Comput., 2007, pp. 92-116.
12. M. S. Bartlett, G. C. Littlewort, M. G. Frank, C. Lainscsek, I. R. Fasel, and J. R. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, 2005.
13. M. E. Hoque, D. J. McDuff, and R. W. Picard, "Exploring temporal patterns in classifying frustrated and delighted smiles," IEEE Trans. Affect. Comput. , vol. 3, no. 3, pp. 323-334, Jul.-Sep. 2012.
14. M. E. Jabon, S. J. Ahn, and J. N. Bailenson, "Automatically analyzing facial-feature movements to identify human errors," IEEE Intell. Syst., vol. 26, no. 2, pp. 54-63, Mar./Apr. 2011.
15. J. H. Janssen, P. Tacken, J. J. G. DeVries, E. L. Van Den Broek, J. H. M. Westerink, P. Haselager, and W. A. IJsselstein, "Machines outperform laypersons in recognizing emotions elicited by autobiographical recollection," Human-Comput. Interact., vol. 28, no. 6, pp. 479-517, 2012.
16. A. Kleinsmith, and N. Bianchi-Berthouze, "Affective body expression perception and recognition: A survey," IEEE Trans. Affect. Comput., vol. 4, no. 1, pp. 1-31, Jan./Mar. 2013.
17. A. Vinciarelli, M. Pantic, H. Bourlard, and A. Pentland, "Social signal processing: State-of-the-art and future perspectives of an emerging domain," in Proc. 16th ACM Int. Conf. Multimedia, 2008, pp. 1061-1070.
18. A. Metallinou, A. Katsamanis, and S. Narayanan, "Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information," Image Vis. Comput., vol. 31, no. 2, pp. 137-152, 2012.
19. H. Aviezer, Y. Trope, and A. Todorov, "Body cues, not facial expressions, discriminate between intense positive and negative emotions," Science, vol. 338, no. 6111, pp. 1225-1229, 2012.
20. A. Jamalian and B. Tversky. "Gestures alter thinking about time," in Proc. 34th Annu. Conf. Cogn. Sci. Soc., 2012, pp. 551-557.
21. D. R. Carney, A. J. C. Cuddy, and A. J. Yap, "Power posing brief nonverbal displays affect neuroendocrine levels and risk tolerance," Psychol. Sci., vol. 21, no. 10, pp. 1363-1368, 2010.
22. A. Kapur, A. Kapur, N. Virji-Babul, G. Tzanetakis, and P. Driessen, "Gesture-based affective computing on motion capture data," in Proc. 1st Int. Conf. Affect. Comput. Intell. Interact., 2005, pp. 1-7. (Pubitemid 43763304)
23. G. Castellano, S. Villalba, and A. Camurri, "Recognising human emotions from body movement and gesture dynamics," in Proc. 2nd Int. Conf. Affect. Comput. Intell. Interact., 2007, pp. 71-82.
24. J. N. Bailenson, N. Yee, S. Brave, D. Merget, and D. Koslow, "Virtual interpersonal touch: Expressing and recognizing emotions through haptic devices," Human-Comput. Interact., vol. 22, no. 3, pp. 325-353, 2007. (Pubitemid 47543507)
25. Y. Gao, N. Bianchi-Berthouze, and H. Meng, "What does touch tell us about emotions in touchscreen-based gameplay?" ACM Trans. Comput.-Human Interact., vol. 19, no. 4, p. 31, 2012.
26. A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed, "Automatic recognition of non-acted affective postures," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 4, pp. 1027-1038, Jan. 2011.
27. P.Salvagnini,H.Salamin,M.Cristani,A.Vinciarelli,andV.Murino, "Learninghowtoteachfromvideolecture:Automaticpredictionof lecture ratings based on teacher's nonverbal behavior," in Proc. IEEE3rdInt.Conf.Cogn. Infocommun.,2012,pp.415-419.
28. N. Ambady, F. J. Bernieri, and J. A. Richeson,"Towards a histology of social behavior: Judgmental accuracy from thin slices of behavior," Adv. Exp. Soc. Psychol., vol. 32, pp. 201-272, 2000.
29. N. Ambady, D. Laplante, T. Nguyen, R. Rosenthal, N. Chaumeton, and W. Levinson, "Surgeons' tone of voice: A clue to malpractice history," Surgery, vol. 132, pp. 5-9, 2002. (Pubitemid 34787813)
30. E. Babad, F. Bernieri, and R. Rosenthal, "Nonverbal communication and leakage in the behavior of biased and unbiased teachers," J. Personality Soc. Psychol., vol. 56, pp. 89-94, 1989.
31. F. Ramseyer and W. Tschacher, "Nonverbal synchrony in psychotherapy: Coordinated body-movement reflects relationship quality and outcome," J. Consulting Clin. Psychol., vol. 79, no. 3, pp. 284-295, 2011.
32. G. Johansson, "Visual perception of biological motion and a model for its analysis' attention," Perception Psychophys., vol. 14, no. 2, pp. 201-211, 1973.
33. G. Mather and L. Murdoch, "Gender discrimination in biological motion displays based on dynamic cues," Proc.: Biol. Sci., vol. 258, no. 1353, pp. 273-279, 1994.
34. K. L. Johnson, S. Gill, V. Reichman, and L. G. Tassinary, "Swagger, sway, and sexuality: Judging sexual orientation from body motion and morphology," J. Personality Soc. Psychol., vol. 93, pp. 321-334, 2007.
35. J. Michalak, N. F. Troje, and T. Heidenreich, "The effects of mindfulness- based cognitive therapy on depressive gait patterns," J. Cogn. Behav. Psychotherapies, vol. 11, pp. 13-27, 2011.
36. A. P. Atkinson, W. H. Dittrich, A. J. Gemmell, and A. W. Young, "Emotion perception from dynamic and static body expressions in point-light and full-light displays," Perception, vol. 33, no. 6, pp. 717-746, 2004. (Pubitemid 39130224)
37. R. Poppe, "Vision-based human motion analysis: An overview," Comput. Vis. Image Understanding, vol. 108, pp. 4-18, 2007. (Pubitemid 47333391)
38. C. C. Martin, D. C. Burkert, K. R. Choi, N. B. Wieczorek, P. M. McGregor, R. A. Herrmann, and P. A. Beling, "A real-time ergonomic monitoring system using the Microsoft Kinect," in Proc. IEEE Syst. Inf. Des. Symp., 2012, pp. 50-55.
39. J. Sung, C. Ponce, B. Selman, and A. Saxena, "Human activity detection from RGBD images," Plan, Activity, and Intent Recognition p. 64, 2011.
40. B. Barron, "When smart groups fail," J. Learn. Sci., vol. 12, no. 3, pp. 307-359, 2003. (Pubitemid 36853158)
41. Microsoft Corp. Redmond WA, USA. Kinect for Xbox 360, 2011.
42. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques. San Mateo, CA, USA: Morgan Kaufmann, 2005.
43. T. Dutta, "Evaluation of the KinectTM sensor for 3-D kinematic measurement in the workplace," Appl. Ergonomics, vol. 43, no. 4, pp. 645-649, 2012.
44. E. Delaherche, M. Chetouani, A. Mahdhaoui, C. Saint-Georges, S. Viaux, and D. Cohen, "Interpersonal synchrony: A survey of evaluation methods across disciplines," IEEE Trans. Affect. Comput. , vol. 3, no. 3, pp. 349-365, Jul.-Sep. 2012.
45. S. S. Wiltermuth and C. Heath, "Synchrony and cooperation," Psychol. Sci., vol. 20, no. 1, pp. 1-5, 2009.
46. L. K. Miles, L. K. Nind, Z. Henderson, and C. N. Macrae, "Moving memories: Behavioral synchrony and memory for self and others," J. Exp. Soc. Psychol., vol. 46, no. 2, pp. 457-460, 2010.
47. C. N. Macrae, C. Neil, O. K. Duffy, L. K. Miles, and J. Lawrence, "A case of hand waving: Action synchrony and person perception," Cognition, vol. 109, no. 1, pp. 152-156, 2008.
48. T. L. Chartrand and J. L. Lakin, "The antecedents and consequences of human behavioral mimicry," Annu. Rev. Psychol., vol. 64, pp. 285-308, 2013.
49. A. Kleinsmith, T. Fushimi, N. Bianchi-Berthouze, "An incremental and interactive affective posture recognition system," in Proc. Int. Workshop Adapt. Interact. Style Affect. Factors, Conjunction Int. Conf. User Model., 2005, pp. 1-13.
50. S. D'Mello and A. Graesser, "Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features," User Model. User-Adapt. Interact., vol. 20, no. 2, pp. 147-187, 2010.
51. A. Kleinsmith, P. Ravindra De Silva, and N. Bianchi-Berthouze, "Cross-cultural differences in recognizing affect from body posture," Interact. Comput., vol. 18, no. 6, pp 1371-1389, 2006. (Pubitemid 44709886)
52. J. Gratch, N. Wang, J. Gerten, E. Fast, and R. Duffy, "Creating rapport with virtual agents," in Intelligent Virtual Agents. Berlin, Germany: Springer, 2007, pp. 125-138. (Pubitemid 350271480)
53. R. El. Kaliouby, A. Teeters, and R. W. Picard, "An exploratory social-emotional prosthetic for autism spectrum disorders," in Proc. Wearable Implantable Body Sens. Netw., Int. Workshop, 2006, pp. 3-4. (Pubitemid 44711266)
54. P. M. Niedenthal, L. W. Barsalou, P. Winkielman, S. Krauth- Gruber, and F. Ric, "Embodiment in attitudes, social perception, and emotion," Pers. Soc. Psychol. Rev., vol. 9, no. 3, pp. 184-211, 2005. (Pubitemid 41356682)
55. S. Goldin-Meadow and M. A. Singer. "From children's hands to adults' ears: Gesture's role in the learning process," Develop. Psychol., vol. 39, no. 3, pp. 509-519, 2003.
56. S. Goldin-Meadow and S. M. Wagner, "How our hands help us learn," Trends Cogn. Sci., vol. 9, no. 5, pp. 234-241, 2005. (Pubitemid 40615111)
57. M. S. H. Aung, B. Romera-Paredes, A. Singh, S. Lim, N. Kanakam, A. C. de C. Williams, and N. Bianchi-Berthouze, "Getting rid of pain-related behavior to improve social and self-perception: A technology-based perspective," in Proc. 14th Int. Workshop Image Anal. Multimedia Interact. Serv., 2013, pp. 1-4.
58. N. Bianchi-Berthouze, "Understanding the role of body movement in player engagement," Human-Comput. Interact., vol. 28, no. 1, pp. 40-75, 2013.
59. N. Savva, A. Scarinzi, and N. Bianchi-Berthouze, "Continuous recognition of player's affective body expression as dynamic quality of aesthetic experience," IEEE Trans. Comput. Intell. Artif. Intell. Games, vol. 4, no. 3, pp. 199-212, Sep. 2012.
60. N. Bianchi-Berthouze, P. Cairns, A. Cox, C. Jennett, and W. W. Kim, "On posture as a modality for expressing and recognizing emotions," in Proc. Emotion HCI, 2008, pp. 74-80.
61. K. Isbister, M. Karlesky, and J. Frye, "Scoop! Using movement to reduce math anxiety and affect confidence," in Proc. Int. Conf. Found. Dig. Games, 2012, pp. 228-230.
62. K. Isbister. "How to stop being a buzzkill: Designing yamove!, a mobile tech mash-up to truly augment social play," in Proc. 14th Int. Conf. Human-comput. Interact. Mobile Devices Serv. Companion, Sep. 2012, pp. 1-4.
63. S. E. Lindley, J. Le. Couteur, and N. L. Berthouze, "Stirring up experience through movement in game play: Effects on engagement and social behaviour," in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2008, pp. 511-514.
64. R. W. Picard, "Affective computing: Challenges," Int. J. Human- Comput. Stud., vol. 59, no. 1, pp. 55-64, 2003.
65. R. A. Calvo and S. K. D'Mello, New Perspectives on Affect and Learning Technologies, vol. 3. New York, NY, USA: Springer, 2011.