Recognizing student emotions and attitudes on the basis of utterances in spoken tutoring dialogues with both human and computer tutors

Speech Communication

Volumn 48, Issue 5, 2006, Pages 559-590

  Litman, Diane J ^a   Forbes Riley, Kate ^a  

^a UNIVERSITY OF PITTSBURGH   (United States)

Author keywords

Emotional speech; Empirical study relevant to adaptive spoken dialogue systems; Predicting user state via machine learning; Prosody; Tutorial dialogue systems

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN COMPUTER INTERACTION; INFORMATION RETRIEVAL SYSTEMS; LEARNING SYSTEMS; SPEECH INTELLIGIBILITY; STUDENTS;

EMOTIONAL SPEECH; EMPIRICAL STUDY RELEVANT TO ADAPTIVE SPOKEN DIALOGUE SYSTEMS; PREDICTING USER STATE VIA MACHINE LEARNING; PROSODY; TUTORIAL DIALOGUE SYSTEMS;

SPEECH RECOGNITION;

EID: 33645971628     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2005.09.008     Document Type: Article

References (85)

1. Aist, G., Kort, B., Reilly, R., Mostow, J., Picard, R., 2002. Adding human-provided emotional scaffolding to an automated reading tutor that listens increases student persistence. In: Proc. Intelligent Tutoring Systems (ITS), p. 992.
2. Aleven, V., Rose, C.P. (Eds.), July 2003. Proc. AIED 2003 Workshop on Tutorial Dialogue Systems: With a View toward the Classroom, Sydney, Australia.
3. Aleven, V., Popescu, O., Koedinger, K., 2001. Towards tutorial dialog to support self-explanation: adding natural language understanding to a cognitive tutor. In: Moore, J.D., Redfield, C.L., Johnson, W.L. (Eds.), Proc. Artificial Intelligence in Education, pp. 246-255.
4. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., Stolcke, A., 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In: Proc. International Conf. on Spoken Language Processing (ICSLP), pp. 203-207.
5. Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E., 2000. Desperately seeking emotions: Actors, wizards, and human beings. In: ISCA Workshop on Speech and Emotion, pp. 195-200.
6. Batliner A., Fischer K., Huber R., Spilker J., and Noth E. How to find trouble in communication. Speech Commun. 40 (2003) 117-143
7. Bhatt, K., Evens, M., Argamon, S., 2004. Hedged responses and expressions of affect in human/human and human/computer tutorial interactions. In: Proc. Cognitive Science.
8. Biddle, E.S., Malone, L., McBride, D., July 2003. Objective measurement of student affect to optimize automated instruction. In: Conati, C., Hudlicka, E., Lisetti, C. (Eds.), Proc. 3rd User Modeling Workshop on Assessing and Adapting to User Attitudes and Effect: Why, When, and How? Johnstown, PA, pp. 65-68.
9. Black, A., Taylor, P., 1997. Festival speech synthesis system: system documentation (1.1.1). The Centre for Speech Technology Research, University of Edinburgh. Available from: .
10. Carletta J. Assessing agreement on classification tasks: the kappa statistic. Comput. Linguistics 22 2 (1996)
11. Cavalluzzi, A., Carolis, B.D., Carofiglio, V., Grassano, G., 2003. Emotional dialogs with an embodied agent. In: Proc. User Modeling Conference. Johnstown, PA, pp. 86-95.
12. Chi M., Leeuw N.D., Chiu M.-H., and Lavancher C. Eliciting self-explanations improves understanding. Cognitive Sci. 18 (1994) 439-477
13. Cohen J. A coefficient of agreement for nominal scales. Educat. Psychol. Measurement 20 (1960) 37-46
14. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol. Bull. 70 (1968) 213-220
15. Coles, G., 1999. Literacy, emotions, and the brain. Reading Online, March 1999, .
16. Conati, C., Chabbal, R., Maclaren, H., July 2003a. A study on using biometric sensors for monitoring user emotions in educational games. In: Conati, C., Hudlicka, E., Lisetti, C. (Eds.), Proc. 3rd User Modeling Workshop on Assessing and Adapting to User Attitudes and Effect: Why, When, and How? Johnstown, PA, pp. 16-22.
17. Conati, C., Hudlicka, E., Lisetti, C. (Eds.), July 2003b. Proc. 3rd User Modeling Workshop on Assessing and Adapting to User Attitudes and Effect: Why, When, and How? Johnstown, PA.
18. Cowie R., and Cornelius R.R. Describing the emotional states that are expressed in speech. Speech Commun. 40 (2003) 5-32
19. Cowie R., Douglas-Cowie E., Tsapatsoulis N., Votsis G., Kollias S., Fellenz W., and Taylor J. Emotion recognition in human-computer interaction. IEEE Signal Process. Mag. 18 January (2001) 32-80
20. Craig, S.D., Graesser, A., 2003. Why am I confused: an exploratory look into the role of affect in learning. In: Mendez-Vilas, A., Gonzalez, J. (Eds.), Advances in Technology-based Education: Towards a Knowledge-based Society, Vol. 3, pp. 1903-1906.
21. de Rosis, F. (Ed.), June 1999. Proc. User Modeling Workshop on Attitude, Personality and Emotions in User-Adapted Interaction, Alberta, Canada.
22. de Rosis, F. (Ed.), July 2001a. Proc. User Modeling Workshop on Attitude, Personality and Emotions in User-Adapted Interaction, Sonthofen, Germany.
23. de Rosis, F. (Ed.), 2001b. Special Issue on User Modeling and Adaptation in Affective Computing, Vol. 11(4).
24. de Rosis, F. (Ed.), 2002. Special Issue on User Modeling and Adaptation in Affective Computing, Vol. 12(1).
25. Devillers, L., Lamel, L., Vasilescu, I., 2003. Emotion detection in task-oriented spoken dialogs. In: Proc. IEEE Internat. Conf. on Multimedia & Expo (ICME).
26. DiEugenio B., and Glass M. The kappa statistic: a second look. Comput. Linguistics 30 1 (2004)
27. DiEugenio, B., Moore, J.D., Paolucci, M., 1997. Learning features that predict cue usage. In: Proc. 35th Annual Meeting of the Association for Computational Linguistics (ACL97), Madrid, Spain.
28. Evens, M., Brandle, S., Chang, R.-C., Freedman, R., Glass, M., Lee, Y.H., Shim, L.S., Woo, C.W., Zhang, Y., Zhou, Y., Michaeland, J.A., Rovick, A.A., 2001. Circsim-tutor: an intelligent tutoring system using natural language dialogue. In: Proc. Twelfth Midwest AI and Cognitive Science Conference, MAICS 2001, Oxford, OH, pp. 16-23.
29. Fan, C., Johnson, M., Messom, C., Sarrafzadeh, A., 2003. Machine vision for an intelligent tutor. In: Proc. 2nd International Conference on Computational Intelligence, Robotics and Autonomous Systems (CIRAS), Singapore.
30. Fischer, K., December 1999. Annotating emotional language data. Verbmobil Report 236.
31. Forbes-Riley, K., Litman, D., 2004. Predicting emotion in spoken dialogue from multiple knowledge sources. In: Proc. Human Language Technology Conf.: 4th Meeting of the North American Chap. of the Assoc. for Computational Linguistics (HLT/NAACL), pp. 201-208.
32. Forbes-Riley, K., Litman, D., Huettner, A., Ward, A., 2005. Dialogue-learning correlations in spoken dialogue tutoring. In: Proc. Internat. Conf. on Artificial Intelligence in Education.
33. Fox B.A. The Human Tutorial Dialogue Project (1993), Lawrence Erlbaum
34. Freund, Y., Schapire, R., 1996. Experiments with a new boosting algorithm. In: Proc. Internat. Conf. on Machine Learning, pp. 148-156.
35. Graesser A., Person N., and Harter D. Teaching tactics and dialog in AutoTutor. Internat. J. Artificial Intell. Educat. 12 3 (2001) 257-279
36. Graesser A., VanLehn K., Rose C., Jordan P., and Harter D. Intelligent tutoring systems with conversational dialogue. AI Mag. 22 4 (2001) 39-51
37. Hausmann R., and Chi M. Can a computer interface support self-explaining?. Internat. J. Cognitive Technol. 7 1 (2002) 4-14
38. Huang X.D., Alleva F., Hon H.W., Hwang M.Y., Lee K.F., and Rosenfeld R. The SphinxII speech recognition system: an overview. Computer, Speech and Language 2 (1993) 137-148
39. Izard C.E. Emotion-cognition relationships and human development. In: Izard C., Kagan J., and Zajonc R. (Eds). Emotions, Cognition, and Behavior (1984), Cambridge University Press, New York 17-37
40. Jordan, P., VanLehn, K., 2002. Discourse processing for explanatory essays in tutorial applications. In: Proc. 3rd SIGdial Workshop on Discourse and Dialogue, pp. 74-83.
41. Jordan P., Makatchev M., and VanLehn K. Abductive theorem proving for analyzing student explanations. In: Hoppe U., Verdejo F., and Kay J. (Eds). Proc. Artificial Intelligence in Education (2003), IOS Press 73-80
42. Jordan, P.W., Makatchev, M., VanLehn, K., 2004. Combining competing language understanding approaches in an intelligent tutoring system. In: Proc. Intelligent Tutoring Systems Conference (ITS), pp. 346-357.
43. Kort, B., Reilly, R., Picard, R.W., 2001. An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion. In: Internat. Conf. on Advanced Learning Technologies (ICALT), pp. 43-48.
44. Krippendorf K. Content Analysis: An Introduction to its Methodology (1980), Sage Publications, Beverly Hills, CA
45. Landis J.R., and Koch G.G. The measurement of observer agreement for categorical data. Biometrics 33 (1977) 159-174
46. Lee, C., Narayanan, S., Pieraccini, R., 2001. Recognition of negative emotions from the speech signal. In: Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).
47. Lee, C., Narayanan, S., Pieraccini, R., 2002. Combining acoustic and language information for emotion recognition. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP).
48. Liscombe, J., Venditti, J., J.Hirschberg, 2003. Classifying subject ratings of emotional speech using acoustic features. In: Proc. EuroSpeech.
49. Litman D. Cue phrase classification using machine learning. J. Artificial Intell. Res. 5 (1996) 53-94
50. Litman, D., Forbes, K., 2003. Recognizing emotion from student speech in tutoring dialogues. In: Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 25-30.
51. Litman, D., Forbes-Riley, K., 2004a. Annotating student emotional states in spoken tutoring dialogues. In: Proc. 5th SIGdial Workshop on Discourse and Dialogue, pp. 144-153.
52. Litman, D.J., Forbes-Riley, K., 2004b. Predicting student emotions in computer-human tutoring dialogues. In: Proc. Association Computational Linguistics (ACL), pp. 352-359.
53. Litman, D., Silliman, S., 2004. ITSPOKE: An intelligent tutoring spoken dialogue system. In: Proc. Human Language Technology Conf.: 4th Meeting of the North American Chap. of the Assoc. for Computational Linguistics (HLT/NAACL) (Companion Volume), pp. 233-236.
54. Litman, D., Hirschberg, J., Swerts, M., 2001. Predicting user reactions to system error. In: Proc. Association of Computational Linguistics (ACL), pp. 362-369.
55. Litman, D., Rosé, C.P., Forbes-Riley, K., VanLehn, K., Bhembe, D., Silliman, S., 2004. Spoken versus typed human and computer dialogue tutoring. In: Proc. Internat. Conf. on Intelligent Tutoring Systems (ITS), pp. 368-379.
56. Maeireizo, B., Litman, D., Hwa, R., 2004. Co-training for predicting emotions with spoken dialogue data. In: Companion Proc. Association for Computational Linguistics (ACL), pp. 203-206.
57. Masters J.C., Barden R.C., and Ford M.E. Affective states, expressive behavior, and learning in children. J. Personality Soc. Psychol. 37 (1979) 380-390
58. Moreno R., Mayer R.E., Spires H.A., and Lester J.C. The case for social agency in computer-based teaching: do students learn more deeply when they interact with animated pedagogical agents. Cognit. Instruct. 19 2 (2001) 177-213
59. Mostow J., and Aist G. Evaluating tutors that listen: an overview of Project LISTEN. In: Forbus K., and Feltovich P. (Eds). Smart Machines in Education (2001), MIT/AAAI Press 169-234
60. Mozziconacci S.J.L. Modeling emotion and attitude in speech by means of perceptually based parameter values. User Model. User-Adapted Interact. 11 4 (2001) 297-326
61. Narayanan, S., 2002. Towards modeling user behavior in human-machine interaction: effect of errors and emotions. In: Proc. ISLE Workshop on Dialogue Tagging for Multi-modal Human Computer Interaction.
62. Nasby W., and Yando R. Selective encoding and retrieval of affectively valent information. J. Personality Soc. Psychol. 43 (1982) 1244-1255
63. Oudeyer P.-Y. The production and recognition of emotions in speech: features and algorithms. Internat. J. Human Comput. Studies 59 1-2 (2002) 157-183
64. Pantic, M., Rothkrantz, L.J.M., 2003. Toward an affect-sensitive multimodal human-computer interaction. In: Proc. IEEE Conf., Vol. 91(9), pp. 1370-1390.
65. Picard R.W., Vyzas E., and Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23 10 (2001)
66. Polzin, T.S., Waibel, A.H., 1998. Detecting emotions in speech. In: Proc. Cooperative Multimodal Communication.
67. Potts R., Morse M., Felleman E., and Masters J.C. Children's emotions and memory for affective narrative content. Motivat. Emot. 10 (1986) 39-57
68. Rickel J., and Johnson W.L. Task-oriented collaboration with embodied agents in virtual worlds. In: Cassell J., Sullivan J., Prevost S., and Churchill E. (Eds). Embodied Conversational Agents (2000), MIT Press 95-122
69. Rosé, C.P., 2000. A framework for robust sentence level interpretation. In: Proc. First Meeting of the North American Chapter of the Association for Computational Lingusitics, pp. 1129-1135.
70. Rosé, C.P., Jordan, P., Ringenberg, M., Siler, S., VanLehn, K., Weinstein, A., 2001. Interactive conceptual tutoring in Atlas-Andes. In: Proc. Artificial Intelligence in Education, pp. 256-266.
71. Rose, C., 2005. Personal communication.
72. Rose, C.P., Aleven, V. (Eds.), June 2002. Proc. ITS 2002 Workshop on Empirical Methods for Tutorial Dialogue Systems, San Sebastian, Spain.
73. Rose, C.P., Freedman, R. (Eds.), 2000. AAAI Working Notes of the Fall Symposium: Building Dialogue Systems for Tutorial Applications.
74. Russell J.A., Bachorowski J., and Fernandez-Dols J. Facial and vocal expressions of emotion. Ann. Rev. Psychol. 54 (2003) 29-349
75. Scherer K.R. Vocal communication of emotion: a review of research paradigms. Speech Communicat. 40 (2003) 227-256
76. Schultz, K., Bratt, E.O., Clark, B., Peters, S., Pon-Barry, H., Treeratpituk, P., 2003. A scalable, reusable spoken conversational tutor: SCoT. In: AIED Supplementary Proceedings, pp. 367-377.
77. Seipp B. Anxiety and academic performance: a meta-analysis of findings. Anxiety Res. 4 (1991) 27-41
78. Shafran, I., Riley, M., Mohri, M., 2003. Voice signatures. In: Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 31-36.
79. Shah F., Evens M., Michael J., and Rovick A. Classifying student initiatives and tutor responses in human-human keyboard-to-keyboard tutoring sessions. Discourse Process. 33 1 (2002)
80. Siegel S., Castellan N., and John J. Nonparametric Statistics for the Behavioral Sciences. second ed. (1988), McGraw-Hill, New York
81. Siegle, G.J., 1994. The balanced affective word list project. Available from: .
82. ten Bosch L. Emotions, speech and the ASR framework. Speech Commun. 40 (2003) 213-225
83. VanLehn, K., Jordan, P.W., Rosé, C., Bhembe, D., Böttner, M., Gaydos, A., Makatchev, M., Pappuswamy, U., Ringenberg, M., Roque, A., Siler, S., Srivastava, R., Wilson, R., 2002. The architecture of Why2-Atlas: a coach for qualitative physics essay writing. In: Proc. 6th Internat. Intell. Tutoring Syst. Conf., pp. 158-167.
84. Witten I.H., and Frank E. Data Mining: Practical Machine Learning Tools and Techniques with Java implementations (1999), Morgan Kaufmann, San Francisco. Available from:
85. Zinn, C., Moore, J.D., Core, M.G., June 2002. A 3-tier planning architecture for managing tutorial dialogue. In: Proc. Intell. Tutoring Syst. Conf. (ITS 2002), Biarritz, France, pp. 574-584.