Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system

Computer Speech and Language

Volumn 25, Issue 1, 2011, Pages 105-126

  Forbes Riley, Kate ^a   Litman, Diane ^b  

^a UNIVERSITY OF PITTSBURGH   (United States)

^b University of Pittsburgh   (United States)

Author keywords

Affective states in spontaneous data; Automatic affect adaptation; Collection and annotation of realistic, representative, and publicly; Spoken dialogue tutoring system; Wizard of Oz experimental design

Indexed keywords

AFFECTIVE STATE; EXPERIMENTAL DESIGN; SPOKEN DIALOGUE; TUTORING SYSTEM; WIZARD OF OZ;

ADAPTIVE SYSTEMS; COMPUTER CRIME; DESIGN; DYNAMIC RESPONSE; LINGUISTICS; SPEECH PROCESSING; SPEECH RECOGNITION; STATISTICS; STUDENTS;

COMPUTER CONTROL SYSTEMS;

EID: 77955427299     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2009.12.002     Document Type: Article

References (72)

1. Ai, H., Litman, D., Forbes-Riley, K., Rotaru, M., Tetreault, J., Purandare, A., 2006. Using system and user performance features to improve emotion detection in spoken tutoring dialogs. In: Proceedings of Interspeech, Pittsburgh, PA, pp. 797-800.
2. Aist, G. Kort, B., Reilly, R., Mostow, J., Picard R., 2002. Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: adding Human-Provided Emotional Scaffolding to an Automated Reading Tutor that Listens. In: Proceedings of Intelligent Tutoring Systems Conference Workshop on Empirical Methods for Tutorial Dialogue Systems.
3. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., Stolcke, A., 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In: Hansen, J.H.L., Pellom, B. (Eds.), Proceedings of the International Conference on Spoken Language Processing (ICSLP), Denver, USA, pp. 2037-2039.
4. A. Batliner, K. Fischer, R. Huber, J. Spilker, and E. Noth How to find trouble in communication Speech Communication 40 1-2 2003 117 143
5. A. Batliner, C. Hacker, S. Steidl, E. Noth, and J. Haas From emotion to interaction: lessons from real human-machine dialogues E. Andre, L. Dybkjær, W. Minker, P. Heisterkamp, Affective Dialogue Systems Proceedings of a Tutorial and Research Workshop, Lecture Notes in Artificial Intelligence vol. 3068 2004 Springer-Verlag Berlin 1 12
6. A. Batliner, S. Steidl, C. Hacker, and E. Noth Private emotions vs. social interaction-a data-driven approach towards analysing emotion in speech User Modeling and User-Adapted Interaction: The Journal of Personalization Research 18 2008 175 206
7. Baylor, A.L., Ryu, J., Shen, E., 2003. The effect of pedagogical agent voice and animation on learning, motivation, and perceived persona. In: Proceedings of ED-MEDIA, Honolulu, Hawaii (June).
8. Bhatt, K., Evens, M., Argamon, S., 2004. Hedged responses and expressions of affect in human/human and human/computer tutorial interactions. In: Proceedings of Cognitive Science (CogSci), Chicago, USA, pp. 114-119.
9. Bonneau-Maynard, H. Devillers, L., Rosset, S., 2000. Predictive performance of dialog systems. In: Proceedings of Language Resources and Evaluation Conference (LREC), Athens, Greece.
10. Chu-Carroll, J., Nickerson, J.S., 2000. Evaluating automatic dialogue strategy adaptation for a spoken dialogue system. In: Proceedings of the 1st Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 202-209.
11. Conati, C., Maclaren, H., 2004. Evaluating a probabilistic model of student affect. In: Proceedings of Intelligent Tutoring Systems Conference(ITS), Maceio, Brazil, pp. 55-66.
12. R. Cowie, and R.R. Cornelius Describing the emotional states that are expressed in speech Speech Communication 40 1-2 2003 5 32
13. S. Craig, A. Graesser, J. Sullins, and B. Gholson Affect and learning: an exploratory look into the role of affect in learning with AutoTutor Journal of Educational Media 29 3 2004 241 250
14. de Vicente, A., Pain, H., 2002. Informing the detection of the students' motivational state: an empirical study. In: Proceedings of the Intelligent Tutoring Systems Conference (ITS), Biarritz, France, pp. 933-943.
15. S.K. D'Mello, S.D. Craig, A. Witherspoon, B. McDaniel, and A. Graesser Automatic detection of learner's affect from conversational cues User Modeling and User-Adapted Interaction: The Journal of Personalization Research 18 2008 45 80
16. P. Ekman, and W.V. Friesen The Facial Action Coding System: A Technique for the Measurement of Facial Movement 1978 Consulting Psychologists Press Palo Alto
17. Forbes-Riley, K., Litman, D., 2006. Modelling user satisfaction and student learning in a spoken dialogue tutoring system with generic, tutoring, and user affect parameters. In: Proceedings Human Language Technology Conference/North American chapter of the Association for Computational Linguistics annual meeting (HLT-NAACL).
18. Forbes-Riley, K., Litman, D., 2007. Investigating human tutor responses to student uncertainty for adaptive system development. In: Ana Paiva, R.P., Picard, R. (Eds.), Proceedings of Affective Computing and Intelligent Interaction (ACII). Springer Press, pp. 678-689.
19. K. Forbes-Riley, and D. Litman Analyzing dependencies between student certainness states and tutor responses in a spoken dialogue corpus L. Dybkjaer, W. Minker, Recent Trends in Discourse and Dialogue 2008 Springer 275 304
20. Forbes-Riley, K., Litman, D., 2009a. Adapting to student uncertainty improves tutoring dialogues. In: Proceedings 14th International Conference on Artificial Intelligence in Education (AIED), Brighton, UK (July).
21. Forbes-Riley, K., Litman, D., 2009b. A user modeling-based performance analysis of a wizarded uncertainty-adaptive dialogue system corpus. In: Proceedings Interspeech, Brighton, UK (September).
22. Forbes-Riley, K., Litman, D., Silliman, S., Tetreault, J., 2006. Comparing synthesized versus pre-recorded tutor speech in an intelligent tutoring spoken dialogue system. In: Proceedings of FLAIRS.
23. Forbes-Riley, K. Litman, D., Rotaru, M., 2008a. Responding to student uncertainty during computer tutoring: a preliminary evaluation. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS), Montreal, Canada (June).
24. Forbes-Riley, K., Litman, D., Silliman, S., Purandare, A., 2008b. Uncertainty corpus: resource to study user affect in complex spoken dialogue systems. In: Proceedings 6th Language Resources and Evaluation Conference (LREC), Marrakech, Morocco (May).
25. K. Forbes-Riley, M. Rotaru, and D. Litman The relative impact of student affect on performance models in a spoken dialogue tutoring system User Modeling and User-Adapted Interaction 18 1-2 2008 11 43
26. A.C. Graesser, and B. Olde How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down Journal of Educational Psychology 95 2003 524 536
27. A. Graesser, N. Person, and J. Magliano Collaborative dialog patterns in naturalistic one-on-one tutoring Applied Cognitive Psychology 9 1995 495 522
28. A.C. Graesser, P. Chipman, B.C. Haynes, and A. Olney AutoTutor: an intelligent tutoring system with mixed-initiative dialogue IEEE Transactions on Education 48 4 2005 612 618
29. J. Gratch, and S. Marsella Fight the way you train: the role and limits of emotions in training for combat The Brown Journal of World Affairs 10 1 2003 63 76
30. Hall, L. Woods, S., Sobral, D., Paiva, A., Dautenhahn, K., Wolke, D., Newall, L., 2004. Designing empathic agents: adults vs. kids. In: Proceedings of the Intelligent Tutoring Systems Conference (ITS), Maceio, Brazil, pp. 604-613.
31. Janarthanam, S., Lemon, O., 2008. User simulations for online adaptation and knowledge-alignment in troubleshooting dialogue systems. In: Proceedings of SEMdial.
32. J. Klein, Y. Moon, and R. Picard This computer responds to user frustration: theory, design, and results Interacting with Computers 14 2002 119 140
33. Kort, B. Reilly, R., Picard, R., 2001. An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion. In: Okamoto, T., Hartley, R., Kinshuk, J., Klus, P. (Eds), Proceedings IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges, Madison, WI, pp. 43-48.
34. J.R. Landis, and G.G. Koch The measurement of observer agreement for categorical data Biometrics 33 1977 159 174
35. C.M. Lee, and S. Narayanan Towards detecting emotions in spoken dialogs IEEE Transactions on Speech and Audio Processing 13 2 2005
36. Lehman, B., Matthews, M. D'Mello, S., Person, N., 2008. What are you feeling? Investigating student affective states during expert human tutoring sessions. In: Proceedings of Intelligent Tutoring Systems Conference (ITS), Montreal, Canada (June).
37. Liscombe, J. Venditti, J., Hirschberg, J., 2003. Classifying subject ratings of emotional speech using acoustic features. In: Proceedings of Interspeech/EuroSpeech, Geneva, Switzerland, pp. 725-728.
38. Litman, D., Forbes-Riley, K., 2004. Annotating student emotional states in spoken tutoring dialogues. In: Proceedings of the SIGdial Workshop on Discourse and Dialogue, Boston, USA, pp. 144-153.
39. D. Litman, and K. Forbes-Riley Recognizing student emotions and attitudes on the basis of utterances in spoken tutoring dialogues with both human and computer tutors Speech Communication 48 5 2006 559 590
40. D.J. Litman, and K. Forbes-Riley Correlations between dialogue acts and learning in spoken tutoring dialogues Journal of Natural Language Engineering: Special Issue on Educational Applications 12 2 2006 161 176
41. Litman, D., Forbes-Riley, K., 2009a. Improving (meta)cognitive tutoring by detecting and responding to uncertainty. In: Working Notes of the Cognitive and Metacognitive Educational Systems AAAI Symposium, Arlington, VA (November).
42. Litman, D., Forbes-Riley, K., 2009b. Spoken tutorial dialogue and the feeling of another's knowing. In: Proceedings 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), London, UK (September).
43. D.J. Litman, and S. Pan Designing and evaluating an adaptive spoken dialogue system User Modeling and User-Adapted Interaction 12 2/3 2002 111 137
44. Liu, K., Picard, R.W., 2005. Embedded empathy in continuous, interactive health assessment. In: CHI Workshop on HCI Challenges in Health Assessment.
45. Mairesse, F., Walker, M., 2008. Trainable generation of big-five personality styles through data-driven parameter estimation. In: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), Columbus, Ohio (June).
46. S. McQuiggan, B. Mott, and J. Lester Modeling self-efficacy in intelligent tutoring systems: an inductive approach User Modeling and User-Adapted Interaction (UMUAI) 18 1-2 2008 81 123
47. Möller, S., 2005. Parameters for quantifying the interaction with spoken dialogue telephone services. In: Proceedings of the SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal, pp. 166-177.
48. Narayanan, S., 2002. Towards modeling user behavior in human-machine interaction: effect of errors and emotions. In: Proceedings of the ISLE Workshop on Dialogue Tagging for Multi-modal Human Computer Interaction, Edinburgh, Scotland.
49. J.L. Nietfeld, C.K. Enders, and G. Schraw A monte carlo comparison of measures of relative and absolute monitoring accuracy Educational and Psychological Measurement 2006
50. Oberlander, J., Nowson, S., 2006. Whose thumb is it anyway? classifying author personality from weblog text. In: Proceedings of the 46 = 4th Annual Meeting of the Association for Computational Linguistics (ACL), Sydney, Australia (July).
51. P.-Y. Oudeyer The production and recognition of emotions in speech: features and algorithms International Journal of Human Computer Studies 59 1-2 2002 157 183
52. H. Pon-Barry, K. Schultz, E.O. Bratt, B. Clark, and S. Peters Responding to student uncertainty in spoken tutorial dialogue systems International Journal of Artificial Intelligence in Education 16 2006 171 194
53. K. Porayska-Pomsta, M. Mavrikis, and H. Pain Diagnosing and acting on student affect: the tutor's perspective User Modeling and User-Adapted Interaction: The Journal of Personalization Research 18 2008 125 173
54. H. Prendinger, and M. Ishizuka Let's talk! socially intelligent agents for language conversation training IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans (Special Issue on Socially Intelligent Agents-The Human in the Loop) 31 5 2001 465 471
55. Ringenberg, M., VanLehn, K., 2006. Scaffolding problem solving with annotated worked-out examples to promote deep learning. In: Proceedings of the International Conference on Intelligent Tutoring Systems, pp. 624-634.
56. P. Rozin, and A. Cohen High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of maturally occurring facial expressions of americans Emotion 3 2003 68 75
57. G.M.E. Saadawi, R. Azevedo, M. Castine, V. Payne, O. Medvedeva, E. Tseytlin, E. Legowski, D. azen Jukic, and R.S. Crowley Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold Advances in Health Sciences Education 2009
58. Shafran, I., Riley, M., Mohri, M., 2003. Voice signatures. In: Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), St. Thomas, US Virgin Islands, pp. 31-36.
59. S. Singh, D. Litman, M. Kearns, and M. Walker Optimizing dialogue management with reinforcement learning: experiments with the NJFun system Journal of Artificial Intelligence Research 16 2002 105 133
60. J.R. Tetreault, and D.J. Litman A reinforcement learning approach to evaluating state representations in spoken dialogue systems Speech Communication (Special Issue on Evaluating new methods and models for advanced speech-based interactive systems) 50 8-9 2008 683 696
61. Tsukahara, W., Ward, N., 2001. Responding to subtle, fleeting changes in the user's internal state. In: Proceedings of the SIG-CHI on Human Factors in Computing Systems, Seattle, WA. ACM, pp. 77-84.
62. VanLehn, K., Jordan, P.W., Rosé, C.P., Bhembe, D., Böttner, M., Gaydos, A., Makatchev, M., Pappuswamy, U., Ringenberg, M., Roque, A., Siler, S., Srivastava, R., Wilson, R., 2002. The architecture of Why2-Atlas: a coach for qualitative physics essay writing. In: Proceedings of Intelligent Tutoring Systems.
63. K. VanLehn, S. Siler, and C. Murray Why do only some events cause learning during human tutoring? Cognition and Instruction 21 3 2003 209 249
64. Vidrascu, L., Devillers, L., 2005. Detection of real-life emotions in dialogs recorded in a call center. In: Proceedings of INTERSPEECH, Lisbon, Portugal.
65. Walker, M.A., Cahn, J.E., Whittaker, S.J., 1997a. Improving linguistic style: social and affective bases for agent personality. In: Mller, J. (Ed.), Proceedings of Autonomous Agents.
66. Walker,M.A., Litman, D., Kamm, C., Abella, A., 1997b. PARADISE: a general framework for evaluating spoken dialogue agents. In: Proceedings of the Annual Meeting of the Association of Computational Linguistics (ACL), Madrid, Spain, pp. 271-280.
67. Walker, M., Fromer, J., Narayanan, S., 1998. Learning optimal dialogue strategies: a case study of a spoken dialogue agent for email. In: Proceedings of ACL/COLING.
68. Walker, M., Passonneau, R., Boland, J., 2001. Quantitative and qualitative evaluation of DARPA Communicator spoken dialogue systems. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Toulouse, France, pp. 515-522.
69. Walker, M., Rudnicky, A., Prasad, R., Aberdeen, J., Bratt, E., Garofolo, J., Hastie, H., Le, A., Pellom, B., Potamianos, A., Passonneau, R., Roukos, S., Sanders, G., Seneff, S., Stallard, D., 2002. DARPA Communicator: cross-system results for the 2001 evaluation. In: Proceedings of the International Conference on Spoken Language Processing (ICSLP), Denver, Colorado, USA, pp. 269-272.
70. Wang, N., Johnson, W., Rizzo, P., Shaw, E., Mayer, R., 2005. Experimental evaluation of polite interaction tactics for pedagogical agents. In: Proceedings of Intelligent User Interface Conference (IUI), pp. 12-19.
71. N. Wang, W. Johnson, R.E. Mayer, P. Rizzo, E. Shaw, and H. Collins The politeness effect: pedagogical agents and learning outcomes International Journal of Human-Computer Studies 66 2 2008 98 112
72. G.N. Yannakakis, J. Hallam, and H.H. Lund Entertainment capture through heart rate activity in physical interactive playgrounds User Modeling and User-Adapted Interaction: The Journal of Personalization Research 18 2008 207 243