Intrinsic and extrinsic evaluation of an automatic user disengagement detector for an uncertainty-adaptive spoken dialogue system

NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Volumn , Issue , 2012, Pages 91-102

  Forbes Riley, Kate ^a   Litman, Diane ^a   Friedberg, Heather ^a   Drummond, Joanna ^a,b  

^a UNIVERSITY OF PITTSBURGH   (United States)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; REGRESSION ANALYSIS; SPEECH PROCESSING;

AFFECTIVE STATE; CORRELATION ANALYSIS; GOLD STANDARDS; IMPROVE PERFORMANCE; MODELING PERFORMANCE; REAL-WORLD TASK; SPOKEN DIALOGUE; SPOKEN DIALOGUE SYSTEM; SYSTEMS PERFORMANCE; UNCERTAINTY;

UNCERTAINTY ANALYSIS;

EID: 84906491889     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (51)

1. S. Afzal and P. Robinson. 2011. Natural affect data: Collection and annotation. In Sidney D'Mello and Rafael Calvo, editors, Affect and Learning Technolo- gies. Springer.
2. G. Aist, B. Kort, R. Reilly, J. Mostow, and R. Picard. 2002. Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: Adding human-provided emotional scaffolding to an automated reading tutor that listens. In Proc. In- telligent Tutoring Systems Conference (ITS) Workshop on Empirical Methods for Tutorial Dialogue Systems, pages 16-28, San Sebastian, Spain.
3. A. Alwan, Y. Bai, M. Black, L. Caseyz, M. Gerosa, M. Heritagez, M. Iseliy, M. Jonesz, A. Kazemzadeh, S. Lee, S. Narayanan, P. Pricex, J. Tepperman, and S. Wangy. 2007. A system for technology based assessment of language and literacy in young children: the role of multiple information sources. In Proceedings of the 9th IEEE International Workshop on Multi- media Signal Processing (MMSP), pages 26-30, Chania, Greece, October.
4. J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke. 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In J. H. L. Hansen and B. Pellom, editors, Proceedings of the International Conference on Spoken Language Processing (ICSLP), pages 2037-2039, Denver, USA.
5. A. Batliner, S. Steidl, C. Hacker, and E. Noth. 2008. Private emotions vs. social interaction - a data-driven approach towards analysing emotion in speech. User Modeling and User-Adapted Interaction: The Journal of Personalization Research, 18:175-206.
6. A. L. Baylor, J. Ryu, and E. Shen. 2003. The effect of pedagogical agent voice and animation on learning, motivation, and perceived persona. In Proceedings of the ED-MEDIA Conference, Honolulu, Hawaii, June.
7. A. Black and P. Taylor. 1997. Festival speech synthesis system: system documentation (1.1.1). The Centre for Speech Technology Research, University of Edinburgh, http://www.cstr.ed.ac.uk/projects/festival/.
8. D. Bohus and E. Horvitz. 2009. Models for multiparty engagement in open-world dialog. In Proceedings of SIGdial, London, UK.
9. C. Conati and H. Maclaren. 2009. Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3):267-303.
10. R. Cowie and R. R. Cornelius. 2003. Describing the emotional states that are expressed in speech. Speech Communication, 40(1-2):5-32.
11. L. Devillers and L. Vidrascu. 2006. Real-life emotions detection with lexical and paralinguistic cues on human-human call center dialogs. In Ninth Inter- national Conference on Spoken Language Processing (ICSLP, pages 801-804, Pittsburgh, PA, September.
12. S. D'Mello, S. Craig, A. Witherspoon, B. McDaniel, and A. Graesser. 2008. Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction: The Journal of Personal- ization Research, 18:45-80.
13. S. D'Mello, B. Lehman, J. Sullins, R. Daigle, R. Combs, K. Vogt, L. Perkins, and A. Graesser. 2010. A time for emoting: When affect-sensitivity is and isn't effective at promoting deep learning. In Intelligent Tutoring Systems Conference, pages 245-254, Pittsburgh, PA, USA, June.
14. J. Drummond and D. Litman. 2011. Examining the impacts of dialogue content and system automation on affect models in a spoken tutorial dialogue system. In Proc. 12th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 312-318, Portland, Oregon, June.
15. M. Dzikovska, J. Moore, N. Steinhauser, and G. Campbell. 2011. Exploring user satisfaction in a tutorial dialogue system. In Proc. 12th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 162-172, Portland, Oregon, June.
16. E. Florian, M. Wollmer, and B. Schuller. 2010. The Munich versatile and fast open-source audio feature extractor. In Proc. ACM Multimedia (MM), pages 1459-1462, Florence, Italy.
17. K. Forbes-Riley and D. Litman. 2009. A user modelingbased performance analysis of a wizarded uncertaintyadaptive dialogue system corpus. In Proc. Inter- speech, Brighton, UK, September.
18. K. Forbes-Riley and D. Litman. 2011a. Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Communication, 53(9-10):1115-1136.
19. K. Forbes-Riley and D. Litman. 2011b. When does disengagement correlate with learning in spoken dialog computer tutoring? In Proceedings 15th Interna- tional Conference on Artificial Intelligence in Educa- tion (AIED), Auckland, NZ, June.
20. K. Forbes-Riley, M. Rotaru, and D. Litman. 2008. The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Model- ing and User-Adapted Interaction, 18(1-2):11-43.
21. K. Forbes-Riley, D. Litman, and H. Friedberg. 2011. Annotating disengagement for spoken dialogue computer tutoring. In Sidney D'Mello and Rafael Calvo, editors, Affect and Learning Technologies. Springer.
22. Jonathan Gratch, Stacy Marsella, Ning Wang, and Brooke Stankovic. 2009. Assessing the validity of appraisal-based models of emotion. In Proceedings of ACII, Amsterdam, Netherlands.
23. X. D. Huang, F. Alleva, H. W. Hon, M. Y. Hwang, K. F. Lee, and R. Rosenfeld. 1993. The SphinxII speech recognition system: An Overview. Computer, Speech and Language.
24. J. H. Jeon, R. Xia, and Y. Liu. 2010. Level of interest sensing in spoken dialog using multi-level fusion of acoustic and lexical evidence. In INTERSPEECH'10, pages 2802-2805.
25. P. Jordan, B. Hall, M. Ringenberg, Y. Cui, and C.P. Rose. 2007. Tools for authoring a dialogue agent that participates in learning studies. In Proc. Artificial Intelli- gence in Education (AIED), pages 43-50.
26. A. Kapoor and R. W. Picard. 2005. Multimodal affect recognition in learning environments. In 13th Annual ACM International Conference on Multimedia, pages 677-682, Singapore.
27. J. Klein, Y. Moon, and R. Picard. 2002. This computer responds to user frustration: Theory, design, and results. Interacting with Computers, 14:119-140.
28. C. M. Lee and S. Narayanan. 2005. Towards detecting emotions in spoken dialogs. IEEE Transactions on Speech and Audio Processing, 13(2), March.
29. B. Lehman, M. Matthews, S. D'Mello, and N. Person. 2008. What are you feeling? Investigating student affective states during expert human tutoring sessions. In Intelligent Tutoring Systems Conference (ITS), pages 50-59, Montreal, Canada, June.
30. D. Litman and K. Forbes-Riley. 2009. Spoken tutorial dialogue and the feeling of another's knowing. In Pro- ceedings 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), London, UK, September.
31. K. Liu and R. W. Picard. 2005. Embedded empathy in continuous, interactive health assessment. In CHI Workshop on HCI Challenges in Health Assessment.
32. A. Martalo, N. Novielli, and F. de Rosis. 2008. Attitude display in dialogue patterns. In Proc. AISB 2008 Sym- posium on Affective Language in Human andMachine, pages 1-8, Aberdeen, Scotland, April.
33. S. McQuiggan, B. Mott, and J. Lester. 2008. Modeling self-efficacy in intelligent tutoring systems: An inductive approach. User Modeling and User-Adapted Interaction (UMUAI), 18(1-2):81-123, February.
34. T. Paek and Y.-C. Ju. 2008. Accommodating explicit user expressions of uncertainty in voice search or something like that. In Proceedings of the 9th Annual Conference of the International Speech Communica- tion Association (INTERSPEECH 08), pages 1165-1168, Brisbane, Australia, September.
35. H. Pon-Barry and S. Shieber. 2011. Recognizing uncertainty in speech. EURASIP Journal on Advances in Signal Processing.
36. H. Pon-Barry, K. Schultz, E. Owen Bratt, B. Clark, and S. Peters. 2006. Responding to student uncertainty in spoken tutorial dialogue systems. International Jour- nal of Artificial Intelligence in Education, 16:171-194.
37. K. Porayska-Pomsta, M. Mavrikis, and H. Pain. 2008. Diagnosing and acting on student affect: the tutor's perspective. User Modeling and User-Adapted In- teraction: The Journal of Personalization Research, 18:125-173.
38. H. Prendinger and M. Ishizuka. 2005. The Empathetic Companion: A character-based interface that addresses users' affective states. International Journal of Applied Artificial Intelligence, 19(3):267-285.
39. M. Rotaru and D. Litman. 2009. Discourse structure and performance analysis: Beyond the correlation. In Pro- ceedings 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), London, UK.
40. B. Schuller, R. Muller, F. Eyben, J. Gast, B. Hrnler, M. Wollmer, G. Rigoll, A. Hthker, and H. Konosu. 2009a. Being bored? recognising natural interest by extensive audiovisual integration for real-life application. Image and Vision Computing Journal, Special Issue on Visual and Multimodal Analysis of Human Spontaneous Behavior, 27:1760-1774.
41. B. Schuller, S. Steidl, and A. Batliner. 2009b. The Interspeech 2009 Emotion Challenge. In Proceedings of the 10th Annual Conference of the Inter- national Speech Communication Association (Inter- speech), ISCA, Brighton, UK, September.
42. B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. Muller, and S. Narayanan. 2010. The Interspeech 2010 Paralinguistic Challenge. In Pro- ceedings of the 11th Annual Conference of the In- ternational Speech Communication Assocation (Inter- speech), pages 2794-2797, Chiba, Japan, September.
43. I. Shafran, M. Riley, and M. Mohri. 2003. Voice signatures. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 31-36, St. Thomas, US Virgin Islands.
44. C. Sidner and C. Lee. 2003. An architecture for engagement in collaborative conversations between a robot and a human. Technical Report TR2003-12, MERL.
45. A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. Van Ess- Dykema, K. Ries, E. Shriberg, D. Jurafsky, R. Martin, and M. Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3).
46. W. Tsukahara and N. Ward. 2001. Responding to subtle, fleeting changes in the user's internal state. In Pro- ceedings of the SIG-CHI on Human factors in comput- ing systems, pages 77-84, Seattle, WA. ACM.
47. K. VanLehn, P. W. Jordan, C. Rosé, D. Bhembe, M. Böttner, A. Gaydos, M. Makatchev, U. Pappuswamy, M. Ringenberg, A. Roque, S. Siler, R. Srivastava, and R. Wilson. 2002. The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In Proc. Intl. Conf. on Intelligent Tutoring Systems.
48. L. Vidrascu and L. Devillers. 2005. Detection of reallife emotions in dialogs recorded in a call center. In Proceedings of INTERSPEECH, Lisbon, Portugal.
49. M. Walker, A. Rudnicky, R. Prasad, J. Aberdeen, E. Bratt, J. Garofolo, H. Hastie, A. Le, B. Pellom, A. Potamianos, R. Passonneau, S. Roukos, G. Sanders, S. Seneff, and D. Stallard. 2002. DARPA communicator: Cross-system results for the 2001 evaluation. In Proc. ICSLP.
50. W. Wang and J. Hirschberg. 2011. Detecting levels of interest from spoken dialog with multistream prediction feedback and similarity based hierarchical fusion learning. In Proc. 12th Annual Meeting of the Spe- cial Interest Group on Discourse and Dialogue (SIG- DIAL), pages 152-161, Portland, Oregon, June.
51. N. Wang, W.L. Johnson, R. E. Mayer, P. Rizzo, E. Shaw, and H. Collins. 2008. The politeness effect: Pedagogical agents and learning outcomes. International Journal of Human-Computer Studies, 66(2):98-112.