A framework for model-based evaluation of spoken dialog systems

ACL-08: HLT - Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue

Volumn , Issue , 2008, Pages 182-189

  Möller, Sebastian ^a   Ward, Nigel G ^b  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^b University of Texas at El Paso   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL RESEARCH;

DIALOG SYSTEMS; EVALUATION METHODS; EVALUATION PROBLEMS; MODEL-BASED EVALUATION; ONE PARTS; SERVICE PROVIDER; SPOKEN DIALOG SYSTEMS; SYSTEM DESIGNERS;

QUALITY CONTROL;

EID: 84857770344     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1622064.1622099     Document Type: Conference Paper

References (24)

1. M. Araki, and S. Doshita. 1997. Automatic Evaluation Environment for Spoken Dialogue Systems. Dialogue Processing in Spoken Language Systems, ECAI'96 Workshop Proceedings, Springer Lecture Notes in Artificial Intelligence No. 1236, 183-194, Springer, Berlin. (Pubitemid 127089501)
2. N. O. Bernsen, H. Dybkjær, and L. Dybkjær. 1998. Designing Interactive Speech Systems: From First Ideas to User Testing. Springer, Berlin.
3. N. O. Bernsen, L. Dybkjær, L., and S. Kiilerich. 2004. Evaluating Conversation with Hans Christian Andersen. Proc. 4th Int. Conf. on Language Resources and Evaluation (LREC 2004), 3, pp. 1011-1014, Lisbon.
4. K.-P. Engelbrecht, and S. Möller. 2007. Using Linear Regression Models for the Prediction of Data Distributions. Proc. 8th SIGdial Workshop on Discourse and Dialogue, Antwerp, pp. 291-294.
5. P. Heisterkamp. 2003. "Do not attempt to light with match!": Some Thoughts on Progress and Research Goals in Spoken Dialog Systems. Proc. 8th Europ. Conf. on Speech Communication and Technology (Eurospeech 2003 - Switzerland).
6. K. S. Hone, and R. Graham. 2000. Towards a Tool for the Subjective Assessment of Speech System Interfaces (SASSI). Natural Language Engineering, 3(3-4): 287-303.
7. ITU-T Rec. P.851. 2003. Subjective Quality Evaluation of Telephone Services Based on Spoken Dialogue Systems. International Telecommunication Union, Geneva.
8. ITU-T Suppl. 24 to P-Series Rec. 2005. Parameters Describing the Interaction with Spoken Dialogue Systems. International Telecommunication Union, Geneva.
9. ISO Standard 9241 Part 11. 1998. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) - Part 11: Guidance on Usability. International Organization for Standardization, Geneva.
10. U. Jekosch. 2000. Sprache hören und beurteilen: Ein Ansatz zur Grundlegung der Sprachqualitätsbeurteilung. Habilitation thesis (unpublished), Universität/Gesamthochschule, Essen.
11. R. López-Cózar, A. De la Torre, J. Segura, and A. Rubio. 2003. Assessment of Dialog Systems by Means of a New Simulation Technique. Speech Communication, 40: 387-407.
12. S. Möller, P. Smeele, H. Boland, and J. Krebber. 2007. Evaluating Spoken Dialogue Systems According to De-Facto Standards: A Case Study. Computer Speech and Language, 21: 26-53. (Pubitemid 44537644)
13. S. Möller, R. Englert, K.-P. Engelbrecht, V. Hafner, A. Jameson, A. Oulasvirta, A. Raake, and N. Reithinger. 2006. MeMo: Towards Automatic Usability Evaluation of Spoken Dialogue Services by User Error Simulations. Proc. 9th Int. Conf. on Spoken Language Processing (Interspeech 2006 - ICSLP), Pittsburgh PA, pp. 1786-1789.
14. S. Möller, and J. Heimansberg. 2006. Estimation of TTS Quality in Telephone Environments Using a Reference-free Quality Prediction Model. Proc. 2nd ISCA/DEGA Tutorial and Research Workshop on Perceptual Quality of Systems, Berlin, pp. 56-60.
15. S. Möller. 2005. Quality of Telephone-Based Spoken Dialogue Systems. Springer, New York NY.
16. S. Möller. 2005b. Perceptual Quality Dimensions of Spoken Dialogue Systems: A Review and New Experimental Results. Proc. 4th European Congress on Acoustics (Forum Acusticum Budapest 2005), Budapest, pp. 2681-2686.
17. A. Oulasvirta, S. Möller, K.-P. Engelbrecht, and A. Jameson. 2006. The Relationship of User Errors to Perceived Usability of a Spoken Dialogue System. Proc. 2nd ISCA/DEGA Tutorial and Research Workshop on Perceptual Quality of Systems, Berlin, pp. 61-67.
18. T. Paek. 2007. Toward Evaluation that Leads to Best Practices: Reconciling Dialog Evaluation in Research and Industry. Bridging the Gap: Academic and Industrial Research in Dialog Technologies Workshop Proceedings, Rochester, pp. 40-47.
19. R. Pieraccini, J. Huerta. 2005. Where Do We and Commercial Spoken Dialog Systems. Proc. 6th SIGdial Workshop on Discourse and Dialogue, Lisbon, pp. 1-10.
20. A. W. Rix, J. G. Beerends, D.-S. Kim, P. Kroon, and O. Ghitza. 2006. Objective Assessment of Speech and Audio Quality - Technology and Applications. IEEE Trans. Audio, Speech, Lang. Process, 14: 1890-1901.
21. M. Walker, C. Kamm, and D. Litman. 2000. Towards Developing General Models of Usability with PARADISE. Natural Language Engineering, 6: 363-377.
22. M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella. 1997. PARADISE: A Framework for Evaluating Spoken Dialogue Agents. Proc. of the ACL/EACL 35th Ann. Meeting of the Assoc. for Computational Linguistics, Madrid, Morgan Kaufmann, San Francisco CA, pp. 271-280.
23. N. Ward, A. G. Rivera, K. Ward, and D. G. Novick. 2005. Root Causes of Lost Time and User Stress in a Simple Dialog System. Proc. 9th European Conf. on Speech Communication and Technology (Interspeech 2005), Lisboa.
24. N. Ward, and W. Tsukahara. 2003. A Study in Responsiveness in Spoken Dialogue. International Journal of Human-Computer Studies, 59: 603-630.