Evaluating spoken dialogue systems according to de-facto standards: A case study

Computer Speech and Language

Volumn 21, Issue 1, 2007, Pages 26-53

  Möller, Sebastian ^a   Smeele, Paula ^b   Boland, Heleen ^c   Krebber, Jan ^a  

^a RUHR UNIVERSITY BOCHUM   (Germany)

^b TNO   (Netherlands)

^c EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

PERCEPTIVE DIMENSIONS; PREDICTION MODELS; QUESTIONNAIRE METHODS; SPOKEN DIALOGUE SYSTEMS;

DATABASE SYSTEMS; NUMBER THEORY; OPTIMIZATION; RELIABILITY; USABILITY ENGINEERING;

SPEECH RECOGNITION;

EID: 33749548286     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2005.11.003     Document Type: Article

References (30)

1. Bernsen N.O., Dybkjær H., and Dybkjær L. Designing Interactive Speech Systems: From First Ideas to User Testing (1998), Springer, Berlin
2. Bernsen, N.O., Dybkjær, H., Dybkjær, L., 1998. Principles for the design of cooperative spoken human-machine dialogue. In: Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP'96), Philadelphia PA, vol. 2, pp. 729-732.
3. Dahlbäck N., Jönsson A., and Ahrenberg L. Wizard-of-Oz studies - Why and How?. Knowledge-Based Systems 6 (1993) 258-266
4. Danieli M., and Gerbino E. Metrics for evaluating dialogue strategies in a spoken language system. Empirical Methods in Discourse Interpretation and Generation. Papers from the 1995 AAAI Symposium, Stanford CA (1995), AAAI Press, Menlo Park, CA 34-39
5. Dybkjær L., and Bernsen N.O. Usability Issues in Spoken Dialogue Systems. Natural Language Engineering 6 (2000) 243-271
6. Dybkjær L., Bernsen N.O., and Minker W. Evaluation and usability of multimodal spoken language dialogue systems. Speech Communication 43 (2004) 33-54
7. Fraser N. Assessment of interactive systems. In: Gibbon D., Moore R., and Winski R. (Eds). Handbook on Standards and Resources for Spoken Language Systems (1997), Mouton de Gruyter, Berlin 564-615
8. Fraser N.M., and Gilbert G.N. Simulating speech systems. Computer Speech and Language 5 (1991) 81-99
9. Glass, J., Polifroni, J., Seneff, S., Zue, V., 2000. Data collection and performance evaluation of spoken dialogue systems: the MIT experience. In: Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP 2000), Beijing, vol. 4, pp. 1-4.
10. Hartikainen, M., Salonen, E.-P., Turunen, M., 2004. Subjective evaluation of spoken dialogue systems using SERVQUAL method. In: Proceedings of the 8th International Conference on Spoken Language Processing (Interspeech 2004 - ICSLP), Jeju Island, vol. 3, pp. 2272-2276.
11. Hirschman L., and Thompson H.S. Overview of evaluation in speech and natural language processing. In: Cole R., Mariani J., Uszkoreit A., Zaenen H., and Zue V. (Eds). Survey of the State of the Art in Human Language Technology (1997), Cambridge University Press and Giardini Editori, Pisa 409-414
12. Hone, K.S., Graham, R., 2001. Subjective assessment of speech-system interface usability. In: Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech 2001 - Scandinavia), Aalborg, vol. 3, pp. 2082-2086.
13. Hone K.S., and Graham R. Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering 6 (2000) 303-387
14. ITU-T Rec. P.851, 2003. Subjective Quality Evaluation of Telephone Services Based on Spoken Dialogue Systems. International Telecommunication Union, Geneva.
15. Jekosch, U., 2000. Sprache hören und beurteilen: Ein Ansatz zur Grundlegung der Sprachqualitätsbeurteilung. Habilitation thesis, Universität/GH, Essen (Engl. version: Voice and Speech Quality Perception. Assessment and Evaluation, Springer, Berlin, 2005).
16. Möller S. Quality of Telephone-Based Spoken Dialogue Systems (2005), Springer, New York, NY
17. Möller, S., 2002. A new taxonomy for the quality of telephone services based on spoken dialogue systems. In: Proceedings of the 3rd SIGdial Workshop on Discourse and Dialogue, Philadelphia, PA, pp. 142-153.
18. Möller S., and Skowronek J. An analysis of quality prediction models for telephone-based spoken dialogue systems. Acta Acustica United with Acustica 90 (2004) 1112-1130
19. Parasuraman A., Zeithaml V.A., and Berry L.L. SERVQUAL: a multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing 64 (1988) 12-40
20. Polifroni, J., Hirschman, L., Seneff, S., Zue, V., 1992. Experiments in evaluating interactive spoken language systems. In: Proceedings of the DARPA Speech and Natural Language Workshop, Harriman, CA, pp. 28-33.
21. Price, P.J., Hirschman, L., Shriberg, E., Wade, E., 1992. Subject-based evaluation measures for interactive spoken language systems. In: Proceedings of the DARPA Speech and Natural Language Workshop, Harriman, CA, pp. 34-39.
22. Rajman M., Bui T.H., Rajman A., Seydoux F., Trutnev A., and Quarteroni S. Assessing the usability of a dialogue management system designed in the framework of a rapid dialogue prototyping methodology. Acta Acustica United with Acustica 90 (2004) 1096-1111
23. Simpson, A., Fraser, N.M., 1993. Black box and glass box evaluation of the SUNDIAL system. In: Proceedings of the 3rd European Conference on Speech Communication and Technology (Eurospeech'93), Berlin, vol. 2, pp. 1423-1426.
24. Skowronek, J., 2002. Entwicklung von Modellierungsansätzen zur Vorhersage der Dienstequalität bei der Interaktion mit einem natürlichsprachlichen Dialogsystem (Development of Quality-of-Service-Prediction Models for the Interaction with a Spoken Dialogue System). Unpublished Diploma thesis, IKA, Ruhr-Universität, Bochum.
25. Suhm, B., 2003. Towards best practices for speech user interface design. In: Proceedings of the 8th European Conference on Speech Communication and Technology (Eurospeech 2003 -Switzerland), Geneva, vol. 3, pp. 2217-2220.
26. van Leeuwen D., and Steeneken H. Consumer off-the-shelf speech technology product and service evaluation. In: Gibbon D., Merlins I., and Moore R.K. (Eds). Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation (2000), Kluwer Academic Publishers, Boston, MA 204-239
27. Walker, M., Rudnicky, A., Aberdeen, J., Bratt, E.G., Garofolo, J., Hastie, H., Le, A., Pellom, B., Potamianos, A., Passonneau, R., Prasad, R., Roukos, S., Sanders, G., Seneff, S., Stallard, D., 2002. DARPA Communicator Evaluation: Progress from 2000 to 2001. In: Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002), Denver, CO, vol. 1, pp. 273-276.
28. Walker M., Kamm C., and Litman D. Towards developing general models of usability with PARADISE. Natural Language Engineering 6 (2000) 363-377
29. Walker M.A., Litman D.J., Kamm C.A., and Abella A. Evaluating spoken dialogue agents with PARADISE: two case studies. Computer Speech and Language 12 (1998) 317-347
30. Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A., 1997. PARADISE: a framework for evaluating spoken dialogue agents. In: Proceedings of the ACL/EACL 35th Meeting of the Assoc. for Computational Linguistics, Madrid, pp. 271-280.