Evaluation and usability of multimodal spoken language dialogue systems

Speech Communication

Volumn 43, Issue 1-2, 2004, Pages 33-54

  Dybkjær, Laila ^a   Bernsen, Niels Ole ^a   Minker, Wolfgang ^b  

^a Natural Interactive Systems Laboratory   (Denmark)

^b UNIVERSITY OF ULM   (Germany)

Author keywords

Evaluation projects; Generalisations; Guidelines; State of the art; Theory

Indexed keywords

COMPUTER GRAPHICS; MOBILE TELECOMMUNICATION SYSTEMS; PERSONAL DIGITAL ASSISTANTS; SPEECH COMMUNICATION; TELECOMMUNICATION SERVICES;

EVALUATION PROJECTS; GENERALIZATIONS; GUIDELINES; STATE-OF-THE-ART;

SPEECH PROCESSING;

EID: 2942598511     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2004.02.001     Document Type: Article

References (66)

1. Allen J., Schubert L.K., Ferguson G., Heeman P., Hwang C.H., Kato T., Light M., Martin N.G., Miller B.W., Poesio M., Traum D.R. The TRAINS project: A case study in building a conversational planning agent. Journal of Experimental and Theoretical AI (JETAI). 7:1995;7-48.
2. Almeida, L., Amdal, I., Beires, N., Boualem, M., Boves, L., den Os, L., Filoche, P., Gomes, R., Knudsen, J., Kvale, K., Rugelbak, J., Tallec, C., Warakagoda, N., 2002. Implementing and evaluating a multimodal tourist guide. In: Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, Copenhagen. pp. 1-7.
3. Baekgaard, A., Bernsen, N.O., Brøndsted, T., Dalsgaard, P., Dybkjær, H., Dybkjær, L., Kristiansen, J., Larsen, L., Lindberg, B., Maegaard, B., Music, B., Offersgaard, L., Povlsen, C., 1995. The Danish spoken dialogue project - a general overview. In: Proceedings of the ESCA Workshop on Spoken Dialogue Systems, Vigsø, Denmark. pp. 89-92.
4. Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E., 2000. Desperately seeking emotions: Actors, wizards, and human beings. In: Proceedings of the ISCA Workshop on Speech and Emotion: A Conceptual Framework for Research, Belfast. pp. 195-200.
5. Beringer, N., Kartal, U., Louka, K., Schiel, F., Türk, U., 2002. PROMISE - a procedure for multimodal interactive system evaluation. In: Proceedings of the LREC Workshop on Multimodal Resources and Multimodal Systems Evaluation, Las Palmas. pp. 77-80.
6. Bernsen N.O. Foundations of multimodal representations. A taxonomy of representational modalities. Interacting with Computers. 6(4):1994;347-371.
7. Bernsen N.O. Towards a tool for predicting speech functionality. Speech Communication. 23:1997;181-210.
8. Bernsen N.O. Multimodality in language and speech systems - from theory to design support tool. Granström B., House D., Karlsson I. Multimodality in Language and Speech Systems. 2002;93-148 Kluwer Academic Publishers.
9. Bernsen N.O. User modelling in the car. Proceedings of the Ninth International Conference on User Modeling (UM 2003), Lecture Notes in Artificial Intelligence. 2003;378-382 Springer Verlag.
10. Bernsen N.O., Dybkjær H., Dybkjær L. Designing interactive speech systems. From first ideas to user testing. 1998;Springer Verlag.
11. Bernsen N.O., Dybkjær L. A theory of speech in multimodal systems. Dalsgaard P., Lee C.-H., Heisterkamp P., Cole R. Proceedings of the ESCA Workshop on Interactive Dialogue in Multi-Modal Systems. 1999;105-108 European Speech Communication Association, Kloster Irsee, Germany, Bonn.
12. Bernsen, N.O., Dybkjær, L., 2000. A methodology for evaluating spoken language dialogue systems and their components. In: Proceedings of the Second International Conference on Language Resources and Evaluation (LREC, 2000), Athens. pp. 183-188.
13. Bickmore, T., Cassell, J., 2002. Phone vs. face-to-face with virtual persons. In: Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, Copenhagen. pp. 15-22.
14. Boros, M., Eckert, W., Gallwitz, F., Görz, G., Hanrieder, G., Niemann, H., 1996. Towards understanding spontaneous speech: Word accuracy vs. concept accuracy. In: Proceedings of the International Conference on Spoken Language Processing (ICSLP 1996), Philadelphia, Vol. 2. pp. 1009-1012.
15. Bossemeyer R.W., Schwab E.C. Automated alternate billing services at Ameritech: Speech recognition and the human interface. Speech Technology Magazine. 5:1991;24-30.
16. Brusilovsky, P., Corbett, A., de Rosis, F. (Eds.), 2003. User Modeling 2003 Proceedings of the 9th International Conference, UM 2003, Johnstown, PA, USA, June 2003, Springer Lecture Notes in Artificial Intelligence, Vol. 2702.
17. Buisine, S., Martin, J.C., 2003. Experimental evaluation of bi-directional multimodal interaction with conversational agents. In: Proceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction (INTERACT 2003), Zürich, Switzerland.
18. Bühler, D., Minker, W., Häussler, J., Krüger, S., 2002. Flexible multimodal human-machine interaction in mobile environments. In: Proceedings of the ECAI Workshop on Artificial Intelligence in Mobile System (AIMS), Lyon. pp. 66-70.
19. Cassell J., Sullivan J., Prevost S., Churchill E. Embodied Conversational Agents. 2000;MIT Press.
20. Cohen I., Sebe N., Chen L., Garg A., Huang T. Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, Special Issue on Face Recognition. 91(1-2):2003;160- 187.
21. Cohen, P., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., J. Clow, J., 1997. QuickSet: Multimodal interaction for distributed applications. In: Fifth ACM International Multimedia Conference. pp. 31-40.
22. Dybkjær L., Bernsen N.O. Usability issues in spoken language dialogue systems. Natural Language Engineering, Special Issue on Best Practice in Spoken Language Dialogue System Engineering. 6:2000;243-272.
23. Dybkjær, L., Bernsen, N.O., Blasig, R., Buisine, S., Fredriksson, M., Gustafson, J., Martin, J.C., Wirn, M., 2003. Evaluation criteria and evaluation plan. Technical Report, NICE Deliverable D7.1, University of Southern Denmark.
24. Dybkjær, L., Bernsen, N.O., Carlson, R., Chase, L., Dahlbäck, N., Failenschmid, K., Heid, U., Heisterkamp, P., Jönsson, A., Kamp, H., Karlsson, I., Kuppevelt, J., Lamel, L., Paroubek, P., Williams, D. 1998a. The DISC approach to spoken language systems development and evaluation. In: Proceedings of the First International Conference on Language Resources and Evaluation (LREC 1998), Granada. pp. 185-189.
25. Dybkjær L., Bernsen N.O., Dybkjær H. A methodology for diagnostic evaluation of spoken human-machine dialogue. International Journal of Human Computer Studies, Special Issue on Miscommunication. 48:1998;605-625.
26. Ekman P., Friesen W. Unmasking the Face: A Guide to Recognize Emotions from Facial Clues. 1975;Prentice-Hall, Inc.
27. Ferguson, G., Allen, J., 1998. TRIPS: An integrated intelligent problem-solving assistant. In: Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI 1998), Madison, USA. pp. 567-572.
28. Fraser, N., 1997. Spoken dialogue system evaluation: A first framework for reporting results. In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 1997), Rhodes. pp. 1907-1910.
29. Gärtner, U., König, W., Wittig, T., 2001. Evaluation of manual vs. speech input when using a driver information system in real traffic. In: Proceedings of the International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Aspen, Colorado.
30. Gibbon, D., Moore, R., Winski, R., 1997. Handbook of Standards and Resources for Spoken Language Systems. Walter de Gruyter.
31. Gilbert, N., Cheepen, C., Failenschmid, K., Williams, D., 1999. Guidelines for advanced spoken dialogue design. Available from 〈http://www.soc.surrey.ac.uk/research/guidelines〉.
32. Glass, J., Polifroni, J., Seneff, S., Zue, V., 2000. Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In: Proceedings of the International Conference on Spoken Language Processing, (ICSLP 2000), Beijing, Vol. 4. pp. 1-4.
33. Grice P. Logic and conversation. Cole P., Morgan J.L. Syntax and Semantics, Vol. 3: Speech Acts. 1975;41-58 Academic Press, New York.
34. Gustafson, J., Bell, L., Beskow, J., Boye, J., Carlson, R., Edlund, J., Granström, B., House, D., Wiren, M., 2000. AdApt - a multimodal conversational dialogue system in an apartment domain. In: Proceedings of the International Conference on Spoken Language Processing (ICSLP 2000), Vol. 2. Beijing, pp. 134-137.
35. Gustafson, J., Lindberg, N., Lundeberg, M., 1999. The August spoken dialogue system. In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 1999), Budapest. pp. 1151-1154.
36. Handrieder, G., Heisterkamp, P., Brey, T., 1998. Fly with the EAGLES: Evaluation of the ACCeSS spoken dialogue system. In: Proceedings of the International Conference on Spoken Language Processing (ICSLP 1998), Sydney. pp. 503-506.
37. Hirschberg, J., Swerts, M., Litman, D., 2001. Labeling corrections and aware sites in spoken dialogue systems. In: Proceedings of the 2nd SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark. pp. 72-79.
38. Hjalmarson, A., 2002. Evaluating AdApt, a multi-modal conversational dialogue system using PARADISE. Ph.D. Thesis, KTH, Stockholm.
39. Karlsson, I., 1999. A survey of existing methods and tools for development and evaluation of speech synthesis and speech synthesis quality in SLDSs. Technical report, DISC Deliverable D2.3.
40. King, M., Maegard, B., Schutz, J., des Tombes, L., 1996. EAGLES - Evaluation of natural language processing systems. Technical Report EAG-EWG-PR.2.
41. Komatani, K., Ueno, S., Kawahara, T., Okuno, H., 2003. User modeling in spoken dialogue systems for flexible guidance generation. In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 2003), Geneva. pp. 745-748.
42. Larsen, L.B., 2003. Assessment of spoken dialogue system usability - What are we really measuring? In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 2003), Geneva. pp. 1945-1948.
43. Leavitt N. Two technologies vie for recognition in speech market. IEEE Computer. 36(6):2003;13-16.
44. Mariani, J., 2002. Technolangue: language technology. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC2002), Las Palmas. Available from 〈http://www.lrec-conf.org/lrec2002/ lrec/isplre/ISP_Description_2002.pdf〉.
45. Mariani, J., Paroubek, P., 1999. Human language technologies evaluation in the European framework. In: Proceedings of the DARPA Broadcast News Workshop. pp. 237-242.
46. Minker, W., Haiber, U., Heisterkamp, P., Scheible, S., 2002. Intelligent dialogue strategy for accessing infotainment applications in mobile environments. In: Proceedings of the ISCA Workshop on Multi-Modal Spoken Dialogue in Mobile Environments. European Speech Communication Association, Kloster Irsee, Germany. Bonn.
47. Oviatt, S., 1997. Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, Special Issue on Multimodal Interfaces. pp. 93-129.
48. Oviatt, S., 2001. Advances in the robust processing of multimodal speech and pen systems. In: Yuen, P.C., Y. T., Wang, P. (Eds.), Multimodal Interfaces for Human Machine Communication, World Scientific Publisher: London, Series on Machine Perception and Artificial Intelligence. pp. 203-218.
49. Pallett, D., Fiscus, J., Fisher, W., Garofolo, J., Lund, B., Martin, A., Przybocki, M., 1994. 1994 benchmark tests for the ARPA spoken language program. In: Proceedings of the ARPA Workshop on Spoken Language Technology. pp. 5-36.
50. Peckham, J., 1993. A new generation of spoken dialogue systems: Results and lessons from the SUNDIAL project. In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 1993), Berlin. pp. 33-40.
51. Polifroni, J., Seneff, S., 2000. Galaxy-II as an architecture for spoken dialogue evaluation. In: Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000), Athens. pp. 725-730.
52. Ramshaw, L., Boisen, S., 1990. An SLS answer comparator. Technical report, BBN Systems and Technologies Corporation. SLS Note 7.
53. Roth S., Chuah M., Kerpedjiev S., Kolojejchick J., Lucas P. Towards an information visualization workspace: Combining multiple means of expression. Human-Computer Interaction. 12:1997;131-185.
54. Sanders, G., Le, A., Garofolo, J., 2002. Effects of word error rate in the DARPA Communicator data during 2000 and 2001. In: Proceedings of the International Conference of Spoken Language Processing, (ICSLP 2002), Denver. pp. 277-280.
55. Seneff, S., Hurley, E., Lau, R., Paoa, C., Schmid, P., Zue, V., 1998. Galaxy-II: A reference architecture for conversational system development. In: Proceedings of the International Conference of Spoken Language Processing, (ICSLP 1998), Sydney. pp. 931-934.
56. Simpson, A., Fraser, N., 1993. Blackbox and glassbox evaluation of the SUNDIAL system. In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 1993), Berlin. pp. 1423-1426.
57. Sturm, J., Cranen, B., Wang, F., Terken, J., Bakx, I., 2002. The effect of prolonged use on multimodal interaction. In: Proceedings of the ISCA Workshop on Multi-Modal Spoken Dialogue in Mobile Environments. European Speech Communication Association, Kloster Irsee, Germany. Bonn.
58. Sturm, J., den Os, E., Boves, L., 1999. Issues in spoken dialogue systems: Experiences with the Dutch ARISE system. In: Proceedings of ESCA Workshop on Interactive Dialogue in Multi-Modal Systems, Kloster Irsee, Germany. pp. 1-4.
59. Temem, J., Lamel, L., Gauvain, J., 1999. The MASK demonstrator: An emerging technology for user-friendly passengers kiosk. In: World Congress on Railway Research.
60. Wahlster, W., 1993. Verbmobil - Translation of face to face dialogues. In: Machine Translation Summary IV, Kobe, Japan.
61. Wahlster, W., Reithinger, N., Blocher, A., 2001. SmartKom: Multimodal communication with a life-like character. In: Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 2001), Aalborg, Denmark. pp. 1547-1550.
62. Walker, M., Hirschmann, L., Aberdeen, J., 2000a. Evaluation for DARPA COMMUNICATOR spoken dialogue systems. In: Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000), Athens, Greece. pp. 735-741.
63. Walker, M., Kamm, C., Litman, D., 2000b. Towards developing general models of usability with PARADISE. Natural Language Engineering, Special Issue on Spoken Dialogue Systems 6 (3).
64. Walker, M., Litman, D., Kamm, C., Abella, A., 1997. PARADISE: A general framework for evaluating spoken dialogue agents. In: Proceedings of the 35th Annual Meeting of the Association of Computational Linguistics (ACL/EACL 1997). pp. 271-280.
65. Walker, M., Rudnicky, A., Prasad, R., Aberdeen, J., Bratt, E., Garofolo, J., Hastie, H., Le, A., Pellom, B., Potamianos, A., Passonneau, R., Roukos, S., Sanders, G., Seneff, S., Stallard, D., 2002. DARPA Communicator: Cross-system results for the 2001 evaluation. In: Proceedings of the International Conference of Spoken Language Processing (ICSLP 2002), Denver. pp. 269-272.
66. Young S., Adda-Decker M., Aubert X., Dugast C., Gauvain J., Kershaw D., Lamel L., Leeuwen D., Pye D., Robinson A., Steeneken H., Woodland P. Multilingual large vocabulary speech recognition: The European SQALE project. Computer Speech and Language. 11:1997;73-89.