Analysis of a new simulation approach to dialog system evaluation

Speech Communication

Volumn 51, Issue 12, 2009, Pages 1234-1252

  Engelbrecht, Klaus Peter ^a   Quade, Michael ^a   Möller, Sebastian ^a  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

Author keywords

Evaluation; Optimization; Prediction model; Spoken dialog system; Usability; User simulation

Indexed keywords

EVALUATION; PREDICTION MODEL; SPOKEN DIALOG SYSTEM; USABILITY; USER SIMULATION;

BEHAVIORAL RESEARCH; HUMAN COMPUTER INTERACTION; SIMULATORS; SOFTWARE AGENTS;

MATHEMATICAL MODELS;

EID: 69849105538     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2009.06.007     Document Type: Article

References (40)

1. Ai, H., Litman, D., 2006. Comparing real-real, simulated-simulated, and simulated-real spoken dialogue corpora. In: Proc. of the AAAI Workshop on Statistical and Empirical Approaches for Spoken Dialogue Systems, Boston, MA, USA.
2. Ai, H., Weng, F., 2008. User simulation as testing for spoken dialog systems. In: Proc. of the 9th SIGdial Workshop on Discourse and Dialogue, Columbus, OH, USA.
3. Ai, H., Litman, D., 2008. Assessing Dialog System User Simulation Evaluation Measures Using Human Judges. In: Proc. of the 46th Ann. Meeting of the Assoc. of Computational Linguistics (ACL), Columbus, OH, USA.
4. Anderson J.R., Bothell D., Byrne M.D., Douglass S., Lebiere C., and Qin Y. An integrated theory of the mind. Psychological Review 111 4 (2004) 1036-1060
5. Araki, M., Doshita, S., 1997. Automatic evaluation environment for spoken dialogue systems. In: ECAI'96: Workshop on Dialogue Processing in Spoken Language Systems, Springer, London, UK, pp. 183-194.
6. Bohus, D., Rudnicky, A.I., 2005. Sorry, I Didn't Catch That! - an investigation of non-understanding errors and recovery strategies. In: Proc. of the 6th SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal.
7. Card S.K., Moran T.P., and Newell A. The Psychology of Human-Computer Interaction (1983), Erlbaum, Hillsdale, NJ, USA
8. Chung, G., 2004. Developing a flexible spoken dialog system using simulation. In: Proc. of the 42nd Annual Meeting on Association for Computational Linguistics, Barcelona, Spain.
9. Eckert, W., Levin, E., Pieraccini, R., 1997. User modeling for spoken dialogue system evaluation. In: Proc. of the IEEE Workshop on Automatic Speech Recognition and Understanding, Santa Barbara, CA, USA.
10. Engelbrecht, K.-P., Kruppa, M., Möller, S., Quade, M., 2008. MeMo workbench for semi-automated usability testing. In: Proc. of Interspeech 2008, Brisbane, Australia.
11. Fraser N.M., and Gilbert G.N. Simulating speech systems. Computer Speech and Language 5 (1991) 81-99
12. Hermann, F., Niedermann, I., Peissner, M., Henke, K., Naumann, A., 2007. Users interact differently: towards a usability-oriented taxonomy. In: Proc. of HCI International 2007.
13. Ito, A., Shimada, K., Suzuki, M., Makino, S. 2006. A user simulator based on VoiceXML for evaluation of spoken dialog systems. In: Proc. of Interspeech 2006, Pittsburgh, PA, USA.
14. ITU-T Supplement 24 to P-Series Recommendations, 2005. Parameters describing the interaction with spoken dialogue systems. International Telecommunication Union, Geneva, Switzerland.
15. ITU-T Recommendation P.851, 2003. Subjective quality evaluation of telephone services based on spoken dialogue systems. International Telecommunication Union, Geneva, Switzerland.
16. Ivory, M.Y., Hearst, M.A., 2000. The state of the art in automating usability evaluation of user interfaces (Technical Report UCB/CSD-00-1105). EECS Department, University of California, Berkeley, CA, USA.
17. Janarthanam, S., Lemon, O., 2008. User simulations for online adaptation and knowledge-alignment in Troubleshooting dialogue systems. In: Proc. of SEMDIAL 2008 (LONDIAL), London, UK.
18. John B.E., and Salvucci D.D. Multipurpose prototypes for assessing user interfaces in pervasive computing systems. IEEE Pervasive Computing 4 4 (2005) 27-34
19. Kieras D.E. Model-based evaluation. In: Jacko J., and Sears A. (Eds). The Human-Computer Interaction Handbook (2003), Erlbaum, Mahwah, NJ, USA 1191-1208
20. López-Cózar R., de la Torre A., Segura J.C., and Rubio A.J. Assessment of dialogue systems by means of a new simulation technique. Speech Communication 40 3 (2003) 387-407
21. Möller, S., Englert, R., Engelbrecht, K.-P., Hafner, V., Jameson, A., Oulasvirta, A. et al., 2006. MeMo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In: Proc. of Interspeech 2006, Pittsburgh, PA, USA.
22. Möller, S., Engelbrecht, K.-P., Oulasvirta, A., 2007a. Analysis of communication failures for spoken dialogue systems. In: Proc. of Interspeech 2007, Antwerp, Belgium.
23. Möller S., Smeele P., Boland H., and Krebber J. Evaluating spoken dialogue systems according to De-Facto standards: a case study. Computer Speech and Language 21 (2007) 26-53
24. Möller, S., Engelbrecht, K.-P., Pucher, M., Fröhlich, P., Huo, L., Heute, U., et al., 2007c. TIDE: a testbed for interactive spoken dialogue system evaluation. In: Proc. of 12th International Conference "Speech and Computer" (SPECOM'2007), Moscow, Russia.
25. Newell A. Unified Theories of Cognition (1990), Harvard University Press, Cambridge, MA, USA
26. Nielsen J. Usability Engineering (1993), Academic Press, San Diego, CA, USA
27. Norman D.A. Categorization of action slips. Psychological Review 88 (1981) 1-15
28. Norman D.A. Some observations on mental models. In: Gentner D., and Stevens A.L. (Eds). Mental Models (1983), Erlbaum, Hillsdale, NJ, USA 7-14
29. Pietquin, O., Renals, S., 2002. ASR system modeling for automatic evaluation and optimization of dialogue systems. In: Proc. of the IEEE Int. Conf. on Acoust. Speech Signal Process. (ICASSP 2002), Orlando, FL, USA.
30. Pietquin, O., 2006. Consistent goal-directed user model for realisitc man-machine task-oriented spoken dialogue. In: Proc. of the IEEE Internat. Conf. on Multimedia and Expo (ICME), Toronto, ON, Canada.
31. Pietquin O., 2009. Machine learning methods for spoken dialog simulation and optimization. In: Mellouk, A., Chebira, A. (Eds.), Machine Learning, In-Teh, pp. 167-184.
32. Rieser, V., Lemon, O., 2006. Cluster-based user simulations for learning dialogue strategies. In: Proc. of Interspeech 2006, Pittsburgh, PA, USA.
33. Schatzmann, J., Georgila, K., Young, S., 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proc. of the 6th SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal.
34. Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S., 2007a. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In: Proc. of HLT/NAACL, Rochester, NY, USA.
35. Schatzmann, J., Thomson, B., Young, S., 2007b. Statistical user simulation with a hidden agenda. In: Proc. of the 8th SIGDial Workshop on Discourse and Dialogue, Antwerp, Belgium.
36. Schatzmann, J., Thomson, B., Young, Steve, 2007c. Error simulation for training statistical dialogue systems. In: Proc. of ASRU, Kyoto, Japan.
37. Scheffler, K., Young, S., 2001. Corpus-based dialogue simulation for automatic strategy learning and evaluation. In: Proc. of the NAACL-2001 Workshop on Adaptation in Dialogue Systems, Pittsburgh, PA, USA.
38. Seneff S. Response planning and generation in the MERCURY flight reservation system. Computer Speech and Language 16 (2002) 283-312
39. Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A., 1997. PARADISE: a framework for evaluating spoken dialogue agents. In: Proc. of the ACL/EACL 35th Ann. Meeting of the Assoc. for Computational Linguistics, Madrid, Spain.
40. Walker M., Kamm C., and Litmann A. Towards developing general models of usability with PARADISE. Natural Language Engineering 6 3-4 (2000) 363-377