Empirical methods for evaluating dialog systems

Proceedings of the SIGDIAL 2001 Workshop - 2nd Annual Meeting of the Special Interest Group on Discourse and Dialogue

Volumn , Issue , 2001, Pages

  Paek, Tim ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

Descriptive statistics; Dialog evaluation; Dialog metric; Gold standard; Wizard of oz

Indexed keywords

COMPONENT ANALYSIS; DESCRIPTIVE STATISTICS; DIALOG METRIC; DIALOGUE EVALUATION; DIALOGUE SYSTEMS; EMPIRICAL METHOD; EVALUATING SYSTEMS; GOLD STANDARDS; PERFORMANCE; WIZARD OF OZ;

EID: 85119329181     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. Bersen, N. O., Dybkjaer, H. & Dybkjaer, L. (1998). Designing interactive speech systems: From first ideas to user testing. Springer-Verlag.
2. Clark, H.H. (1996). Using language. Cambridge University Press.
3. Clark, H.H. & Brennan, S.A. (1991). Grounding in communication. In Perspectives on Socially Shared Cognition, APA Books, pp.127-149.
4. Clark, H.H. & Schaefer, E.F. (1987). Collaborating on contributions to conversations. Language and Cognitive Processes, 2/1, pp.19-41.
5. Clark, H.H. & Schaefer, E.F. (1989). Contributing to discourse. Cognitive Science, 13, pp.259-294.
6. Danieli, M. & Gerbino, E. (1995). Metrics for evaluating dialogue strategies in a spoken language system. In Proc. of AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pp. 34-39.
7. Eckert, W., Levin, E. & Pieraccini, R. (1998). Automatic evaluation of spoken dialogue systems. In TWLT13: Formal semantics and pragmatics of dialogue, pp. 99-110.
8. Gibbon, D., Moore, R. & Winski, R. (Eds.) (1998). Handbook of standards and resources for spoken language systems. Spoken Language System Assessment, 3, Walter de Gruyter, Berlin.
9. Glass, J., Polifroni, J., Seneff, S. & Zue, V. (2000). Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In Proc. of ICSLP.
10. Horvitz, E. & Paek, T. (1999). A computational architecture for conversation. In Proc. of 7th International Conference on User Modeling, Springer Wien, pp. 201-210.
11. Kamm, C., Walker, M.A. & Litman, D. (1999). Evaluating spoken language systems. In Proc. of AVIOS.
12. Lamel, L., Rosset S. & Gauvain, J.L. (2000). Considerations in the design and evaluation of spoken language dialog systems. In Proc. of ICSLP.
13. Litman, D. & Pan, S. (1999). Empirically evaluating an adaptable spoken dialogue system. In Proc. of 7th International Conference on User Modeling, Springer Wien, pp. 55-64.
14. Paek, T. & Horvitz, E. (2000). Conversation as action under uncertainty. In Proc. of 16th UAI, Morgan Kaufmann, pp. 455-464.
15. Paek, T. & Horvitz, E. (1999). Uncertainty, utility, and misunderstanding. In Proc. of AAAI Fall Symposium on Psychological Models of Communication, pp. 85-92.
16. Rudnicky, A. (2000). Understanding system performance in dialog systems. MSR Invited Talk.
17. Walker, M.A., Litman, D., Kamm, C. & Abella, A. (1997). PARADISE: A framework for evaluating spoken dialogue agents. In Proc. of 35th ACL.