Developing and testing general models of spoken dialogue system performance

2nd International Conference on Language Resources and Evaluation, LREC 2000

Volumn , Issue , 2000, Pages

  Walker, Marilyn ^a   Kamm, Candace ^a   Boland, Julie ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EXPERIMENTAL CONDITIONS; PERFORMANCE MODEL; PREDICTIVE MODELS; PREDICTIVE POWER; SPOKEN DIALOGUE SYSTEM; SPOKEN LANGUAGE DIALOGUE SYSTEM; TREE-BASED MODEL; USER SATISFACTION;

SPEECH PROCESSING;

EID: 85009141263     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Baggia, Paolo, Giuseppe Castagneri, and Morena Danieli, 1998. Field trials of the italian arise train timetable system. In Interactive Voice Technology for Telecommunications Applications, IVTTA.
2. Bernsen, Niels Ole, Hans Dybkjaer, and Laila Dybkjaer, 1996. Principles for the design of cooperative spoken human-machine dialogue. In International Conference on Spoken Language Processing, ICSLP 96.
3. Brieman, Leo, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone, 1984. Classification and Regression Trees. Wadsworth and Brooks, Monterey California.
4. Danieli, M. and E. Gerbino, 1995. Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation.
5. Hirschman, L., M. Bates, D. Dahl, W. Fisher, J. Garofolo, D. Pallett, K. Hunicke-Smith, P. Price, A. Rudnicky, and E. Tzoukermann, 1993. Multi-site data collection and evaluation in spoken language understanding. In Proceedings of the Human Language Technology Workshop.
6. Hirschman, L. and C. Pao, 1993. The cost of errors in a spoken language system. In Proceedings of the Third European Conference on Speech Communication and Technology.
7. Hirschman, Lynette, Deborah A. Dahl, Donald P. McKay, Lewis M. Norton, and Marcia C. Linebarger, 1990. Beyond class A: A proposal for automatic evaluation of discourse. In Proceedings of the Speech and Natural Language Workshop.
8. Jack, M. A., J. C. Foster, and F. W. Stentiford, 1992. Intelligent dialogues in automated telephone services. In International Conference on Spoken Language Processing, ICSLP.
9. Kamm, Candace, Diane Litman, and Marilyn A. Walker, 1998. From novice to expert: The effect of tutorials on user expertise with spoken dialogue systems. In Proceedings of the International Conference on Spoken Language Processing, ICSLP98.
10. Larsen, Lars Bo, 1999. Combiningobjective and subjective data in evaluation of spoken dialogues. In ESCA Workshop on Interactive Dialogue in Multi-Modal Systems.
11. Litman, Diane J., Shimei Pan, and Marilyn A. Walker, 1998. Evaluating Response Strategies in a Web-Based Spoken Dialogue Agent. In Proceedings of ACL/COLING 98:36th Annual Meeting of the Association of Computational Linguistics.
12. Litman, Diane J., Marilyn A. Walker, and Michael J. Kearns, 1999. Automatic detection of poor speech recognition at the dialogue level. In Proceedings of the 37th Annual Meeting of the Association of Computational Linguistics, ACL99.
13. Love, Stephen, R. T. Dutton, J. C. Foster, M. A. Jack, and F. W. M. Stentiford, 1994. Identifying salient usability attributes for automated telephone services. In International Conference on Spoken Language Processing, IC-SLP.
14. Pallett, David S., 1985. Performance assessment of automatic speech recognizers. J. Res. Natl. Bureau of Standards, 90:371-387.
15. Polifroni, J., S. Seneff, J. Glass, and T. Hazen, 1998. Evaluation methodology for a telephone-based conversational system. In Proc. First International Conference on Language Resources and Evaluation, Granada, Spain.
16. Ralston, J. V., D. B. Pisoni, and John W. Mullennix, 1995. Perception and comprehension of speech. In Syrdal, Bennet, and Greenspan (eds.), Applied Speech Technology. CRC Press, pages 233-287.
17. Rudnicky, A. I., 1993. Factors affecting choice of speech over keyboard and mouse in a simple data-retrieval task. In EUROSPEECH93.
18. Sanderman, Angielen, Janienke Sturm, Els den Os, Lou Boves, and Anita Cremers, 1998. Evaluation of the dutchtrain timetable information system developed in the arise project. In Interactive Voice Technology for Telecommunications Applications, IVTTA.
19. Shriberg, Elizabeth, Elizabeth Wade, and Patti Price, 1992. Human-machine problem solving using spoken language systems (SLS): Factors affecting performance and user satisfaction. In Proceedings of the DARPA Speech and NL Workshop.
20. Sparck-Jones, Karen and Julia R. Galliers, 1996. Evaluating Natural Language Processing Systems. Springer.
21. Walker, M., J. Fromer, G. Di Fabbrizio, C. Mestel, and D. Hindle, 1998a. What can I say: Evaluating a spoken language interface to email. In Proceedings of the Conference on Computer Human Interaction (CHI 98).
22. Walker, M. A., D. Litman, C. A. Kamm, and A. Abella, 1997. PARADISE: A general framework for evaluating spoken dialogue agents. In Proceedings of the 35th Annual Meeting of the Association of Computational Linguistics, ACL/EACL 97.
23. Walker, Marilyn, Julie Boland, and Candace Kamm, 1999. The utility of elapsed time as a usability metric for spoken dialogue systems. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding, AS-RUU99.
24. Walker, Marilyn A., 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research.
25. Walker, Marilyn A., Jeanne C. Fromer, and Shrikanth Narayanan, 1998b. Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email. In Proceedings of the 36th Annual Meeting of the Association of Computational Linguistics, COLING/ACL 98.
26. Walker, Marilyn A., Lynette Hirschman, and John Aberdeen, 2000a. Evaluation for DARPA communicator spoken dialogue systems. In Proc. Second International Conference on Language Resources and Evaluation, Athens, Greece.
27. Walker, Marilyn A., Candace A. Kamm, and Diane J. Litman, 2000b. Towards developing general models of usability with PARADISE. Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems.