Paradise: A framework for evaluating spoken dialogue agents

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn 1997-July, Issue , 1997, Pages 271-280

  Walker, Marilyn A ^a   Litman, Diane J ^a   Kamm, Candace A ^a   Abella, Alicia ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIALOGUE STRATEGY; DIALOGUE SYSTEMS; PERFORMANCE; RELATIVE CONTRIBUTION; SPOKEN DIALOGUE; SYSTEM EVALUATION; TASK COMPLEXITY;

SPEECH PROCESSING;

EID: 85065183198     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Abella, Alicia, Michael K Brown, and Bruce Buntschuh. 1996. Development principles for dialog-based interfaces. In ECAI-96 Spoken Dialog Processing Workshop, Budapest, Hungary.
2. Bates, Madeleine and Damaris Ayuso. 1993. A proposal for incremental dialogue evaluation. In Proceedingsof theDARPASpeechandNL Workshop,pages 319-322.
3. Carberry, S. 1989. Plan recognition and its use in understanding dialogue. In A. Kobsa and W. Wahlster, editors, User Models in Dialogue Systems. Springer Verlag, Berlin, pages 133-162.
4. Carletta, Jean C. 1996. Assessing the reliability of subjective codings. ComputationalLinguistics, 22(2):249-254.
5. Chu-Carrol, Jennifer and Sandra Carberry. 1995. Response generation in collaborative negotiation. In Proceedings of the Conference of the 33rdAnnual Meeting of the Association for Computational Linguistics, pages 136-143.
6. Cohen, Paul. R. 1995. Empirical Methodsfor Artificial Intelligence. MIT Press, Boston.
7. Danieli, M., W. Eckert, N. Fraser, N. Gilbert, M. Guyomard, P. Heisterkamp, M. Kharoune, J. Magadur, S. McGlashan, D. Sadek, J. Siroux, and N. Youd. 1992. Dialogue manager design evaluation. Technical Report Project Esprit 2218 SUNDIAL, WP6000-D3.
8. Danieli, Morena and Elisabetta Gerbino. 1995. Metrics for evaluating dialogue strategies in a spoken language system. In Proceedingsofthe 1995AAAISpring Symposium on Empirical Methods in Discourse Interpretation and Generation, pages 34-39.
9. Doyle, Jon. 1992. Rationality and its roles in reasoning. ComputationalIntelligence, 8(2):376-409.
10. Fraser, Norman M. 1995. Quality standards for spoken dialogue systems: a report on progress in EAGLES. In ESCA Workshop on Spoken Dialogue Systems Vigso, Denmark, pages 157-160.
11. Gale, William, Ken W. Church, and David Yarowsky. 1992. Estimating upper and lower bounds on the performance of word-sense disambiguation programs. In Proc. of3OthACL, pages 249-256, Newark, Delaware.
12. Grosz, Barbara J. and Candace L. Sidner. 1986. Attentions, intentions and the structure of discourse. Computational Linguistics, 12:175-204.
13. Hirschberg, Julia and Christine Nakatani. 1996. A prosodic analysis of discourse segments in direction-giving monologues. In 34th Annual Meeting of the Associationfor ComputationalLinguistics, pages 286-293.
14. Hirschman, Lynette, Deborah A. Dahl, Donald P.McKay, Lewis M. Norton, and Marcia C. Linebarger. 1990. Beyond class A: A proposal for automatic evaluation of discourse. In ProceedingsoftheSpeechandNatural Language Workshop,pages 109-113.
15. Hirschman, Lynette and Christine Pao. 1993. The cost of errors in a spoken language system. In Proceedings ofthe ThirdEuropean Conferenceon Speech Communication and Technology,pages 1419-1422.
16. Joshi, Aravind K., Bonnie L. Webber, and Ralph M. Weischedel. 1984. Preventing false inferences. In COLING84: Proc. lOthInternational Conference on ComputationalLinguistics., pages 134-138.
17. Kamm, Candace. 1995. User interfaces for voice applications. In David Roe and Jay Wilpon, editors, Voice Communication between Humans and Machines. National Academy Press, pages 422-442.
18. Keeney, Ralph and Howard Raiffa. 1976. Decisions with Multiple Objectives: Preferencesand ValueTradeoffs. John Wiley and Sons.
19. Krippendorf, Klaus. 1980. ContentAnalysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, Ca.
20. Litman, Diane and James Allen. 1990. Recognizing and relating discourse intentions and task-oriented plans. In Philip Cohen, Jerry Morgan, and Martha Pollack, editors, Intentions in Communication. MIT Press.
21. Passonneau, Rebecca J. and Diane Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23(1).
22. Polifroni, Joseph, Lynette Hirschman, Stephanie Seneff, and Victor Zue. 1992. Experiments in evaluating interactive spoken language systems. In Proceedings of theDARPASpeechandNL Workshop,pages 28-33.
23. Pollack, Martha, Julia Hirschberg, and Bonnie Webber. 1982. User participation in the reasoning process of expert systems. In ProceedingsFirstNational Conference on Artificial Intelligence, pages pp. 358-361.
24. Shriberg, Elizabeth, Elizabeth Wade, and Patti Price. 1992. Human-machine problem solving using spoken language systems (SLS): Factors affecting performance and user satisfaction. In Proceedings of the DARPASpeech and NL Workshop,pages 49-54.
25. Siegel, Sidney and N. J. Castellan. 1988. Nonparametric Statisticsfor the BehavioralSciences. McGraw Hill.
26. Simpson, A. and N. A. Fraser. 1993. Black box and glass box evaluation of the SUNDIAL system. In Proceedings of the ThirdEuropean Conference on Speech Communication and Technology, pages 1423-1426.
27. Smith, Ronnie W. and Steven A. Gordon. 1997. Effects of variable initiative on linguistic behavior in human-computer spoken natural language dialog. ComputationalLinguistics, 23(1).
28. Sparck-Jones, Karen and Julia R. Galliers. 1996. EvaluatingNaturalLanguageProcessingSystems. Springer.
29. Walker, Marilyn A. 1996. The Effect of Resource Limits and Task Complexity on Collaborative Planning in Dialogue. Artificial Intelligence Journal, 85(1-2): 181-243.
30. Webber, Bonnie and Aravind Joshi. 1982. Taking the initiative in natural language database interaction: Justifying why. In Coling 82, pages 413-419.