An evaluation understudy for dialogue coherence models

ACL-08: HLT - Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue

Volumn , Issue , 2008, Pages 172-181

  Gandhe, Sudeep ^a   Traum, David ^a  

^a University of Southern California   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CO-OCCURRENCE STATISTICS; COHERENCE MODELS; DIALOGUE SYSTEMS; EVALUATION METHODS; HUMAN JUDGMENTS; MACHINE TRANSLATIONS; N-GRAMS; RESEARCH COMMUNITIES;

SPEECH PROCESSING;

EID: 84857764949     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1622064.1622098     Document Type: Conference Paper

References (32)

1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. In To appear in Computational Linguistics.
2. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proc. ACL-05.
3. Regina Barzilay, Noemie Elhadad, and Kathleen McKeown. 2002. Inferring strategies for sentence ordering in multidocument summarization. JAIR, 17:35-55.
4. Harry Bratt, John Dowding, and Kate Hunicke-Smith. 1995. The sri telephone-based atis system. In Proceedings of the Spoken Language Systems Technology Workshop, January.
5. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. In proceedings of EACL-2006.
6. Wieland Eckert, Esther Levin, and Roberto Pieraccini. 1997. User modeling for spoken dialogue system evaluation. In Automatic Speech Recognition and Understanding, pages 80-87, Dec.
7. Sudeep Gandhe and David Traum. 2007. Creating spoken dialogue characters from corpora without annotations. In Proceedings of Interspeech-07.
8. Sudeep Gandhe, Andrew Gordon, and David Traum. 2006. Improving question-answering with linking dialogues. In International Conference on Intelligent User Interfaces (IUI), January.
9. Kalliroi Georgila, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: Learning and evaluation. In proceedings of Interspeech.
10. James Henderson, Oliver Lemon, and Kallirroi Georgila. 2005. Hybrid reinforcement/supervised learning for dialogue policies from communicator data. In proceedings of IJCAI workshop.
11. Kate S. Hone and Robert Graham. 2000. Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems.
12. Daniel Jurafsky and James H. Martin. 2000. SPEECH and LANGUAGE PROCESSING: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice-Hall.
13. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30:81-93.
14. Klaus Krippendorff. 2004. Content Analysis, An Introduction to Its Methodology 2nd Edition. Sage Publications.
15. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, Sapporo, Japan.
16. Mirella Lapata. 2006. Automatic evaluation of information ordering. Computational Linguistics, 32(4):471-484.
17. Esther Levin, Roberto Pieraccini, and Wieland Eckert. 1997. Learning dialogue strategies within the markov decision process framework. In Automatic Speech Recognition and Understanding, pages 72-79, Dec.
18. Chin-Yew Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out.
19. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In In ACL 2003: Proc. of the 41st Annual Meeting of the Association for Computational Linguistics, July.
20. Kishore A. Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In Technical Report RC22176 (W0109-022), IBM Research Division, September.
21. Amruta Purandare and Diane Litman. 2008. Analyzing dialog coherence using transition patterns in lexical and semantic features. In Proceedings 21st International FLAIRS Conference, May.
22. Antonio Roque, Hua Ai, and David Traum. 2006. Evaluation of an information state-based dialogue manager. In Brandial 2006: The 10th Workshop on the Semantics and Pragmatics of Dialogue.
23. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In proceedings of HLT/NAACL, Rochester, NY.
24. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proc. ACL-06.
25. David R. Traum and James F. Allen. 1994. Discourse obligations in dialogue processing. In proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics (ACL-94), pages 1-8.
26. David R. Traum, Susan Robinson, and Jens Stephan. 2004. Evaluation of multi-party virtual reality dialogue interaction. In In Proceedings of Fourth International Conference on Language Resources and Evaluation (LREC), pages 1699-1702.
27. David Traum, William Swartout, Jonathan Gratch, and Stacy Marsella. 2005. Virtual humans for non-team interaction training. In AAMAS-05 Workshop on Creating Bonds with Humanoids, July.
28. M. Walker, C. Kamm, and D. Litman. 2000. Towards developing general models of usability with PARADISE. Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems.
29. Richard Wallace. 2003. Be Your Own Botmaster, 2nd Edition. ALICE A. I. Foundation.
30. Joseph Weizenbaum. 1966. Eliza - a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36-45, January.
31. Jason D. Williams and Steve Young. 2006. Partially observable markov decision processes for spoken dialog systems. Computer Speech and Language, 21:393-422.
32. Jason D. Williams. 2007. A method for evaluating and comparing user simulations: The cramer-von mises divergence. In IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).