Evaluating language understanding accuracy with respect to objective outcomes in a dialogue system

EACL 2012 - 13th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings

Volumn , Issue , 2012, Pages 471-481

  Dzikovska, Myroslava O ^a   Bell, Peter ^a   Isard, Amy ^a   Moore, Johanna D ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; LINEAR REGRESSION; LINGUISTICS; SPEECH PROCESSING;

CONFUSION MATRICES; DIALOGUE SYSTEMS; EVALUATION METRICS; LANGUAGE UNDERSTANDING; MULTIPLE LINEAR REGRESSIONS; PREDICTIVE MODELS; PREDICTIVE PERFORMANCE; STANDARD EVALUATIONS;

SPEECH RECOGNITION;

EID: 84952648259     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Kate Acomb, Jonathan Bloom, Krishna Dayanidhi, Phillip Hunter, Peter Krogh, Esther Levin, and Roberto Pieraccini. 2007. Technical support dialog systems: Issues, problems, and solutions. In Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, pages 25-31, Rochester, NY, April.
2. Gwendolyn C. Campbell, Natalie B. Steinhauser, Myroslava O. Dzikovska, Johanna D. Moore, Charles B. Callaway, and Elaine Farrow. 2009. The DeMAND coding scheme: A "common language" for representing and analyzing student discourse. In Proceedings of 14th International Conference on Artificial Intelligence in Education (AIED), poster session, Brighton, UK, July.
3. Myroslava O. Dzikovska, Charles B. Callaway, Elaine Farrow, Johanna D. Moore, Natalie B. Steinhauser, and Gwendolyn E. Campbell. 2009. Dealing with interpretation errors in tutorial dialogue. In Proceedings of the SIGDIAL 2009 Conference, pages 38-45, London, UK, September.
4. Myroslava Dzikovska, Diana Bental, Johanna D. Moore, Natalie B. Steinhauser, Gwendolyn E. Campbell, Elaine Farrow, and Charles B. Callaway. 2010a. Intelligent tutoring with natural language support in the Beetle II system. In Sustaining TEL: From Innovation to Learning and Practice - 5th European Conference on Technology Enhanced Learning, (EC-TEL 2010), Barcelona, Spain, October.
5. Myroslava O. Dzikovska, Johanna D. Moore, Natalie Steinhauser, Gwendolyn Campbell, Elaine Farrow, and Charles B. Callaway. 2010b. Beetle II: A system for tutoring and computational linguistics experimentation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL-2010) demo session, Uppsala, Sweden, July.
6. Kate Forbes-Riley and Diane J. Litman. 2006. Modelling user satisfaction and student learning in a spoken dialogue tutoring system with generic, tutoring, and user affect parameters. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL'06), pages 264-271, Stroudsburg, PA, USA.
7. Kate Forbes-Riley, Diane Litman, Amruta Purandare, Mihai Rotaru, and Joel Tetreault. 2007. Comparing linguistic features for modeling learning in computer tutoring. In Proceedings of the 2007 conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, pages 270-277, Amsterdam, The Netherlands. IOS Press.
8. Narendra K. Gupta, Gökhan Tür, Dilek Hakkani-Tür, Srinivas Bangalore, Giuseppe Riccardi, and Mazin Gilbert. 2006. The AT&T spoken language understanding system. IEEE Transactions on Audio, Speech & Language Processing, 14(1):213-222.
9. Pamela W. Jordan, Maxim Makatchev, and Umarani Pappuswamy. 2006. Understanding complex natural language explanations in tutorial applications. In Proceedings of the Third Workshop on Scalable Natural Language Understanding, ScaNaLU'06, pages 17-24.
10. Lars Bo Larsen. 2003. Issues in the evaluation of spoken dialogue systems using objective and subjective measures. In Proceedings of the 2003 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 209-214.
11. David D. Lewis. 1991. Evaluating text categorization. In Proceedings of the workshop on Speech and Natural Language, HLT'91, pages 312-318, Stroudsburg, PA, USA.
12. Diane Litman, Johanna Moore, Myroslava Dzikovska, and Elaine Farrow. 2009. Using natural language processing to analyze tutorial dialogue corpora across domains and modalities. In Proceedings of 14th International Conference on Artificial Intelligence in Education (AIED), Brighton, UK, July.
13. Yusuke Miyao, Rune Sætre, Kenji Sagae, Takuya Matsuzaki, and Jun'ichi Tsujii. 2008. Task-oriented evaluation of syntactic parsers and their representations. In Proceedings of ACL-08: HLT, pages 46- 54, Columbus, Ohio, June.
14. Sebastian Möller, Paula Smeele, Heleen Boland, and Jan Krebber. 2007. Evaluating spoken dialogue systems according to de-facto standards: A case study. Computer Speech & Language, 21(1):26 - 53.
15. Sebastian Möller, Klaus-Peter Engelbrecht, and Robert Schleicher. 2008. Predicting the quality and usability of spoken dialogue services. Speech Communication, pages 730-744.
16. Rodney D. Nielsen, Wayne Ward, and James H. Martin. 2008. Learning to assess low-level conceptual understanding. In Proceedings 21st International FLAIRS Conference, Coconut Grove, Florida, May.
17. Mihai Rotaru and Diane J. Litman. 2006. Exploiting discourse structure for spoken dialogue performance analysis. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP'06, pages 85-93, Stroudsburg, PA, USA.
18. Mark Sammons, V.G.Vinod Vydiswaran, and Dan Roth. 2010. "Ask not what textual entailment can do for you...". In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1199-1208, Uppsala, Sweden, July.
19. Marilyn A. Walker, Candace A. Kamm, and Diane J. Litman. 2000. Towards Developing General Models of Usability with PARADISE. Natural Language Engineering, 6(3).
20. Deniz Yuret, Aydin Han, and Zehra Turgut. 2010. SemEval-2010 task 12: Parser evaluation using textual entailments. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 51-56, Uppsala, Sweden, July.