Different structures for evaluating answers to complex questions: Pyramids won't topple, and neither will human assessors

ACL 2007 - Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2007, Pages 768-775

  Dang, Hoa Trang ^a   Lin, Jimmy ^b  

^a NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

^b UNIVERSITY OF MARYLAND   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALTERNATIVE METHODS; CLASSIFICATION TASKS; COMPLEX QUESTIONS; DIFFERENT STRUCTURE; HIGHLY-CORRELATED; HUMAN ASSESSORS; LARGE-SCALE DEPLOYMENT; QA TRACKS;

COMPUTATIONAL LINGUISTICS;

EID: 44649166130     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (10)

1. H. Dang, J. Lin, and D. Kelly. 2006. Overview of the TREC 2006 question answering track. In Proc. of TREC 2006.
2. W. Hildebrandt, B. Katz, and J. Lin. 2004. Answering definition questions with multiple knowledge sources. In Proc. HLT/NAACL 2004.
3. D. Lewis. 1991. Evaluating text categorization. In Proc. of the Speech and Natural Language Workshop.
4. J. Lin and D. Demner-Fushman. 2005. Automatically evaluating answers to definition questions. In Proc. of HLT/EMNLP 2005.
5. J. Lin and D. Demner-Fushman. 2006. Will pyramids built of nuggets topple over? In Proc. of HLT/NAACL 2006.
6. E. Sormunen. 2002. Liberal relevance criteria of TREC-counting on negligible documents? In Proc. of SIGIR 2002.
7. E. Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. IP&M, 36(5):697-716.
8. E. Voorhees. 2002. The philosophy of information retrieval evaluation. In Proc. of CLEF Workshop.
9. E. Voorhees. 2003. Overview of the TREC 2003 question answering track. In Proc. of TREC 2003.
10. E. Voorhees. 2005. Using question series to evaluate question answering system effectiveness. In Proc. of HLT/EMNLP 2005.