Designing a realistic evaluation of an end-to-end interactive question answering system

Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004

Volumn , Issue , 2004, Pages 989-992

  Wacholder, Nina ^a   Small, Sharon ^b   Bai, Bing ^a   Kelly, Diane ^c   Rittman, Robert ^a   Ryan, Sean ^b   Salkin, Robert ^a   Song, Peng ^a   Sun, Ying ^a   Ting, Liu ^b   Kantor, Paul ^a   Strzalkowski, Tomek ^b  

^a RUTGERS UNIVERSITY   (United States)

^b STATE UNIVERSITY OF NEW YORK   (United States)

^c UNIVERSITY OF NORTH CAROLINA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; NATURAL LANGUAGE PROCESSING SYSTEMS;

COMPLEX QUESTIONS; CROSS EVALUATION; END-TO-END SYSTEMS; PRECISION AND RECALL; QUANTITATIVE TOOL; QUESTION ANSWERING SYSTEMS; REALISTIC EVALUATIONS; STANDARD METRICS;

QUALITY CONTROL;

EID: 84900583939     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (6)

1. Breck, Eric, John Burger, Lisa Ferro, Lynette Hirshman, David House, Marc Light and Inderjeet Manerji (2000) How to evaluate your question answering system every day. and still get real work done. Proceedings of LREC 2000, Athens, Greece.
2. Jones, Karen Sparck (2001). Automatic language and information processing: rethinking evaluation. Natural Language Engineering 7(1):29-46.
3. Small, Sharon, Liu Ting, Nobuyuki Shimuzu and Tomek Strzalkowski (2003) HITIQA, An interactive question answering system: A preliminary report. Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering. June, 2003.
4. Sun, Ying et al. (Under review) Cross Evaluation-a pilot application of a new evaluation mechanism. Submitted to ASIST 2004.
5. Wacholder, Nina et al. (2003) Evaluation of the HITIQA Analyst's Workshops. Report of ARDA AQUAINT 24 month meeting.
6. Voorhees, Ellen (2001) The TREC-9 Question Answering Track. Natural Language Engineering.