How to evaluate your question answering system every day... and still get real work done

2nd International Conference on Language Resources and Evaluation, LREC 2000

Volumn , Issue , 2000, Pages

  Breck, Eric J ^a   Burger, John D ^a   Ferro, Lisa ^a   Hirschman, Lynette ^a   House, David ^a   Light, Marc ^a   Mani, Inderjeet ^a  

^a MITRE CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATED EVALUATION SYSTEMS; HUMAN ASSESSORS; KENDALL'S TAU; QUESTION ANSWERING; QUESTION ANSWERING SYSTEMS; QUESTION ANSWERING TASK;

ARTIFICIAL INTELLIGENCE;

EID: 84900049164     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. Black, E., Abney, S., Flickenger, D., Gdaniec, C., Grishman, R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klavans, J., Liberman, M., Marcus, M., Roukous, S., Santorini, B., Strzalkowski, T. (1991). A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars. In Proceedings of the Fourth DARPA Speech and Natural Language Workshop, pp. 306-311.
2. Breck, E. Burger, J. D. Ferro, L. House, D. Light, M., Mani, I. (1999). A sys called Qanda. In (Voorhees & Harman, 1999).
3. Burstein, J., Kukich, K., Wolff, S., Lu, C., and Chodorow, M. (1998). Computer Analysis of Essays. NCME Symposium on Automated Scoring, April 1998.
4. Chaudri, V. and Fikes, R. co-chairs. (1999). Proceedings of the 1999 AAAI Fall Symposium on Question Answering Systems. North Falmouth, MA: AAAI Technical Report FS-99-02.
5. Egan, J. P. (1975). Signal detection theory and ROC analysis. Academic Press.
6. Goodman, J. (1996). Parsing Algorithms and Metrics. In Proceedings of the 34th Annual Meeting of the ACL. Santa Cruz, CA, June 1996.
7. Green, C. (1969). The Application of Theorem Proving to Question-Answering Systems. Ph. D. Thesis. Stanford University.
8. Green, D. M. and Swets, J. A. (1966). Signal detection theory and Psychophysics. John Wiley and Sons, Inc.
9. Harman, D. (1994). Overview of the Third Text REtrieval Conference (TREC-3). In Proceedings of the Third Text Retrieval Conference (TREC-3). Gaithersberg, Maryland: NIST Special Publication 500-226.
10. Hirschman, L. (1998). Reading Comprehension: A Grand Challenge for Language Understanding. In Proceedings of the first International Conference on Language Resources and Evaluation.
11. Hirschman, L., Light, M., Breck, E., & Burger, J. D. (1999). Deep Read: A Reading Comprehension System. In Proceedings of the Thirty-seventh Annual Meeting of the Association for Computational Linguistics (pp. 325-332).
12. Martin, J., and Lankester, C. (1999). Ask Me Tomorrow: The NRC and University of Ottawa Question Answering System. In (Voorhees & Harman, 1999).
13. Sparck Jones, K. & van Rijsbergen, C. (1975). Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge.
14. Stuart, A. (1983). Kendall's tau. In Samuel Kotz and Norman L. Johnson, (Eds.), Encyclopedia of Statistical Sciences, volume 4, pp. 367-369. John Wiley & Sons.
15. Voorhees, E. M. and Tice, D. (1999). The TREC-8 Question Answering Track Evaluation. In (Voorhees & Harman, 1999).
16. Voorhees, E. and Harman, D. (1999). Proceedings of the The Eighth Text REtrieval Conference (TREC-8). Gaithersburg, MD: NIST Special Publication.
17. Waltz, D. (1978). An English language questionanswering system for a large relational database. Communications of the ACM, 21, July 1978, pp. 526-539.