On the reliability of factoid question answering evaluation

ACM Transactions on Asian Language Information Processing

Volumn 6, Issue 1, 2007, Pages

  Sakai, Tetsuya ^a  

^a TOSHIBA CORPORATION   (Japan)

Author keywords

Evaluation metrics; Question answering

Indexed keywords

EVALUATION METRICS; QUESTION ANSWERING (QA);

DATA PROCESSING; QUERY LANGUAGES; RELIABILITY THEORY;

COMPUTER SCIENCE;

EID: 34247192084     PISSN: 15300226     EISSN: 15583430     Source Type: Journal    
DOI: 10.1145/1227850.1227853     Document Type: Article

References (23)

1. BUCKLEY, C. AND VOORHEES, E. M. 2000. Evaluating evaluation measure stability. In Proceedings of ACM SIGIR 2000. 33-40.
2. EFRON, B. AND TIBSHIRANI, R. 1993. An Introduction to the Bootstrap. Chapman & Hall/CRC, Boca Raton, FL.
3. FUKUMOTO, J., KATO, T., AND MASUI, F. 2004. Question answering challenge for five ranked answers and list answers - overview of NTCIR4 QAC2 subtask 1 and 2-. In Working Notes of NTCIR-4. 283-290.
4. JÄRVELIN, K. AND KEKÄLÄINEN, J. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20, 4, 422-446.
5. KATO, T., MASUI, F., FUKUMOTO, J., AND KANDO, N. 2004. Characterization of list-type question answering and its evaluation measures (in Japanese). In Information Processing Society of Japan SIG Technical Reports FI-76-16 / NL-163-16. 115-122.
6. MAGNINI, B., VALLIN, A., AYACHE, C., ERBACH, G., PEÑAS, A., DE RIJKE, M., ROCHA, P., SIMOV, K., AND SUTCLIFFE, R. 2004. Overview of the CLEF 2004 multilingual question answering track.
7. SAKAI, T. 2004a. New performance metrics based on multigrade relevance: Their application to question answering. In Proceedings of NTCIR-4.
8. SAKAI, T. 2004b. A note on the reliability of Japanese question answering evaluation. In Information Processing Society of Japan SIG Technical Reports FI-77-7. 57-64.
9. SAKAI, T. 2005. The effect of topics sampling in sensitivity comparisons of information retrieval metrics. In NTCIR-5 Proceedings. 505-512.
10. SAKAI, T. 2006a. Bootstrap-based comparisons of IR metrics for finding one relevant document. In Proceedings of Asia Information Retrieval Symposium (AIRS) 2006, Lecture Notes in Computer Science 4182. 374-389.
11. SAKAI, T. 2006b. Evaluating evaluation metrics based on the bootstrap. In Proceedings of ACM SIOIR 2006. 525-532.
12. SAKAI, T. 2006c. Give me just one highly relevant document: P-measure. In Proceedings of ACM SIGIR 2006. 695-696.
13. SAKAI, T. 2006d. On the reliability of information retrieval metrics based on graded relevance. Information Processing and Management. 531-548.
14. SAKAI, T. 2006e. On the task of finding one highly relevant document with high precision. Information Processing Society of Japan Digital Courier 2, 174-188.
15. SAKAI, T. 2006f. On the task of finding one highly relevant document with high precision. Information Processing Society of Japan Transactions on Databases TOD29, 13-27.
16. SAKAI, T., SAITO, Y., ICHIMURA, Y., KOYAMA, M., AND KOKUBU, T. 2004a. Toshiba ASKMi at NTCIR-4 QAC2. In Proceedings of NTCIR-4.
17. SAKAI, T., SAITO, Y., ICHIMURA, Y., KOYAMA, M., KOKUBU, T., AND MANABE, T. 2004b. ASKMi: A Japanese question answering system based on semantic role analysis. In Proceedings of RIAO 2004. 215-231.
18. SOBOROFF, I. 2004. On evaluating web search with very few relevant documents. In Proceedings of ACM SIGIR 2004. 530-531.
19. SPARCK JONES, K., WALKER, S., AND ROBERTSON, S. E. 2000. A probabilistic model of information retrieval: development and comparative experiments. Information Processing and Management 36, 779-808 (Part I) and 809-840 (Part II).
20. VOORHEES, E. M. 2002. Overview of the TREC 2001 question answering track. In Proceedings of TREC 2001.
21. VOORHEES, E. M. 2004. Overview of the TREC 2003 question answering track. In Proceedings of TREC 2003.
22. VOORHEES, E. M. 2005. Overview of the TREC 2004 question answering track. In Proceedings of TREC 2004.
23. VOORHEES, E. M. AND BUCKLEY, C. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of ACM SIGIR 2002. 316-323.