Comparing metrics across TREC and NTCIR: The robustness to pool depth bias

ACM SIGIR 2008 - 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Proceedings

Volumn , Issue , 2008, Pages 691-692

  Sakai, Tetsuya ^a  

^a NewsWatch Inc

Author keywords

Evaluation metrics; Graded relevance; Test collection

Indexed keywords

EID: 57349141449     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390334.1390454     Document Type: Conference Paper

References (8)

1. Buckley, C. and Voorhees, E. M.: Retrieval Evaluation with Incomplete Information, ACM SIGIR 2004 Proceedings, pp. 25-32, 2004.
2. Järvelin, K. and Kekäläinen, J.: Cumulated Gain-Based Evaluation of IR Techniques, ACM TOIS, Vol. 20, No. 4, pp. 422-446, 2002.
3. Moffat, A. and Zobel, J.: Rank-Biased Precision for Measurement of Retrieval Effectiveness, ACM TOIS, under review, 2008.
4. Sakai, T.: Evaluating Evaluation Metrics based on the Bootstrap, ACM SIGIR 2006 Proceedings, pp. 525-532, 2006.
5. Sakai, T.: Comparing Metrics across TREC and NTCIR: The Robustness to System Bias, Information Processing Society of Japan SIG Technical Reports, 2008-FI-91, to appear, 2008.
6. Sakai, T. and Kando, N.: On Information Retrieval Metrics Designed for Evaluation with Incomplete Relevance Assessments, Information Retrieval, 2008. http ://www.springerlink.com/content/k41j115214032614/fulltext.pdf (open access)
7. Yilmaz, E. and Aslam, J. A.: Estimating Average Precision with Incomplete and Imperfect Judgments, CIKM 2006 Proceedings, 2006.
8. Zobel, J.: How Reliable are the Results of Large-Scale Information Retrieval Experiments? ACM SIGIR '98 Proceedings, pp. 307-314, 1998.