Comparing metrics across TREC and NTCIR: The robustness to system bias

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2008, Pages 581-590

  Sakai, Tetsuya ^a  

^a NewsWatch Inc   (Japan)

Author keywords

Evaluation metrics; Graded relevance; Test collection

Indexed keywords

EVALUATION METRICS; GRADED RELEVANCE; NEW SYSTEM; PAIRWISE STATISTICAL SIGNIFICANCE; Q-MEASURES; RANDOM SAMPLE; SYSTEM BIAS; TEST COLLECTION; UNBIASED CONDITIONS;

KNOWLEDGE MANAGEMENT;

METRIC SYSTEM;

EID: 70349242289     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1458082.1458159     Document Type: Conference Paper

References (31)

1. Ahlgren, P. and Gröonqvist, L.: Evaluation of Retrieval Effectiveness with Incomplete Relevance Data: Theoretical and Experimental Comparison of Three Measures, Information Processing and Management, Volume 44, pp. 212-225, 2008.
2. Aslam, J. A. and Yilmaz, E.: Inferring Document Relevance from Incomplete Information, ACM CIKM 2007 Proceedings, pp. 633-642, 2007.
3. Baillie, M., Azzopardi, L. and Ruthven, I.: Evaluating Epistemic Uncertainty under Incomplete Assessments, Information Processing and Management, 44(2), pp. 811-837, 2008.
4. Bompada, T. et al.: On the Robustness of Relevance Measures with Incomplete Judgments, ACM SIGIR 2007 Proceedings, pp. 359-366, 2007.
5. Buckley, C. and Voorhees, E. M.: Evaluating Evaluation Measure Stability, ACM SIGIR 2000 Proceedings, pp. 33-40, 2000.
6. Buckley, C. and Voorhees, E. M.: Retrieval Evaluation with Incomplete Information, ACM SIGIR 2004 Proceedings, pp. 25-32, 2004.
7. Buckley, C. et al.: Bias and the Limits of Pooling for Large Collections, Information Retrieval, Vol. 10, Number 6, pp. 491-508, 2007.
8. Burges, C. et al.: Learning to Rank using Gradient Descent, ACM ICML 2005 Proceedings, pp. 89-96, 2005.
9. Büttcher et al.: Reliable Information Retrieval Evaluation with Incomplete and Biased Judgements, ACM SIGIR 2007 Proceedings., pp. 63-70, 2007.
10. Carterette, B.: Robust Test Collections for Retrieval Evaluation, ACM SIGIR 2007 Proceedings, pp. 55-62, 2007.
11. Carterette, B. et al.: Evaluation Over Thousands of Queries, ACM SIGIR 2008 Proceedings, pp. 651-658, 2008.
12. Järvelin, K. and Kekäläinen, J.: Cumulated Gain-Based Evaluation of IR Techniques, ACM TOIS, Vol. 20, No. 4, pp. 422-446, 2002.
13. Kando, N.: Overview of the Sixth NTCIR Workshop, NTCIR-6 Proceedings, pp. i-ix, 2007.
14. Moffat, A. and Zobel, J.: Rank-Biased Precision for Measurement of Retrieval Effectiveness, ACM TOIS, to appear, 2008.
15. Robertson, S.: A New Interpretation of Average Precision, ACM SIGIR 2008 Proceedings, pp. 689-690, 2008.
16. Sakai, T. and Sparck Jones, K.: Generic Summaries for Indexing in Information Retrieval, ACM SIGIR 2001 Proceedings, pp.190-198, 2001.
17. Sakai, T.: Evaluating Evaluation Metrics based on the Bootstrap, ACM SIGIR 2006 Proceedings, pp. 525-532, 2006.
18. Sakai, T.: On the Reliability of Information Retrieval Metrics based on Graded Relevance, Information Processing and Management, 43(2), pp. 531-548, 2007.
19. Sakai, T.: On Penalising Late Arrival of Relevant Documents in Information Retrieval Evaluation with Graded Relevance, Proceedings of the First International Workshop on Evaluating Information Acess (EVIA 2007), pp. 32-43, 2007. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings6/ EVIA/1.pdf
20. Sakai, T.: Alternatives to Bpref, ACM SIGIR 2007 Proceedings, pp. 71-78, 2007.
21. Sakai, T.: Evaluating Information Retrieval Metrics based on Bootstrap Hypothesis Tests, IPSJ Transactions on Databases, Vol.48, No.SIG 9 (TOD35), pp.11-28, 2007. Also available in IPSJ Digital Courier, Vol.3, pp.625-642, 2007. http://www.jstage.jst.go.jp/article/ipsjdc/3/0/625/-pdf
22. Sakai, T. and Kando, N.: On Information Retrieval Metrics Designed for Evaluation with Incomplete Relevance Assessments, Information Retrieval, 2008. http://www.springerlink.com/content/k41j1152140326l4/fulltext.pdf (open access)
23. Sakai, T.: Comparing Metrics across TREC and NTCIR: The Robustness to Pool Depth Bias, ACM SIGIR 2008, pp. 691-692, 2008.
24. Sanderson, M. and Joho, H.: Forming Test Collections with No System Pooling, ACM SIGIR 2004 Proceedings, pp. 33-40, 2004.
25. Sormunen, E.: Liberal Relevance Criteria of TREC - Counting on Negligible Documents? ACM SIGIR 2002 Proceedings, pp. 324-330, 2002.
26. Voorhees, E. M.: The Philosophy of Information Retrieval Evaluation, CLEF 2001 Proceedings, LNCS 2406, pp. 355-370, 2002.
27. Voorhees, E. M.: Overview of the TREC 2003 Robust Retrieval Track, TREC 2003 Proceedings, 2004.
28. Voorhees, E. M.: Overview of the TREC 2004 Robust Retrieval Track, TREC 2004 Proceedings, 2005.
29. Webber, W., Moffat, A., Zobel, J. and Sakai, T.: Precision-At-Ten Considered Redundant, ACM SIGIR 2008 Proceedings, pp. 695-696, 2008.
30. Yilmaz, E. and Aslam, J. A.: Estimating Average Precision with Incomplete and Imperfect Judgments, CIKM 2006 Proceedings, 2006.
31. Zobel, J.: How Reliable are the Results of Large-Scale Information Retrieval Experiments? ACM SIGIR '98 Proceedings, pp. 307-314, 1998.