Alternatives to Bpref

Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Volumn , Issue , 2007, Pages 71-78

  Sakai, Tetsuya ^a,b  

^a NewsWatch Inc   (Japan)

^b TOSHIBA CORPORATION   (Japan)

Author keywords

Evaluation metrics; Graded relevance; Test collection

Indexed keywords

EVALUATION METRICS; GRADED RELEVANCE; NORMALISED DISCOUNTED CUMULATIVE GAIN; TEST COLLECTIONS;

MATRIX ALGEBRA; PROBLEM SOLVING; ROBUSTNESS (CONTROL SYSTEMS); SYSTEM STABILITY;

INFORMATION RETRIEVAL;

EID: 36448993626     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1277741.1277756     Document Type: Article

References (17)

1. Aslam, J. A. and Savell, R.: On the Effectiveness of Evaluating Retrieval Systems in the Absence of Relevance Judgments, ACM SIGIR. 2003 Proceedings, pp. 361-362, 2003.
2. Aslam, J. A., Pavlu, V. and Yilmaz, E.: A Statistical Method for System Evaluation Using Incomplete Judgments, ACM SIGIR 2006 Proceedings, pp. 541-548, 2006.
3. Buckley, C. and Voorhees, E. M.: Retrieval Evaluation with Incomplete Information, ACM SIGIR 2004 Proceedings, pp. 25-32, 2004.
4. Carterette, B., Allan, J. and Sitaraman, R.: Minimal Test Collections for Retrieval Evaluation, ACM SIGIR 2006 Proceedings, pp. 268-275, 2006.
5. Cormack, G. V., Palmer, C. R. and Clarke, C. L. A. Efficient Construction of Large Test Collections, ACM SIGIR '98 Proceedings, pp. 282-289 (1998).
6. De Beer, J. and Moens, M.-F.: Rpref - A Generalization of Bpref towards Graded Relevance Judgments, ACM SIGIR 2006 Proceedings, pp. 637-638, 2006.
7. Järvelin, K. and Kekäläinen, J.: Cumulated Gain-Based Evaluation of IR, Techniques, ACM Transactions on Information Systems, Vol. 20, No. 4, pp. 422-446, 2002.
8. Kando, N.: Overview of the Fifth NTCIR, Workshop, NTCIR-5 Proceedings, 2005.
9. Sakai, T.: Evaluating Evaluation Metrics based on the Bootstrap, ACM SIGIR 2006 Proceedings, pp. 525-532, 2006.
10. Sakai, T.: On the Reliability of Information Retrieval Metrics based on Graded Relevance, Information Processing and Management, 43(2), pp. 531-548, 2007.
11. Sakai, T.: On Penalising Late Arrival of Relevant Documents in Information Retrieval Evaluation with Graded Relevance, Proceedings of the First International Workshop on Evaluating Information Acess (EVIA 2007), to appear, 2007.
12. Soboroff, I., Nicholas, C. and Cahan, P.: Ranking Retrieval Systems without Relevance Judgments, ACM SIGIR 2001 Proceedings, pp. 66-73, 2001.
13. Soboroff, I.: Dynamic Test Collections: Measuring Search Effectiveness on the Live Web, ACM SIGIR 2006 Proceedings, pp. 276-283, 2006.
14. Turpin, A. and Scholer, F.: User Performance versus Precision Measures for Simple Search Tasks, ACM SIGIR. 2006 Proceedings, pp. 11-18, 2006.
15. Voorhees, E. M. and Buckley, C.: The Effect of Topic Set Size on Retrieval Experiment Error, ACM SIGIR 2002 Proceedings, pp. 316-323, 2002.
16. Yilmaz, E. and Aslam, J. A.: Estimating Average Precision with Incomplete and Imperfect Judgments, CIKM 2006 Proceedings, 2006.
17. Zobel, J.: How Reliable are the Results of Large-Scale Information Retrieval Experiments? ACM SIGIR '98 Proceedings, pp. 307-314, 1998.