On information retrieval metrics designed for evaluation with incomplete relevance assessments

Information Retrieval

Volumn 11, Issue 5, 2008, Pages 447-470

  Sakai, Tetsuya ^a   Kando, Noriko ^b  

^a NewsWatch Inc   (Japan)

^b NATIONAL INSTITUTE OF INFORMATICS   (Japan)

Author keywords

Evaluation metrics; Incompleteness; Relevance assessments; Test collections

Indexed keywords

EID: 50849122035     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-008-9059-7     Document Type: Article

References (38)

1. Aslam, J. A., & Savell, R. (2003). On the effectiveness of evaluating retrieval systems in the absence of relevance judgments. In ACM SIGIR 2003 Proceedings (pp. 361-362).
2. Aslam, J. A., Pavlu, V., & Yilmaz, E. (2006). A statistical method for system evaluation using incomplete judgments. In ACM SIGIR 2006 Proceedings (pp. 541-548).
3. Bompada, T., Chang, C.-C., Chen, J., Kumar, R., & Shenoy, R. (2007). On the robustness of relevance measures with incomplete judgments. In ACM SIGIR 2007 Proceedings (pp. 359-366).
4. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. In ACM SIGIR 2004 Proceedings (pp. 25-32).
5. Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005). Learning to rank using gradient descent. In ACM ICML 2005 Proceedings (pp. 89-96).
6. Büttcher, S., Clarke, C. L. A., Yeung, P. C. K., & Soboroff, I. (2007). Reliable information retrieval evaluation with incomplete and biased judgements. In ACM SIGIR 2007 Proceedings (pp. 63-70).
7. Carterette, B., Allan, J., & Sitaraman, R. (2006). Minimal test collections for retrieval evaluation. In ACM SIGIR 2006 Proceedings (pp. 268-275).
8. Cleverdon, C. W. (1967). The cranfield tests on index language devices. In Aslib Proceedings (Vol. 19, pp. 173-192).
9. Cormack, G. V., Palmer, C. R., & Clarke, C. L. A. (1998). Efficient construction of large test collections. In ACM SIGIR '98 Proceedings (pp. 282-289).
10. De Beer, J., & Moens, M.-F. (2006). Rpref-a generalization of bpref towards graded relevance judgments. In ACM SIGIR 2006 Proceedings (pp. 637-638).
11. Efron, B., & Tibshirani, R. (1993). Introduction to the bootstrap. Chapman & Hall/CRC.
12. Grönqvist, L. (2005). Evaluating latent semantic vector models with synonym tests and document retrieval. In ELECTRA Workshop-Methodologies and Evaluating of Lexical Cohesion Techniques in Real-World Applications (pp. 86-88).
13. Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422-446.
14. Kando, N. (2007). Overview of the Sixth NTCIR Workshop. In NTCIR-6 Proceedings (pp. i-ix).
15. Kekäläinen, J. (2005). Binary and graded relevance in IR evaluations-comparison of the effects on ranking of IR systems. Information Processing and Management, 41, 1019-1033.
16. Moffat, A., Webber, W., & Zobel, J. (2007). Strategic system comparisons via targeted relevance judgments. In ACM SIGIR 2007 Proceedings (pp. 375-382).
17. Moffat, A., & Zobel, J. (2008). Rank-biased precision for measurement of retrieval effectiveness (under review).
18. Sakai, T. (2006a). Bootstrap-based comparisons of IR metrics for finding one relevant document. In AIRS 2006: Lecture Notes in Computer Science 4182 (pp. 374-389). Springer-Verlag.
19. Sakai, T. (2006b). Evaluating evaluation metrics based on the bootstrap. In ACM SIGIR 2006 Proceedings (pp. 525-532).
20. Sakai, T. (2006c). On the task of finding one highly relevant document with high precision. Information Processing Society of Japan Transactions on Databases, 47(SIG4(TOD29)), 13-27
21. Sakai, T. (2007a). Alternatives to Bpref. In ACM SIGIR 2007 Proceedings (pp. 71-78).
22. Sakai, T. (2007b). Evaluating information retrieval metrics based on bootstrap hypothesis tests. Information Processing Society of Japan Transactions on Databases 48(SIG9(TOD35)), 11-28
23. Sakai, T. (2007c). evaluating the task of finding one relevant document using incomplete relevance data. In Forum on Information Technology 2007 Information Technology Letters (pp. 91-94).
24. Sakai, T. (2007d). On penalising late arrival of relevant documents in information retrieval evaluation with graded relevance. In Proceedings of the First International Workshop on Evaluating Information Acess (EVIA 2007) (pp. 32-43).
25. Sakai, T. (2007e). On the properties of evaluation metrics for finding one highly relevant document. Information Processing Society of Japan Transactions on Databases, 48(SIG9(TOD35)), 29-46.
26. Sakai, T. (2007f). On the reliability of information retrieval metrics based on graded relevance. Information Processing and Management, 43(2), 531-548.
27. Sakai, T. (2008a). Comparing metrics across TREC and NTCIR: The robustness to pool depth bias (under review).
28. Sakai, T. (2008b). Comparing metrics across TREC and NTCIR: The robustness to system bias (under review).
29. Sakai, T., & Sparck Jones, K. (2001). Generic summaries for indexing in information retrieval. In ACM SIGIR 2001 Proceedings (pp. 190-198).
30. Soboroff, I., Nicholas, C., & Cahan, P. (2001). Ranking retrieval systems without relevance judgments. In ACM SIGIR 2001 Proceedings (pp. 66-73).
31. Sormunen, E. (2002). Liberal relevance criteria of TREC-counting on negligible documents? In ACM SIGIR 2002 Proceedings (pp. 324-330).
32. Voorhees, E. M. (2001). Evaluation by highly relevant documents. In ACM SIGIR 2001 Proceedings (pp. 74-82).
33. Voorhees, E. M. (2002). The effect of topic set size on retrieval experiment error. In ACM SIGIR 2002 Proceedings (pp. 316-323).
34. Voorhees, E. M. (2004). Overview of the TREC 2003 robust retrieval Track. In TREC 2003 Proceedings.
35. Voorhees, E. M. (2005). Overview of the TREC 2004 robust retrieval track. In TREC 2004 proceedings.
36. Voorhees, E. M., & Buckley, C. (2002). The philosophy of information retrieval evaluation. In Proceedings of CLEF 2001, Lecture Notes in Computer Science 2406 (pp. 355-370).
37. Yilmaz, E., & Aslam, J. A. (2006). Estimating average precision with incomplete and imperfect judgments. In ACM CIKM 2006 Proceedings (pp. 102-111).
38. Zobel, J. (1998). How Reliable are the results of large-scale information retrieval experiments? In ACM SIGIR '98 Proceedings (pp. 307-314).