On the reliability of information retrieval metrics based on graded relevance

Information Processing and Management

Volumn 43, Issue 2, 2007, Pages 531-548

  Sakai, Tetsuya ^a  

^a TOSHIBA CORPORATION   (Japan)

Author keywords

Cumulative gain; Evaluation; Graded relevance; Q measure; Reliability

Indexed keywords

DATABASE SYSTEMS; Q FACTOR MEASUREMENT; SENSITIVITY ANALYSIS;

AVERAGE PRECISION; CUMULATIVE GAIN; GRADED RELEVANCE; INFORMATION RETRIEVAL METRICS;

INFORMATION RETRIEVAL;

EID: 33750437740     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2006.07.020     Document Type: Article

References (20)

1. Buckley, C., & Voorhees, E. M. (2000). Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2000) (pp. 33-40).
2. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. In Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2004) (pp. 25-32).
3. Chen, K.-H., Chen, H.-H., Kando, N., Kuriyama, K., Lee, S., Myaeng, S.-H., et al. (2003). Overview of CLIR task at the third NTCIR workshop. In Proceedings of the 3rd NTCIR workshop on research in information retrieval, automatic text summarization and question answering (NTCIR-3).
4. Della Mea V., and Mizzaro S. Measuring retrieval effectiveness: a new proposal and a first experimental validation. Journal of the American Society for Information Science and Technology 55 6 (2004) 530-543
5. Hull, D. (1993). Using statistical testing in the evaluation of retrieval experiments. In Proceedings of the 16th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '93) (pp. 329-338).
6. Järvelin K., and Kekäläinen J. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20 4 (2002) 422-446
7. Kekäläinen J. Binary and graded relevance in IR evaluations - comparison of the effects on ranking of IR systems. Information Processing and Management 41 (2005) 1019-1033
8. Korfhage R.R. Information storage and retrieval (1997), Wiley Computer Publishing
9. Sakai, T. (2003). Average gain ratio: a simple retrieval performance measure for evaluation with multiple relevance levels. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2003) (pp. 417-418).
10. Sakai, T. (2004). Ranking the NTCIR systems based on multigrade relevance. In Proceedings of Asia information retrieval symposium 2004 (pp. 170-177).
11. Sakai, T. (2005a). The effect of topic sampling on sensitivity comparisons of information retrieval metrics. In Proceedings of the 5th NTCIR workshop on research in information access technologies (NTCIR-5).
12. Sakai, T. (2005b). The reliability of metrics based on graded relevance. In Proceedings of Asia information retrieval symposium 2005. Lecture notes in computer science: Vol. 3689 (pp. 1-16).
13. Sakai, T. (2006a). Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2006).
14. Sakai, T. (2006b). Give me just one highly relevant document: P-measure. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2006).
15. Sakai, T. (2006c). On the task of finding one highly relevant document with high precision. Information Processing Society of Japan Transactions on Databases 47 SIG4 (TOD29), 13-27.
16. Sanderson, M., & Zobel, J. (2005). Information retrieval system evaluation: effort, sensitivity, and reliability. In Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2005) (pp. 162-169).
17. Voorhees, E. M., & Buckley, C. (2002). The effect of topic set size on retrieval experiment error. In Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2002) (pp. 316-323).
18. Voorhees, E. M. (2001). Evaluation by highly relevant documents. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2001) (pp. 74-82).
19. Voorhees, E. M. (2005). Overview of the TREC 2004 robust retrieval track. In Proceedings of the 13th text retrieval conference (TREC 2004).
20. Vu, H.-T., & Gallinari, P. (2005). On effectiveness measures and relevance functions in ranking INEX systems. In Proceedings of Asia information retrieval symposium 2005. Lecture notes in computer science: Vol. 3689 (pp. 312-327).