Evaluating evaluation metrics on the bootstrap

Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn 2006, Issue , 2006, Pages 525-532

  Sakai, Tetsuya ^a  

^a TOSHIBA CORPORATION   (Japan)

Author keywords

Bootstrap; Evaluation; Graded relevance; Test collection

Indexed keywords

BOOTSTRAP; BOOTSTRAP ESTIMATE; GRADED RELEVANCE; TEST COLLECTION;

CORRELATION METHODS; ERROR ANALYSIS; FUNCTION EVALUATION; HEURISTIC METHODS; STATISTICAL MECHANICS; STATISTICAL METHODS;

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS;

EID: 33750340100     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. Asakawa, S. and Selberg, E.: The New MSN Search Engine Developed by Microsoft (in Japanese), Information Processing Society of Japan Magazine, Vol 46, No. 9, pp. 1008-1015, 2005.
2. Buckley, C. and Voorhees, E. M.: Evaluating Evaluation Measure Stability, ACM SIGIR 2000 Proceedings, pp. 33-40, 2000.
3. Efron, B. and Tibshirani, R.: An Introduction to the Bootstrap, Chapman & Hall/CRC, 1993.
4. Hull, D.: Using Statistical Testing in the Evaluation of Retrieval Experiments, ACM SIGIR '93 Proceedings, pp. 329-338, 1993.
5. Järvelin, K. and Kekäläinen, J.: Cumulated Gain-Based Evaluation of IR Techniques, ACM Transactions on Information Systems, Vol. 20, No. 4, pp. 422-446, 2002.
6. Johnson, D. H.: The Insignificance of Statistical Significance Testing, Journal of Wildlife Management Vol. 63, Issue. 3, pp. 763-772, 1999.
7. Kekäläinen, J.: Binary and Graded Relevance in IR evaluations - Comparison of the Effects on Ranking of IR Systems, Information Processing and Management, Vol. 41, pp.1019-1033, 2005.
8. Lin, W.-H. and Hauptmann, A.: Revisiting the Effect of Topic Set Size on Retrieval Error, ACM SIGIR 2005 Proceedings, pp. 637-638, 2005.
9. NTCIR: http://research.nii.ac.jp/ntcir/
10. Sakai, T.: The Effect of Topic Sampling on Sensitivity Comparisons of Information Retrieval Metrics, NTCIR-5 Proceedings, pp. 505-512, 2005.
11. Sakai, T. et al.: Toshiba BRIDJE at NTCIR-5 CLIR: Evaluation using Geometric Means, NTCIR-5 Proceedings, pp. 56-63, 2005.
12. Sakai, T.: On the Reliability of Information Retrieval Metrics based on Graded Relevance, Information Processing and Management, to appear, 2006.
13. Sanderson, M. and Zobel, J.: Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability, ACM SIGIR 2005 Proceedings, pp. 162-169, 2005.
14. Savoy, J.: Statistical Inference in Retrieval Effectiveness Evaluation, Information Processing and Management, Vol. 33, No. 4, pp. 495-512, 1997.
15. Voorhees, E. M. and Buckley, C.: The Effect of Topic Set Size on Retrieval Experiment Error, ACM SIGIR 2002 Proceedings, pp. 316-323, 2002.
16. Voorhees, E. M.: Overview of the TREC 2004 Robust Retrieval Track, TREC 2004 Proceedings, 2005.
17. Vu, H.-T. and Gallinari, P.: On Effectiveness Measures and Relevance Functions in Ranking INEX Systems, AIRS 2005 Proceedings, LNCS 3689, pp. 312-327, 2005.