A comparison of the optimality of statistical significance tests for information retrieval evaluation

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 925-928

  Urbano, Julián ^a   Marrero, Mónica ^a   Martín, Diego ^b  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

^b UNIVERSIDAD POLITÉCNICA DE MADRID   (Spain)

Author keywords

Bootstrap; Evaluation; Permutation; Randomization; Sign test; Statistical significance; Student's t test; Wilcoxon test

Indexed keywords

BOOTSTRAP; EVALUATION; PERMUTATION; RANDOMIZATION; SIGN TEST; STATISTICAL SIGNIFICANCE; STUDENT'S T TESTS; WILCOXON TEST;

INFORMATION RETRIEVAL; OPTIMIZATION; RESEARCH;

TESTING;

EID: 84883083423     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484163     Document Type: Conference Paper

References (7)

1. G. V. Cormack and T. R. Lynam. Validity and Power of t-test for Comparing MAP and GMAP. In ACM SIGIR, pages 753-754, 2007.
2. T. Sakai. Evaluating Evaluation Metrics Based on the Bootstrap. In ACM SIGIR, pages 525-532, 2006.
3. M. Sanderson and J. Zobel. Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability. In ACM SIGIR, pages 162-169, 2005.
4. M. D. Smucker, J. Allan, and B. Carterette. A Comparison of Statistical Significance Tests for Information Retrieval Evaluation. In ACM CIKM, pages 623-632, 2007.
5. M. D. Smucker, J. Allan, and B. Carterette. Agreement Among Statistical Significance Tests for Information Retrieval Evaluation at Varying Sample Sizes. In ACM SIGIR, pages 630-631, 2009.
6. E. M. Voorhees. Topic Set Size Redux. In ACM SIGIR, pages 806-807, 2009.
7. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In ACM SIGIR, pages 307-314, 1998.