A comparison of statistical significance tests for information retrieval evaluation

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2007, Pages 623-632

  Smucker, Mark D ^a   Allan, James ^a   Carterette, Ben ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

Bootstrap; Hypothesis test; Permutation; Randomization; Sign; Statistical significance; Student's t test; Wilcoxon

Indexed keywords

BOOTSTRAP; HYPOTHESIS TEST; PERMUTATION; RANDOMIZATION; SIGN; STATISTICAL SIGNIFICANCE; WILCOXON;

INFORMATION RETRIEVAL; INFORMATION SERVICES; KNOWLEDGE MANAGEMENT; STUDENTS;

STATISTICAL TESTS;

EID: 63449088172     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1321440.1321528     Document Type: Conference Paper

References (22)

1. G. E.P.Box,W. G. Hunter, and J.S. Hunter. Statistics for Experimenters. John Wiley & Sons, 1978.
2. J.V.Bradley. Distribution-Free Statistical Tests. Prentice-Hall, 1968.
3. C. Buckley. trec-eval. http://trec.nist.gov/trec-eval/trec-eval.8.0.tar. gz.
4. P.R.Cohen. Empirical methods for artificial intelligence. MIT Press, 1995.
5. G. Cormack and T. Lynam. Validity and power of t-test for comparing map and gmap. In SIGIR '07. ACM Press, 2007.
6. G. V. Cormack and T. R. Lynam. Statistical precision of information retrieval evaluation. In SIGIR '06, pages 533-540. ACM Press, 2006.
7. E. S. Edgington. Randomization Tests.Marcel Dekker, 1995.
8. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall/CRC, 1998.
9. R.A.Fisher. The Design of Experiments. Oliver and Boyd, first edition, 1935.
10. D. Hull. Using statistical testing in the evaluation of retrieval experiments. In SIGIR '93, pages 329 - 338, New York, NY, USA, 1993. ACM Press.
11. O. Kempthorne and T. E. Doerfler. The behavior of some significance tests under experimental randomization. Biometrika, 56(2):231 - 248, August 1969.
12. M. Matsumoto and T. Nishimura. Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul., 8(1):3 - 30, 1998.
13. W. Mendenhall, D. D. Wackerly, and R. L. Scheaffer. Mathematical Statistics with Applications. PWS-KENT Publishing Company, 1990.
14. E. W. Noreen. Computer Intensive Methods for Testing Hypotheses. John Wiley, 1989.
15. R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2004. 3-900051-07-0.
16. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In SIGIR '06, pages 525 - 532. ACM Press, 2006.
17. M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In SIGIR '05, pages 162 - 169. ACM Press, 2005.
18. J. Savoy. Statistical inference in retrieval effectiveness evaluation. IPM, 33(4):495 - 512, 1997.
19. C. J. van Rijsbergen. Information Retrieval. Butterworths, second edition, 1979. http://www.dcs.gla.ac.uk/Keith/Preface.html.
20. E. M. Voorhees and D. K. Harman, editors. TREC. MIT Press, 2005.
21. W. J. Wilbur. Non-parametric significance tests of retrieval performance comparisons. J. Inf. Sci., 20(4):270 - 284, 1994.
22. F. Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80 - 83, December 1945.