Multiple testing in statistical analysis of systems-based information retrieval experiments

ACM Transactions on Information Systems

Volumn 30, Issue 1, 2012, Pages

  Carterette, Benjamin A ^a  

^a UNIVERSITY OF DELAWARE   (United States)

Author keywords

Effectiveness evaluation; Experimental design; Information retrieval; Statistical analysis; Test collections

Indexed keywords

AD HOC RETRIEVAL; EFFECTIVENESS EVALUATION; EXPERIMENTAL DESIGN AND ANALYSIS; EXPERIMENTAL ENVIRONMENT; GLOBAL ANALYSIS; HIGH QUALITY; INFORMATION RETRIEVAL RESEARCH; MULTIPLE TESTING; ORDERS OF MAGNITUDE; P-VALUES; SIGNIFICANCE TESTING; STATISTICAL HYPOTHESIS TEST; STATISTICAL HYPOTHESIS TESTING; TEST COLLECTION;

DESIGN OF EXPERIMENTS; EXPERIMENTS; STATISTICAL METHODS; STATISTICAL TESTS; STATISTICS;

INFORMATION RETRIEVAL;

EID: 84859011471     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/2094072.2094076     Document Type: Article

References (38)

1. ARMSTRONG, T. G., MOFFAT, A.,WEBBER, W., AND ZOBEL, J. 2009a. EvaluatIR: Measurement and certification of IR systems. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 25-26.
2. ARMSTRONG, T. G., MOFFAT, A., WEBBER, W., AND ZOBEL, J. 2009b. Improvements that don't add up: Ad hoc retieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management.
3. BERGER, J. O. 2003. Could Fisher, Jeffreys and Neyman have agreed on testing? Stat. Sci. 18, 1, 1-32. (Pubitemid 41461821)
4. BOX, G. E. P. 1979. Robustness in the strategy of scientific model building. In Robustness in Statistics, R. Launer and G. Wilkinson, Eds.
5. BRETZ, F.,HOTHORN, T., AND WESTFALL, P. 2010. Multiple Comparisons Using R 1st Ed. CRC Press.
6. BUCKLEY, C. AND VOORHEES, E. M. 2000. Evaluating evaluation measure stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 33-40.
7. CARTERETTE, B. 2011. Model-based inference about IR systems. In Proceedings of the 3rd International Conference on the Theory of Information Retrieval (ICTIR).
8. CARTERETTE, B. AND SMUCKER,M.D. 2007. Hypothesis testingwith incomplete relevance judgments. In Proceedings of the 16th ACM International Conference on Information and Knowledge Management. 643-652.
9. CORMACK, G. V. AND LYNAM, T. R. 2006. Statistical precision of information retrieval evaluation. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 533-540.
10. CORMACK, G. V. AND LYNAM, T. R. 2007. Power and bias of subset pooling strategies. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York, NY. To appear.
11. DRAPER, N. R. AND SMITH, H. 1998. Applied Regression Analysis 3rd Ed. Wiley-Interscience.
12. GELMAN, A., CARLIN, J. B., STERN, H. S., AND RUBIN, D. B. 2004. Bayesian Data Analysis. Chapman & Hall/CRC.
13. GELMAN, A.,HILL, J., AND YAJIMA, M. 2009. Why we (usually) don't have to worry about multiple comparisons. arXiv:0907.2578v1 [stat.AP].
14. HOTHORN, T., BRETZ, F., AND WESTFALL, P. 2008. Simultaneous inference in general parametric models. Biomet. J. 50, 3, 346-363.
15. HSU, J. 1996. Multiple Comparisons: Theory and Methods 1st Ed. Chapman and Hall/CRC.
16. HULL, D. A. 1993. Using statistical testing in the evaluation of retrieval experiments. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 329-338.
17. IOANNIDIS, J. P. A. 2005a. Contradicted and initially stronger effects in highly cited clinical research. J. Amer. Medic. Assoc. 294, 2, 218-228. (Pubitemid 41043792)
18. IOANNIDIS, J. P. A. 2005b. Why most published research findings are false. PLoS Medicine 2, 8.
19. MARDIA, K. V., KENT, J. T., AND BIBBY, J. M. 1980. Multivariate Analysis 1st Ed. Academic Press.
20. MCCULLAGH, P. AND NELDER, J. 1989. Generalized Linear Models 2nd Ed. Chapman and Hall/CRC.
21. MILLER, R. G. 1981. Simultaneous Statistical Inference 2nd Ed. Springer.
22. MONAHAN, J. F. 2008. A Primer on Linear Models 1st Ed. Chapman and Hall/CRC.
23. PESARIN, F. 2001. Multivariate Permutation Tests: With Applications in Biostatistics 1st Ed. Wiley.
24. RAUDENBUSH, S.W. AND BRYK, A. S. 2002. Hierarchical LinearModels: Applications and Data AnalysisMethods 2nd Ed. Sage.
25. SANDERSON, M. AND ZOBEL, J. 2005. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of the 28th Annaul International ACM SIGIR Conference on Research and Development in Information Retrieval. 186-193.
26. SAVOY, J. 1997. Statistical inference in retrieval effectiveness evaluation. Inf. Process.Manage. 33, 4, 495-512. (Pubitemid 127392359)
27. SMUCKER, M.,ALLAN, J., AND CARTERETTE, B. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the 16th ACM International Conference on Information and Knowledge Management. 623-632.
28. SMUCKER, M. D., ALLAN, J., AND CARTERETTE, B. 2009. A comparison of statistical significance tests and small sample sizes for IR evaluation. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 630-631.
29. TAGUE-SUTCLIFFE, J. AND BLUSTEIN, J. 1994. A statistical analysis of the TREC-3 data. In Proceedings of the 3rd Text REtrieval Conference (TREC). 385-399.
30. VENABLES, W. N. AND RIPLEY, B. D. 2002. Modern Applied Statistics with S 4th Ed. Springer.
31. VOORHEES, E. M. AND HARMAN, D. 1999. Overview of the 8th Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1-24. NIST Special Publication 500-246.
32. VOORHEES, E. M. AND HARMAN, D. K., Eds. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT Press.
33. WALD, A. 1947. Sequential Analysis. John Wiley, New York.
34. WASSERMAN, L. 2003. All of Statistics. Springer.
35. WASSERMAN, L. 2006. All of Nonparametric Statistics. Springer.
36. WEBBER, W.,MOFFAT, A., AND ZOBEL, J. 2008. Statistical power in retrieval experimentation. In Proceedings of the 17th ACM International Conference on Information and Knowledge Management. 571-580.
37. WILBUR,W. J. 1994. Non-parametric significance tests of retrieval performance comparisons. J. Inf. Sci. 20, 4, 270-284.
38. ZOBEL, J. 1998. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 307-314.