Statistical power in retrieval experimentation

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2008, Pages 571-580

  Webber, William ^a   Moffat, Alistair ^a   Zobel, Justin ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Evaluation; Retrieval experiment; System measurement

Indexed keywords

EVALUATION; HYBRID METHODOLOGIES; RELEVANCE ASSESSMENTS; RETRIEVAL EXPERIMENT; SAMPLE SIZES; STATISTICAL POWER; SYSTEM MEASUREMENT; TEST SETS;

KNOWLEDGE MANAGEMENT; STATISTICAL TESTS;

EXPERIMENTS;

EID: 70349250276     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1458082.1458158     Document Type: Conference Paper

References (22)

1. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In S. Dumais, E. Efthimiadis, D. Hawking, and K. Järvelin, editors, Proc. 29th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 541-548, Seattle, USA, August 2006.
2. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In M. Sanderson, K. Järvelin, J. Allan, and P. Bruza, editors, Proc. 27th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 25-32, Sheffield, United Kingdom, August 2004.
3. B. Carterette. Robust test collections for retrieval evaluation. In C. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. de Vries, editors, Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 55-62, Amsterdam, the Netherlands, July 2007.
4. B. Carterette and M. D. Smucker. Hypothesis testing with incomplete relevance judgments. In M. J. Silvaa, A. A. F. Laender, R. Baeza-Yates, D. L. McGuinness, B. Olstad, Ø. H. Olsen, and A. O. Falcão, editors, Proc. 16th ACM Int. Conf. on Information and Knowledge Management, pages 643-652, Lisboa, Portugal, 2007.
5. B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan. Evaluation over thousands of queries. In S.-H. Myaeng, D. W. Oard, F. Sebastiani, T.-S. Chua, and M.-K. Leong, editors, Proc. 31st Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 651-658, Singapore, Singapore, July 2008.
6. J. Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, 2nd edition, 1988.
7. G. V. Cormack and T. R. Lynam. Validity and power of t-test for comparing MAP and GMAP. In C. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. de Vries, editors, Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 753-754, Amsterdam, the Netherlands, July 2007.
8. B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993.
9. W. L. Hays. Statistics. Harcourt Brace, Fort Worth, 4th edition, 1991.
10. J. M. Hoenig and D. M. Heisey. The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55 (1):19-24, February 2001.
11. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 2009. To appear.
12. A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments. In C. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. de Vries, editors, Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 375-382, Amsterdam, the Netherlands, July 2007.
13. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In S. Dumais, E. Efthimiadis, D. Hawking, and K. Järvelin, editors, Proc. 29th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 525-532, Seattle, USA, August 2006.
14. T. Sakai. Alternatives to Bpref. In C. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. de Vries, editors, Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 71-78, Amsterdam, the Netherlands, July 2007.
15. M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, editors, Proc. 28th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 162-169, Salvador, Brazil, August 2005.
16. J. Savoy. Statistical inference in retrieval effectiveness evaluation. Information Processing & Management, 33(4):495-512, 1997.
17. M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical significance tests for information retrieval evaluation. In M. J. Silvaa, A. A. F. Laender, R. Baeza-Yates, D. L. McGuinness, B. Olstad, Ø. H. Olsen, and A. O. Falcão, editors, Proc. 16th ACM Int. Conf. on Information and Knowledge Management, pages 623-632, Lisboa, Portugal, 2007.
18. E. Voorhees and D. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, 2005.
19. E. M. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In K. J rvelin, M. Beaulieu, R. Baeza-Yates, and Sung Hyon Myaeng, editors, Proc. 25th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 316-323, Tampere, Finland, August 2002.
20. W. Webber, A. Moffat, and J. Zobel. Score standardization for inter-collection comparison of retrieval systems. In S.-H. Myaeng, D. W. Oard, F. Sebastiani, T.-S. Chua, and M.-K. Leong, editors, Proc. 31st Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 51-58, Singapore, Singapore, July 2008.
21. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proc. 15th ACM Int. Conf. on Information and Knowledge Management, pages 102-111, Arlington, Virginia, USA, November 2006.
22. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors, Proc. 21st Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 307-314, Melbourne, Australia, August 1998.