Hypothesis testing with incomplete relevance judgments

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2007, Pages 643-652

  Carterette, Ben ^a   Smucker, Mark D ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

Evaluation; Hypothesis testing; Information retrieval; Test collections

Indexed keywords

EVALUATION; HYPOTHESIS TESTING; REDUCED SETS; RELEVANCE JUDGMENTS; RETRIEVAL SYSTEMS; SIGN TESTS; TEST COLLECTIONS; TIME SPENT;

INFORMATION RETRIEVAL; INFORMATION SERVICES; KNOWLEDGE MANAGEMENT;

TESTING;

EID: 57349104848     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1321440.1321530     Document Type: Conference Paper

References (14)

1. B. Carterette, J. Allan, and R. K. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of SIGIR, pages 268-275, 2006.
2. J. Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Earlbaum Associates, 1988.
3. G. V. Cormack and T. R. Lyman. Power and bias of subset pooling strategies. In Proceedings of SIGIR, pages 837-838, 2007.
4. J. Faraway. Extending the Linear Model with R. Chapman & Hall, 2005.
5. J. M. Hoenig and D. M. Heisey. The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1):11-6, 2001.
6. E.C.Jensen. Repeatable Evaluation of Information Retrieval Effectiveness in Dynamic Environments. PhD thesis, Illinois Institute of Technology, 2006.
7. E. L. Lehmann. Testing Statistical Hypotheses. Springer, 1997.
8. M. Sanderson and J. Zobel. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of SIGIR, pages 186-193, 2005.
9. M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of CIKM, 2007. To appear.
10. C. J. van Rijsbergen. Information Retrieval. Butterworths, London, UK, 1979.
11. W. Venables and B. D. Ripley. Modern Applied Statistics with S. Springer, 2003.
12. E. M. Voorhees. Overview of the 2004 trec robust track. In 13th TREC, 2004.
13. D. Wackerly, W. Mendenhall, and R. L. Sheaffer. Mathematical Statistics With Applications. PWS Publishers, 5th edition, 1995.
14. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of SIGIR, pages 307-314, 1998.