Statistical precision of information retrieval evaluation

Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn 2006, Issue , 2006, Pages 533-540

  Cormack, Gordon V ^a   Lynam, Thomas R ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

Author keywords

Bootstrap; Confidence interval; Precision

Indexed keywords

DATA REDUCTION; INFORMATION ANALYSIS; MATHEMATICAL MODELS; METADATA; STATISTICAL METHODS; PRECISION ENGINEERING;

AVERAGE PRECISION (AP); BOOTSTRAP; CONFIDENCE INTERVAL; MEAN AVERAGE PRECISION (MAP);

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS;

BOOTSTRAP; CONFIDENCE INTERVAL; MEAN AVERAGE PRECISION (MAP);

EID: 33750336173     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1148170.1148262     Document Type: Conference Paper

References (20)

1. BUCKLEY, C., AND VOORHEES, E. M. Evaluating evaluation measure stability. In SIGIR Conference 2000 (Athens, Greece, 2000).
2. CORMACK, G. V., PALMER, C. R., AND CLARKE, C. L. A. Efficient construction of large test collections. In SIGIR Conference 1998 (Melbourne, Australia, 1998).
3. EFRON, B., AND TSIBIRANI, R. J. An Introduction to the Bootstrap. Chapman and Hall, New York, 1994.
4. FISHER, R. A. Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society 22 (1925), 700-725.
5. GLASS, G. V. Meta-analysis at 25. http://glass.ed.asu.edu/gene/papers/ meta25.html, 2000.
6. HULL, D. A. Using statistical testing in the evaluation of retrieval experiments. In Research and Development in Information Retrieval (1993), pp. 329-338.
7. LENHARD, J. Models and statistical inference: The controversy between Fisher and Neyman-Pearson. British Journal for the Philosophy of Science (2006).
8. ROTHMAN, K. J., AND GREENLAND. S. Modern Epidemiology. Lippincott Williams & Wilkins, 1998.
9. SANDERSON, M., AND JOHNO, H. Test collections with no system pooling. In SIGIR Conference 2004 (Sheffield, UK, 2004).
10. SANDERSON. M., AND ZOBEL, J. Information retrieval evaluation: Effort, sensitivity, and reliability. In SIGIR Conference 2005 (Salvador, Brazil, 2005).
11. SAVOY, J. Statistical inference in retrieval effectiveness evaluation. Information Processing and Management 33, 4 (1997), 495-512.
12. TAGUE-SUTCLIFFE, J. The pragmatics of information retrieval experimentation, revisited. Information Processing and Management 28, 4 (1992), 467-490.
13. TAGUE-SUTCLIFFE, J., AND BLUSTEIN, J. A Statistical analysis of the TREC-3 data. In Proceedings of TREC-3, The Third Information Retrieval Conference (1994), pp. 385-398.
14. VOORHEES, E., AND HARMAN. D. Overview of the Sixth Text REtrieval Conference (TREC-6). In 6th Text REtrieval Conference (Gaithersburg, MD, 1997).
15. VOORHEES, E. M. Variations in relevance judgements and the measurement of retrieval effectiveness. In SIGIR Conference 1998 (Melbourne, Australia, 1998).
16. VOORHEES, E. M. Overview of the TREC-2004 robust track. In 13th Text REtrieval Conference (Gaithersburg, MD, 2004).
17. VOORHEES, E. M., AND BUCKLEY, C. The effect of topic set size on retrieval experiment error. In SIGIR Conference 2002 (Tampere, Finland, 2002).
18. VOORHEES, E. M., AND BUCKLEY, C. Retrieval evaluation with incomplete information. In SIGIR Conference 2004 (Sheffield, UK, 2004).
19. VOORHEES, E. M., AND HARMAN, D. K., Eds. TREC - Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
20. ZOBEL, J. How reliable are the results of large-scale information retrieval experiments? In SIGIR Conference 1998 (Melbourne, Australia, 1998).