Evaluation over thousands of queries

ACM SIGIR 2008 - 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Proceedings

Volumn , Issue , 2008, Pages 651-658

  Carterette, Ben ^a   Pavlu, Virgil ^b   Kanoulas, Evangelos ^b   Aslam, Javed A ^b   Allan, James ^a  

^a University of Massachusetts Amherst   (United States)

^b Northeastern University   (United States)

Author keywords

Evaluation; Information retrieval; Million query track; Test collections

Indexed keywords

DOCUMENT SELECTIONS; EVALUATION; EVALUATION MEASURES; MILLION QUERY TRACK; RELEVANCE JUDGMENTS; TEST COLLECTIONS;

INFORMATION SERVICES; RESEARCH AND DEVELOPMENT MANAGEMENT;

INFORMATION RETRIEVAL;

EID: 57349133736     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390334.1390445     Document Type: Conference Paper

References (21)

1. J, Allan, B. Carterette, J. A. Aslam, V. Pavlu, B. Dachev, and E. Kanoulas. Overview of the TREC 2007 Million Query Track. In Proceedings of TREC, 2007.
2. J. A. Aslam and V. Pavlu. A practical sampling strategy for efficient retrieval evaluation, technical report.
3. J. A. Aslam and V. Pavlu. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of ECIR, pages 198-209. 2007.
4. J. A. Aslam, V. Pavlu, and E. Yilmaz. Measure-based metasearch. In Proceedings of SIGIR, pages 571-572, 2005.
5. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Proceedings of SIGIR, pages 541-548, 2006.
6. D. Bodoff and P. Li. Test theory for assessing ir test collection. In Proceedings of SIGIR, pages 367 374, 2007.
7. R. L. Brerman. Generalizability Theory. Springer-Verlag, New York, 2001.
8. K. R. W. Brewer and M. Hanif. Sampling With Unequal Probabilities. Springer, New York, 1983.
9. C. Buckley, D. Dimmick, I. Soboroff, and E. Voorhees. Bias and the limits of pooling. In Proceedings of SIGIR, pages 619-620, 2006.
10. B. Carterette. Robust test collections for retrieval evaluation. In Proceedings of SIGIR, pages 55 62, 2007.
11. B. Carterette, J. Allan, and R.. K. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of SIGIR, pages 268 275, 2006.
12. B. Carterette and M. Smucker. Hypothesis testing with incomplete relevance judgments. In Proceedings of CIKM, pages 643-652, 2007.
13. C. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2004 Terabyte Track. In Proceedings of TREC, 2004.
14. C. L. A. Clarke, F. Scholer, and I. Soboroff. The TREC 2005 terabyte track. In Proceedings of TREC, 2005.
15. E. C. Jensen. Repeatable Evaluation of Information Retrieval Effectiveness in Dynamic Environments. PhD thesis, Illinois Institute of Technology, 2006.
16. M. Sanderson and J. Zobel. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of SIGIR, pages 162 169, 2005.
17. K. Sparck Jones and C. J. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59-75, 1976.
18. W. L. Stevens. Sampling without replacement with probability proportional to size. Journal of the Royal Statistical Society. Series B (Methodological), Vol. 20, No. 2. (1958), pp. 393-397.
19. S. K. Thompson. Sampling. Wiley Series in Probability and Mathematical Statistics, 1992.
20. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proceedings of CIKM, pages 102 111, 2006.
21. J. Zobel. How reliable are the results of large-scale retrieval experiments? In Proceedings of SIGIR, pages 307-314, 1998.