Minimal test collections for retrieval evaluation

Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn 2006, Issue , 2006, Pages 268-275

  Carterette, Ben ^a   Allan, James ^a   Sitaraman, Ramesh ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

Algorithms; Evaluation; Information retrieval; Test collections; Theory

Indexed keywords

ALGORITHMS; CHARACTER SETS; ELECTRONIC DOCUMENT EXCHANGE; INFORMATION THEORY; PARAMETER ESTIMATION; DATA ACQUISITION; DATA REDUCTION; DATA STRUCTURES;

DOCUMENT JUDGMENTS; EVALUATION METRICS; TEST COLLECTIONS; ACCURATE ESTIMATIONS; NATURAL ALGORITHMS; RELEVANCE JUDGMENTS;

INFORMATION RETRIEVAL;

EID: 33750359727     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1148170.1148219     Document Type: Conference Paper

References (16)

1. C. Buckley and E. M. Voorhees. Evaluating Evaluation Measure Stability. In Proceedings of SIGIR, pages 33-40, 2000.
2. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of SIGIR, pages 25-32, 2004.
3. B. Carterette and J. Allan. Incremental Test Collections. In Proceedings of CIKM, pages 680-687, 2005.
4. G. V. Cormack, C. R. Palmer, and C. L. Clarke. Efficient Construction of Large Test Collections. In Proceedings of SIGIR, pages 282-289, 1998.
5. R. Gentleman and R. Ihaka. The R Language. In Proceedings of the 28th Symposium on the Interface, 1997.
6. M. Kendall. Rank Correlation Methods. Griffin, London, UK, fourth edition, 1970.
7. M. Sanderson and H. Joho. Forming test collections with no system pooling. In Proceedings of SIGIR, pages 33-40, 2004.
8. M. Sanderson and J. Zobel. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of SIGIR, pages 186-193, 2005.
9. A. Singhal. Challenges in Running a Commercial Search Engine. In Proceedings of SIGIR, page 432, 2005. Keynote Talk.
10. I. Soboroff, C. Nicholas, and P. Cahan. Ranking Retrieval Systems without Relevance Judgments. In Proceedings of SIGIR, pages 66-73, 2001.
11. K. Sparck Jones and C. J. van Rijsbergen. Information Retrieval Test Collections. Journal of Documentation, 32(1):59-75, 1976.
12. M. A. Stephens. EDF Statistics for Goodness of Fit and Some Comparisons. Journal of the American Statistical Association, 69:730-737, 1974.
13. E. Voorhees. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. In Proceedings of SIGIR, pages 315-323, 1998.
14. E. Voorhees. Overview of the TREC 2005 Robust Retrieval Track. In TREC 2005 Notebook, 2005.
15. E. M. Voorhees. The philosophy of information retrieval evaluation. In CLEF '01: Revised Papers from the Second Workshop of CLEF, pages 355-370, London, UK, 2002. Springer-Verlag.
16. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of SIGIR, pages 307-314, 1998.