Measuring the reusability of test collections

WSDM 2010 - Proceedings of the 3rd ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2010, Pages 231-239

  Carterette, Ben ^a   Gabrilovich, Evgeniy ^b   Josifovski, Vanja ^b   Metzler, Donald ^b  

^a University of Delaware   (United States)

^b YAHOO RESEARCH   (United States)

Author keywords

Evaluation; Reusability; Test collections

Indexed keywords

CONFIDENCE INTERVAL; CONSTRUCTION PHASE; DATA SETS; EMPIRICAL EVALUATIONS; NEW SYSTEM; TEST COLLECTION;

INFORMATION RETRIEVAL; TESTING; WORLD WIDE WEB;

REUSABILITY;

EID: 77950961451     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1718487.1718516     Document Type: Conference Paper

References (16)

1. J. A. Aslam and E. Yilmaz. Inferring document relevance from incomplete information. In Proc. 16th Intl. Conf. on Information and Knowledge Management, pages 633-642. ACM, 2007.
2. C. Buckley, D. Dimmick, I. Soboroff, and E. Voorhees. Bias and the limits of pooling. In Proc. 29th Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 620-621, 2006.
3. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proc. 27th Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 25-32. ACM, 2004.
4. S. Büttcher, C. L. A. Clarke, P. C. K. Yeung, and I. Soboroff. Reliable information retrieval evaluation with incomplete and biased judgements. In Proc. 30th Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 63-70. ACM, 2007.
5. B. Carterette. Robust test collections for information retrieval evaluation. In Proc. 30th Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 55-62, 2007.
6. B. Carterette. Low-Cost and Robust Evaluation of Information Retrieval Systems. PhD thesis, University of Massachuetts, 2008.
7. B. Carterette and J. Allan. Semiautomatic evaluation of retrieval systems using document similarities. In Proc. 16th Intl. Conf. on Information and Knowledge Management, pages 873{876, New York, NY, USA, 2007. ACM.
8. B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1994.
9. E. C. Jensen, S. M. Beitzel, A. Chowdhury, and O. Frieder. Repeatable evaluation of search services in dynamic environments. Transactions on Information Systems, 26(1):1, 2007.
10. T. Sakai. Alternatives to bpref. In Proc. 30th Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 71-78. ACM, 2007.
11. T. Sakai. Comparing metrics across TREC and NTCIR: the robustness to system bias. In Proc. 17th Intl. Conf. on Information and Knowledge Management, pages 581-590. ACM, 2008.
12. K. Spärck Jones and C. J. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59{75, 1976.
13. C. J. van Rijsbergen. Information Retrieval. Butterworths, London, 2nd edition, 1979.
14. E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA, 2005.
15. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proc. 15th Intl. Conf. on Information and Knowledge Management, pages 102-111, 2006.
16. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Proc. 21st Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 307-314, 1998.