Strategic system comparisons via targeted relevance judgments

Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Volumn , Issue , 2007, Pages 375-382

  Moffat, Alistair ^a   Webber, William ^a   Zobel, Justin ^b  

^a UNIVERSITY OF MELBOURNE   (Australia)

^b RMIT UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

DATABASE SYSTEMS; QUERY PROCESSING; TEXT PROCESSING;

RANK-BIASED PRECISION; RELEVANCE JUDGMENTS; STRATEGIC SYSTEM; TEXT RETRIEVAL SYSTEMS;

INFORMATION RETRIEVAL SYSTEMS;

EID: 36448950502     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1277741.1277806     Document Type: Article

References (21)

1. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Dumais et al. [2006], pages 541-548.
2. C. Buckley and E. M. Voorhees. Evaluating evaluation, measure stability. In N. J. Belkin, P. Ingwersen, and M.-K. Leong, editors, Proc. Twenty-Third Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 33-40, Athens, Greece, September 2000. ACM Press, New York.
3. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In M. Sanderson, K. Järvelin, J. Allan, and P. Bruza, editors, Proc. Twenty-Seventh Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 25-32, Sheffield, England, August 2004. ACM Press, New York.
4. C. Buckley and E. M. Voorhees. Retrieval system, evaluation. In TREC: Experiment and Evaluation in Information Retrieval, chapter 3, pages 53-75. MIT Press, Cambridge, Massachusetts, 2005.
5. B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Dumais et al. [2006], pages 268-275.
6. G. V. Cormack and T. R. Lynam. Statistical precision of information retrieval evaluation. In Dumais et al. [2006], pages 533-540.
7. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Croft et al. [1998], pages 282-289.
8. W. B. Croft, A. Moffat, C. J. van, Rijsbergen, R. Wilkinson, and J. Zobel, editors. Proc. Twenty-First Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, Melbourne, Australia, August 1998. ACM Press, New York.
9. S. Dumais, E. N. Efthimiadis, D. Hawking, and K. Jarvelin, editors. Proc. Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, August 2006. ACM Press, New York.
10. D. Harman. Evaluation issues in information retrieval. Information Processing & Management, 28(4):439-440, 1992.
11. K. Jarvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
12. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Marchionini et al. [2005], pages 154-161.
13. G. Marchionini, A. Moffat, J. Tate, R. Baeza-Yates, and N. Ziviani, editors. Proc. Twenty-Eighth Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, Salvador, Brazil, August 2005. ACM Press, New York.
14. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. 2007. Under review.
15. G. Salton. The state of retrieval system, evaluation,. Information Processing & Management, 28(4):441-449, 1992.
16. M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In Marchionini et al. [2005], pages 162-169.
17. T. Saracevic. Evaluation of evaluation in information retrieval. In E. A. Fox, P. Ingwersen, and R. Fidel, editors, Proc. Eighteenth Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 138-146, Seattle, Washington, July 1995. ACM Press, New York.
18. J. Tague-Sutcliffe. The pragmatics of information retrieval experimentation, revisited. Information Processing & Management, 28(4):467-490, 1992.
19. E. M. Voorhees. The philosophy of information retrieval evaluation. In C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, editors, Proc. 2001 Cross Language Evaluation Forum Workshop, pages 355-370, Darmstadt, Germany, September 2001. LNCS volume 2406.
20. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In P. S. Yu, V. J. Tsotras, E. A. Fox, and B. Liu, editors, Proc. 2006 ACM CIKM Int. Conf. Information and Knowledge Management, pages 102-111, Arlington, Virginia, November 2006. ACM Press, New York.
21. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Croft et al. [1998], pages 307-314.