Score standardization for robust comparison of retrieval systems

ADCS 2007 - Proceedings of the Twelfth Australasian Document Computing Symposium

Volumn , Issue , 2007, Pages 1-8

  Webber, William ^a   Moffat, Alistair ^a   Zobel, Justin ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Average precision; Retrieval system evaluation; Standardization

Indexed keywords

AVERAGE PRECISION; EVALUATION METRICS; MEASURE OF EFFECTIVENESS; RELEVANCE JUDGEMENT; RETRIEVAL SYSTEMS; STANDARD TESTS; SYSTEM EFFECTIVENESS; TEST COLLECTION;

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS;

STANDARDIZATION;

EID: 84876700004     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Article

References (19)

1. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In Proc. 28th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 27-34, Salvador, Brazil, 2005.
2. D. Banks, P. Over, and N.-F. Zhang. Blind men and elephants: six approaches to TREC data. Information Retrieval, 1(1): 7-34, April 1999.
3. D. Bodoff and P. Li. Test theory for assessing IR test collections. In Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 367-374, Amsterdam, The Netherlands, July 2007.
4. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In Proc. 23rd Ann. Int. ACMSIGIR Conf. on Research and Development in Information Retrieval, pages 33-40, Athens, Greece, 2000.
5. C. W. Cleverdon. The significance of the Cranfield tests on ndex languages. In Proc. 14th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 3-12, Chicago, IL, USA, 1991.
6. W. L. Hays. Statistics. Harcourt Brace, Fort Worth, 4th edition, 1991.
7. K. J̈arvelin and J. Kek̈al̈ainen. Cumulated gain-based evaluationof IR techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
8. S. Mizzaro and S. Robertson. Hits hits TREC: exploring IR evaluation results with network analysis. In Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 479-486, Amsterdam, The Netherlands, July 2007.
9. A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments. In Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 375-382, Amsterdam, The Netherlands, July 2007.
10. S. Robertson. On GMAP: and other transformations. In Proc. 15th ACM Int. Conf. on Information and Knowledge Management, pages 78-83, Arlington, VA, USA, November 2006.
11. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proc. 29th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 525-532, Seattle,WA, USA, 2006.
12. M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In Proc. 28th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 162-169, Salvador, Brazil,2005.
13. M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical significance tests for information retrieval evaluation. In Proc. 16th ACM Int. Conf. on Information and Knowledge Management, Lisboa, Portugal, 2007.
14. K. Sparck Jones and C. J. van Rijsbergen. Report on the need for and provision of an 'ideal' test collection. Technical report, University Computer Laboratory, Cambridge, 1975.
15. J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In Proc. TREC-3, November 1994. NIST Special Publication 500-225.
16. E. Voorhees and D. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. TheMIT Press, 2005.
17. E. M. Voorhees. Overview of the TREC 2004 robust retrieval track. In Proc. TREC-13, November 2004. NIST SpecialPublication 500-261.
18. E. M. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In Proc. 25th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 316-323, Tampere, Finland, 2002.
19. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Proc. 21st Ann. Int. ACMSIGIR Conf. on Research and Development in Information Retrieval, pages 307-314, Melbourne, Australia, August 1998.