Score standardization for inter-collection comparison of retrieval systems

ACM SIGIR 2008 - 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Proceedings

Volumn , Issue , 2008, Pages 51-58

  Webber, William ^a   Moffat, Alistair ^a   Zobel, Justin ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Average precision; Evaluation; Retrieval experiment; System measurement

Indexed keywords

INFORMATION RETRIEVAL; INFORMATION SERVICES; RESEARCH AND DEVELOPMENT MANAGEMENT; STANDARDIZATION;

AVERAGE PRECISION; EVALUATION; EVALUATION METRIC; RETRIEVAL EXPERIMENT; RETRIEVAL SYSTEMS; SYSTEM EVALUATIONS; SYSTEM MEASUREMENT; SYSTEM PERFORMANCES;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 57349160444     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390334.1390346     Document Type: Conference Paper

References (24)

1. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between IR effectiveness measures and user satisfaction. In Clarke et al. [2007], pages 773-774.
2. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In Marchionini et al. [2005], pages 27-34.
3. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Dumais et al. [2006]. pages 541-548.
4. D. Bodoff and P. Li. Test theory for assessing IR. test collections. In Clarke et al. [2007], pages 367-374.
5. C. Buckley. The SMART project at TREC. In Voorhees and Harman. [2005], chapter 1.3.
6. C. Buckley and E. Voorhees. Retrieval system evaluation. In Voorhees and Harman [2005], chapter 3.
7. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In E. Yannakoudis, N. Belkin, M. Leong, and R Ingwersen, editors, Proc. 23rd Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 33-40, Athens, Greece, August 2000.
8. C. L. A. Clarke, N. Fuhr, N. Kando, W. Kraaij, and A. de Vries, editors. Proc. 30th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Amsterdam, July 2007.
9. S. Dumais, E. N. Efthimiadis, D. Hawking, and K. Järvelin, editors. Proc. 29th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Seattle, Washington, August 2006.
10. W. L. Hays. Statistics. Harcourt Brace, Fort Worth, 4th edition, 1991.
11. S. B. Huffman and M. Höchster. How well does result relevance predict session satisfaction? In Clarke et al. [2007], pages 567-574.
12. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422-446, 2002.
13. G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, editors. Proc. 28th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Salvador, Brazil, August 2005.
14. S. Mizzaro and S. Robertson. HITS hits TREC: exploring IR evaluation results with network analysis. In Clarke et al. [2007], pages 479-486.
15. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., to appear.
16. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Dumais et al. [2006], pages 525-532.
17. M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In Marchionini et al. [2005], pages 162-169.
18. J. Savoy. Statistical inference in retrieval effectiveness evaluation. Information Processing & Management, 33(4):495-512, 1997.
19. M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical, significance tests for information retrieval evaluation. In M. J. Silva, A. A. F. Laender, R. Baeza-Yates, D. L. McGuinness, B. Olstad, Ø. H. Olsen, and A. O. Falcão, editors, Proc. 16th ACM Int. Conf. on Information and Knowledge Management, pages 623-632, Lisboa, Portugal, November 2007.
20. J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In D. K. Harman, editor, Proc. TREC-3, pages 385-398, November 1994. NIST Special Publication 500-225.
21. A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Dumais et al. [2006], pages 11-18.
22. E. Voorhees and D. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, Cambridge, Mass., 2005.
23. W. Webber, A. Moffat, and J. Zobel. Score standardization for robust comparison of retrieval systems. In M. Wu, A. Turpin, and A. Spink, editors, Proc. 12th Australasian Document Computing Symposium, pages 1-8, Melbourne, December 2007.
24. J. Zobel. How reliable are the results of large-scale information retrieval, experiments? In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors, Proc. 21st Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 307-314, Melbourne, Australia, August 1998.