Quantifying test collection quality based on the consistency of relevance judgements

SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2011, Pages 1063-1072

  Scholer, Falk ^a   Turpin, Andrew ^b   Sanderson, Mark ^a  

^a RMIT UNIVERSITY   (Australia)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Information retrieval evaluation; Search engines

Indexed keywords

ERRORS; INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; TESTING;

ALTERNATE APPROACHES; GROUND TRUTH DATA; HUMAN ASSESSMENT; LEVEL OF DETAIL; RELEVANCE ASSESSMENTS; RELEVANCE JUDGEMENT; RETRIEVAL SYSTEMS; TEST COLLECTION;

SEARCH ENGINES;

EID: 80052132320     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2009916.2010057     Document Type: Conference Paper

References (24)

1. O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? Using mechanical turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 15-16, 2009.
2. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In Proceedings of the ACM SIGIR international conference on Research and development in information retrieval, pages 667-674, 2008.
3. Y. Bernstein and J. Zobel. Redundant documents and search effectiveness. In Proceedings of the ACM CIKM international conference on Information and knowledge management, pages 736-743, 2005. (Pubitemid 44022252)
4. B. Carterette and I. Soboroff. The effect of assessor error on IR system evaluation. In Proceedings of the ACM SIGIR international conference on Research and development in information retrieval, pages 539-546, 2010.
5. B. Carterette and I. Soboroff. The effect of assessor error on ir system evaluation. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 539-546, 2010.
6. C. Cleverdon. The evaluation of systems used in information retrieval (1958: Washington). In Proceedings of the International Conference on Scientific Information - Two Volumes, pages 687-698. Washington: National Academy of Sciences, National Research Council, 1959.
7. C. W. Cleverdon. The effect of variations in relevance assessments in comparative experimental tests of index languages. Cranfield Library Report 3, Cranfield Institute of Technology, 1970.
8. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Proceedings of the ACM SIGIR international conference on Research and development in information retrieval, pages 282-289, 1998.
9. C. D. Gull. Seven years of work on the organization of materials in the special library. American Documentation, 7(4):320-329, 1956.
10. D. K. Harman. The TREC test collection. In E. M. Voorhees and D. K. Harman, editors, TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
11. B. Hedin, D. Oard, S. Tomlinson, and J. Baron. Overview of the TREC 2009 legal track. In Proceedings of the 18th Text REtrieval Conference, pages 4:1-40, 2010.
12. R. V. Katter. The influence of scale form on relevance judgments. Information Storage and Retrieval, 4(1):1-11, 1968.
13. M. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information storage and retrieval, 4(4):343-359, 1968.
14. M. Sanderson, F. Scholer, and A. Turpin. Relatively relevant: Assessor shift in document judgements. In Proceedings of the Australasian Document Computing Symposium, pages 60-67, Melbourne, Australia, 2010.
15. D. Sheskin. Handbook of parametric and nonparametric statistical procedures, 4th ed. CRC Press, 2007.
16. M. D. Smucker, J. Allan, and B. Carterette. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the ACMCIKM international conference on information and knowledge management, pages 623-632, 2007.
17. R. Thorne. The efficiency of subject catalogues and the cost of information searches. Journal of documentation, 11:130-148, 1955.
18. E. M. Voorhees. Variations in relevance judgements and the measurement of retrieval effectiveness. Information Processing and Management, 36(5):697-716, 2000.
19. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36(5):697-716, 2000.
20. E. M. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 316-323, Tampere, Finland, 2002.
21. W. Webber, A. Moffat, and J. Zobel. Statistical power in retrieval experimentation. In Proceedings of the ACM CIKM International Conference on Information and Knowledge Management, pages 571-580, Napa Valley, California, USA, 2008.
22. W. Webber, D. Oard, F. Scholer, and B. Hedin. Assessor error in stratified evaluation. In Proceedings of the ACM CIKM International Conference on Information and Knowledge Management, pages 571-580, Toronto, Canada, 2010.
23. I. Witten, A. Moffat, and T. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann Publishers, 2nd edition, 1999.
24. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the ACM SIGIR international conference on Research and development in information retrieval, pages 307-314, 1998.