Reliable information retrieval evaluation with incomplete and biased judgements

Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Volumn , Issue , 2007, Pages 63-70

  Büttcher, Stefan ^a   Clarke, Charles L A ^a   Yeung, Peter C K ^a   Soboroff, Ian ^b  

^a UNIVERSITY OF WATERLOO   (Canada)

^b NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

Evaluation; Incomplete judgments; Information retrieval

Indexed keywords

PROBLEM SOLVING; RELIABILITY; SET THEORY; SYSTEMS ANALYSIS;

BIASED JUDGEMENTS; INCOMPLETE JUDGMENTS; MANUAL JUDGEMENTS; POOLING METHODS;

INFORMATION RETRIEVAL;

EID: 36448986732     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1277741.1277755     Document Type: Article

References (16)

1. P. Ahlgren and L. Grönqvist. Retrieval Evaluation with Incomplete Relevance Data: A Comparative Study of Three Measures. In Proceedings of the 15th A CM International Conference on Information and Knowledge Management, pages 872-873, Arlington, USA, November 2006.
2. J. A. Aslam, V. Pavlu, and E. Yilmaz. A Statistical Method for System Evaluation Using Incomplete Judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 541-548, Seattle, USA, 2006.
3. J. A. Aslam and E. Yilmaz. Inferring Document Relevance via Average Precision. In Proceedings of the 29th Annual International A CM SIGIR Conference on Research and Development in Information Retrieval, pages 601-602, Seattle, USA, 2006.
4. C. Buckley and E. M. Voorhees. Retrieval Evaluation with Incomplete Information. In Proceedings of the 27th Annual International A CM SIGIR, Conference on Research and Development in Information Retrieval, pages 25-32, Sheffield, United Kingdom, 2004.
5. S. Büttcher, C. L. A. Clarke, and I. Soboroff. The TREC 2006 Terabyte Track. In Proceedings of TREC 2006, Gaithersburg, USA, November 2006.
6. C. L. A. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2004 Terabyte Track. In Proceedings of the 13th Text REtrieval Conference, Gaithersburg, USA, November 2004.
7. C. Cleverdon. The Cranfield Tests on Index Language Devices. In Readings in Information Retrieval, pages 47-59, 1997.
8. C. Cortes and V. Vapnik. Support-Vector Networks. Machine Learning, 20(3):273-297, September 1995.
9. L. Grönqvist. Evaluating Latent Semantic Vector Models with Synonym Tests and Document Retrieval. In ELECTRA Workshop: Methodologies and Evaluation of Lexical Cohesion Techniques in Real-World Applications Beyond Bag of Words, pages 86-88, Salvador, Brazil, August 2005.
10. K. Jarvelin and J. Kekalainen. Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
11. T. Joachims. Text Categorization with Suport Vector Machines: Learning with Many Relevant Features. In Proceedings of the 10th European Conference on Machine Learning, pages 137-142, Chemnitz, Germany, April 1998.
12. T. Joachims. Transductive Inference for Text Classification using Support Vector Machines. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 200-209, Bled, Slovenia, June 1999.
13. M. G. Kendall. A New Measure of Rank Correlation. Biometrika, (30):81-89, 1938.
14. E. M. Voorhees. The Philosophy of Information Retrieval Evaluation. In Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum, pages 355-370, London, UK, 2002.
15. E. Yilmaz and J. A. Aslam. Estimating Average Precision with Incomplete and Imperfect Judgments. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management, pages 102-111, Arlington, USA, 2006.
16. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 307-314, Melbourne, Australia, 1998.