An analysis of systematic judging errors in information retrieval

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 105-114

  Kazai, Gabriella ^b   Craswell, Nick ^a   Yilmaz, Emine ^b   Tahaghoghi, S M M ^a  

^a MICROSOFT   (United States)

^b MICROSOFT RESEARCH   (United Kingdom)

Author keywords

bias; noise; relevence

Indexed keywords

BIAS; FOUR-GROUP; NOISE; RELEVENCE; RETRIEVAL SYSTEMS; TEST COLLECTION; WEB SEARCHES; WIKIPEDIA;

ERRORS; INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; KNOWLEDGE MANAGEMENT; MANAGEMENT SCIENCE; SEARCH ENGINES; WORLD WIDE WEB;

SYSTEMATIC ERRORS;

EID: 84871082747     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2396779     Document Type: Conference Paper

References (22)

1. O. Alonso and R. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In Proc. of the 33rd European Conference on Advances in Information Retrieval, ECIR'11, pages 153-164. Springer-Verlag, 2011.
2. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In Proc. of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '08, pages 667-674. ACM, 2008.
3. P. Borlund. The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10):913-925, May 2003.
4. C. J. C. Burges, R. Ragno, and Q. V. Le. Learning to rank with nonsmooth cost functions. In NIPS, pages 193-200. MIT Press, 2006.
5. B. Carterette and I. Soboroff. The effect of assessor error on IR system evaluation. In Proc. of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 539-546. ACM, 2010.
6. C. L. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of the fourth ACM international conference on Web search and data mining, WSDM '11, pages 75-84. ACM, 2011.
7. C. Cuadra and R. Katter. The relevance of relevance assessment. Proc. of the American Documentation Institute, page 95âǍŞ99.
8. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), pages 20-28.
9. C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 172-179, 2010.
10. D. Hawking and N. Craswell. The very large collection and web tracks. In E. Voorhees and D. Harman, editors, TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
11. J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, New York, NY, USA, 1 edition, 2008.
12. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proc. of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, pages 64-67. ACM, 2010.
13. K. H. Jiyin He, Krisztian Balog and E. Meij. Heuristic ranking and diversification of web documents. In TREC, 2009.
14. G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: impact of HIT design on comparative system ranking. In Proc. of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11, pages 205-214. ACM, 2011.
15. K. A. Kinney, S. B. Huffman, and J. Zhai. How evaluator domain expertise affects search result relevance judgments. In Proc. of the 17th ACM Conference on Information and Knowledge Management, CIKM '08, pages 591-598. ACM, 2008.
16. E. Pronin. Perception and misperception of bias in human judgment. Trends in cognitive sciences, 11(1):37-43, 2007.
17. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. part iii: Behavior and effects of relevance. J. Am. Soc. Inf. Sci. Technol., 58(13):2126-2144, Nov. 2007.
18. F. Scholer, A. Turpin, and M. Sanderson. Quantifying test collection quality based on the consistency of relevance judgements. In Proc. of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11, pages 1063-1072. ACM, 2011.
19. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proc. of the 21st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, pages 315-323. ACM, 1998.
20. E. M. Voorhees. Evaluation by highly relevant documents. In Proc. of the 24th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '01, pages 74-82. ACM, 2001.
21. E. M. Voorhees and D. K. Harman, editors. TREC: Experimentation and Evaluation in Information Retrieval. MIT Press, 2005.
22. W. Webber, D. W. Oard, F. Scholer, and B. Hedin. Assessor error in stratified evaluation. In Proc. of the 19th ACM International Conference on Information and Knowledge Management, CIKM '10, pages 539-548. ACM, 2010.