Document features predicting assessor disagreement

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 745-748

  Chandar, Praveen ^a   Webber, William ^b   Carterette, Ben ^a  

^a University of Delaware   (United States)

^b UNIVERSITY OF MARYLAND   (United States)

Author keywords

Evaluation; Retrieval experiment

Indexed keywords

DOCUMENT CONTENTS; EVALUATION; LOGISTIC MODELS; METASEARCH; PREDICTIVE POWER; READING LEVEL;

INFORMATION RETRIEVAL;

EID: 84883078303     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484161     Document Type: Conference Paper

References (11)

1. J. A. Aslam, V. Pavlu, and E. Yilmaz. Measure-based metasearch. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '05, page 571, New York, New York, USA, 2005. ACM Press.
2. P. Bailey, P. Thomas, N. Craswell, A. P. D. Vries, I. Soboroff, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In Proceedings of SIGIR, SIGIR '08, pages 667-674. ACM, 2008.
3. M. Bendersky, W. Croft, and Y. Diao. Quality-biased ranking of web documents. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 95-104. ACM, 2011.
4. B. Carterette and I. Soboroff. The effect of assessor error on ir system evaluation. In Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 539-546. ACM, 2010.
5. K. Collins-Thompson and J. Callan. Predicting reading difficulty with statistical language models. J. Am. Soc. Inf. Sci. Technol., 56(13):1448-1462, Nov. 2005.
6. J. Y. Kim, K. Collins-Thompson, P. N. Bennett, and S. T. Dumais. Characterizing web content, user interests, and search behavior by reading level and topic. In Proceedings of the fifth ACM international conference on Web search and data mining, WSDM '12, pages 213-222, New York, NY, USA, 2012. ACM.
7. E. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management, 36(5):697-716, Sept. 2000.
8. J. Wang and D. Soergel. A user study of relevance judgments for e-discovery. Proceedings of the American Society for Information Science and Technology, 47:1-10, 2010.
9. W. Webber. Re-examining the effectiveness of manual review. In Proc. SIGIR Information Retrieval for E-Discovery Workshop, pages 2:1-8, Beijing, China, July 2011.
10. W. Webber, P. Chandar, and B. Carterette. Alternative assessor disagreement and retrieval depth. In Proceeding 21st International Conference on Information and Knowledge Management - CIKM'12, pages 125-134, 2012.
11. W. Webber, B. Toth, and M. Desamito. Effect of written instructions on assessor agreement. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, SIGIR '12, pages 1053-1054, New York, NY, USA, 2012. ACM.