Alternative assessor disagreement and retrieval depth

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 125-134

  Webber, William ^a   Chandar, Praveen ^b   Carterette, Ben ^b  

^a UNIVERSITY OF MARYLAND   (United States)

^b University of Delaware   (United States)

Author keywords

evaluation; retrieval experiment; sampling

Indexed keywords

ACCURATE PREDICTION; EVALUATION; LOGISTIC REGRESSION MODELS; LOGISTIC REGRESSIONS; MODEL PARAMETERS; PREDICTIVE MODELS; RELEVANCE ASSESSMENTS; RETRIEVAL SYSTEMS; STATISTICAL SIGNIFICANCE;

FORECASTING; KNOWLEDGE MANAGEMENT; LOGISTICS; SAMPLING;

REGRESSION ANALYSIS;

EID: 84871086837     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2396781     Document Type: Conference Paper

References (23)

1. Javed Aslam and Mark Montague. Models for metasearch. In W. Bruce Croft, D. J. Harper, D. H. Kraft, and Justin Zobel, editors, Proc. 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 276-284, New Orleans, Louisiana, USA, September 2001.
2. Javed Aslam, Virgiliu Pavlu, and Emine Yilmaz. Measure-based metasearch. In Gary Marchionini, Alistair Moffat, John Tait, R. Baeza-Yates, and N. Ziviani, editors, Proc. 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 571-572, Salvador, Brazil, August 2005. doi: http://doi.acm.org/10.1145/1076034.1076133.
3. Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, A. de Vries, and Emine Yilmaz. Relevance assessment: are judges exchangeable and does it matter? In Sung-Hyon Myaeng, Douglas W. Oard, Fabrizio Sebastiani, Tat-Seng Chua, and Mun-Kew Leong, editors, Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 667-674, Singapore, Singapore, July 2008.
4. D. Bodoff and P. Li. Test theory for assessing IR test collections. In Charles L. A. Clarke, Norbert Fuhr, Noriko Kando, W. Kraaij, and A. de Vries, editors, Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 367-374, Amsterdam, the Netherlands, July 2007.
5. Ben Carterette. Robust test collections for retrieval evaluation. In Charles L. A. Clarke, Norbert Fuhr, Noriko Kando, W. Kraaij, and A. de Vries, editors, Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 55-62, Amsterdam, the Netherlands, July 2007.
6. Ben Carterette and Ian Soboroff. The effect of assessor errors on IR system evaluation. In Hsin-Hsi Chen, Efthimis N. Efthimiadis, Jacques Savoy, Fabio Crestani, and Stephanie Marchand-Maillet, editors, Proc. 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 539-546, Geneva, Switzerland, July 2010.
7. Gordon V. Cormack, Charles L. A. Clarke, Christopher R. Palmer, and Samuel S. L. To. Passage-based refinement (MultiText experiments for TREC-6). In Ellen Voorhees and Donna Harman, editors, Proc. 6th Text REtrieval Conference, pages 303-320, Gaithersburg, Maryland, USA, November 1997. NIST Special Publication 500-240.
8. C. Cuadra and R. Katter. The relevance of relevance assessment. Proc. American Documentation Institute, 4:95-99, 1967.
9. Maura R. Grossman and Gordon V. Cormack. Inconsistent assessment of responsiveness in e-discovery: difference of opinion or human error? In DESI IV: The ICAIL Workshop on Setting Standards for Searching Electronically Stored Information in Discovery Proceedings, pages 1-11, Pittsburgh, PA, USA, June 2011.
10. Donna Harman. Overview of the fourth text REtrieval conference (TREC-4). In Donna Harman, editor, Proc. 4th Text REtrieval Conference, pages 1-23, Gaithersburg, Maryland, USA, November 1995. NIST Special Publication 500-236.
11. D. W. Hosmer, T. Hosmer, S. le Cessie, and S. Lemeshow. A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine, 16:965-980, 1997.
12. Joon Ho Lee. Analyses of multiple evidence combination. In Nicholas J. Belkin, A. Desai Narasimhalu, Peter Willett, and William Hersh, editors, Proc. 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 267-276, Philadelphia, PA, USA, July 1997.
13. Douglas W. Oard, Bruce Hedin, Stephen Tomlinson, and Jason R. Baron. Overview of the TREC 2008 legal track. In Ellen Voorhees and Lori P. Buckland, editors, Proc. 17th Text REtrieval Conference, pages 3:1-45, Gaithersburg, Maryland, USA, November 2008. NIST Special Publication 500-277.
14. Herbert L. Roitblat, Anne Kershaw, and Patrick Oot. Document categorization in legal electronic discovery: computer classification vs. manual review. Journal of the American Society for Information Science and Technology, 61(1):70-80, 2010.
15. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13):2126-2144, 2007.
16. Jacques Savoy. Statistical inference in retrieval effectiveness evaluation. Information Processing & Management, 33(4):495-512, 1997.
17. Victor S. Sheng, F. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Ying Li, Bing Liu, and Sunita Sarawagi, editors, Proc. 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 614-622, Las Vegas, Nevada, USA, August 2008.
18. Ian Soboroff, Charles Nicholas, and Patrick Cahan. Ranking retrieval systems without relevance judgments. In W. Bruce Croft, D. J. Harper, D. H. Kraft, and Justin Zobel, editors, Proc. 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 66-73, New Orleans, Louisiana, USA, September 2001.
19. Andrew Trotman and Dylan Jenkinson. IR evaluation using multiple assessors per topic. In MingFang Wu, Andrew Turpin, and Amanda Spink, editors, Proc. 12th Australasian Document Computing Symposium, pages 9-16, Melbourne, December 2007.
20. Ellen Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors, Proc. 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 315-323, Melbourne, Australia, August 1998.
21. Ellen Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management, 36(5): 697-716, September 2000.
22. William Webber, Douglas W. Oard, Falk Scholer, and Bruce Hedin. Assessor error in stratified evaluation. In Proc. 19th ACM International Conference on Information and Knowledge Management, pages 539-548, Toronto, Canada, October 2010.
23. William Webber, Bryan Toth, and Marjorie Desamito. Effect of written instructions on assessor agreement. In William Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson, editors, Proc. 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Portland, Oregon, USA, August 2012. to appear.