Assessor error in stratified evaluation

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2010, Pages 539-548

  Webber, William ^a   Oard, Douglas W ^b   Scholer, Falk ^c   Hedin, Bruce ^d  

^a UNIVERSITY OF MELBOURNE   (Australia)

^b UNIVERSITY OF MARYLAND   (United States)

^c RMIT UNIVERSITY   (Australia)

^d H5   (United States)

Author keywords

E discovery; Estimation theory; Recall

Indexed keywords

DOUBLE-SAMPLING; E DISCOVERIES; ESTIMATION THEORY; PATENT REVIEW; RECALL; RELEVANT DOCUMENTS; RETRIEVAL EFFECTIVENESS; RETRIEVAL SYSTEMS; SAMPLING ERRORS;

ERRORS; ESTIMATION; KNOWLEDGE MANAGEMENT;

INFORMATION RETRIEVAL;

EID: 78651277941     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1871437.1871508     Document Type: Conference Paper

References (19)

1. J. Baron, D. Lewis, and D. Oard. TREC 2006 legal track overview. In Proc. 15th Text REtrieval Conference, pages 79-98, 2006.
2. C. Buckley, D. Dimmick, I. Soboroff, and E. Voorhees. Bias and the limits of pooling for large collections. Information Retrieval, 10(6): 491-508, 2007.
3. B. Carterette, V. Pavlu, E. Kanoulas, J. Aslam, and J. Allan. Evaluation over thousands of queries. In Proc. 31st ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 651-658, Singapore, Singapore, 2008.
4. C. Cuadra and R. Katter. The relevance of relevance assessment. In Proc. American Documentation Institute, volume 4, pages 95-99, 1967.
5. B. Hedin, S. Tomlinson, J. Baron, and D. Oard. Overview of the TREC 2009 legal track. In Proc. 18th Text REtrieval Conference, pages 4:1-40, 2009.
6. P. Kantor, M. Kim, U. Ibraev, and K. Atasoy. Estimating the number of relevant documents in enormous collections. In Proc. 62nd Annual Meeting of the American Society for Information Science, pages 507-514, 1999.
7. C. Lam and D. Stork. Evaluating classifiers by means of test data with noisy labels. In Proc. 18th International Joint Conference on Artificial Intelligence, Acapulco, Mexico, 2003.
8. D. Lewis. The TREC-4 filtering track. In Proc. 4th Text REtrieval Conference, 1995.
9. D. Oard, B. Hedin, S. Tomlinson, and J. Baron. Overview of the TREC 2008 legal track. In Proc. 17th Text REtrieval Conference, pages 3:1-45, 2008.
10. E. Pronin. Perception and misperception of bias in human judgment. Trends in cognitive sciences, 11(1):37-43, 2007.
11. G. Roda, J. Tait, F. Piroi, and V. Zenz. CLEF-IP 2009: retrieval experiments in the Intellectual Property domain. In Proc. 10th CLEF workshop, 2009.
12. H. Roitblat, A. Kershaw, and P. Oot. Document categorization in legal electronic discovery: computer classification vs. manual review. JASIST, 61(1):70-80, 2010.
13. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13):2126-2144, 2007. (Pubitemid 350229713)
14. A. Tenenbein. A double sampling scheme for estimating from binomial data with misclassifications. Journal of the American Statistical Association, 65(331):1350-1361, September 1970.
15. E. Voorhees. Variations in relevance judgements and the measurement of retrieval effectiveness. In Proc. 21st ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 315-323, Melbourne, Australia, 1998.
16. E. Voorhees and D. Harman. TREC: experiment and evaluation in information retrieval. MIT Press, 2005.
17. W. Wilbur. Human subjectivity and performance limits in document retrieval. Information Processing & Management, 32(5):515-527, 1996.
18. E. Yilmaz and J. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proc. 15th ACM International Conference on Information and Knowledge Management, pages 102-111, Arlington, Virginia, USA, 2006.
19. J. Zobel. How reliable are the results of large-scale information retrieval experiments? In Proc. 21st ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 307-314, Melbourne, Australia, 1998.