Test theory for evaluating reliability of IR test collections

Information Processing and Management

Volumn 44, Issue 3, 2008, Pages 1117-1145

  Bodoff, David ^a  

^a UNIVERSITY OF HAIFA   (Israel)

Author keywords

Evaluation; Generalizabilty theory; Test collections; Test theory

Indexed keywords

ERROR ANALYSIS; INFORMATION RETRIEVAL; OPTIMIZATION; RELIABILITY THEORY;

EVALUATION; GENERALIZABILTY THEORY; TEST COLLECTIONS; TEST THEORY;

DATA PROCESSING;

EID: 40649085568     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2007.11.006     Document Type: Article

References (20)

1. Banks D., Over P., et al. Blind men and elephants: Six approaches to TREC data. Information Retrieval 1 1-2 (1999) 7-34
2. Brennan R.L. Generalizability theory (2001), Springer-Verlag
3. Buckley C., and Voorhees E. Evaluating evaluation measure stability. Proceedings of the 23rd ACM SIGIR conference on research and development in information retrieval (2000), ACM Press, Athens, Greece
4. Burdick R.K., and Graybill F.A. Confidence intervals on variance components (1992), Marcel Dekker, Inc, New York
5. Carterette B., Allan J., et al. Minimal test collections for retrieval evaluation. Proceedings of the 29th ACM SIGIR conference on research and development in information retrieval (2006), ACM, Seattle, Washington
6. Cormack G.V., Palmer C.R., et al. Efficient construction of large test collections. Proceedings of the 21st Annual ACM SIGIR conference on research and development in information retrieval (1998), ACM, Melbourne
7. Crocker, L. & Algina, J. (1986). Introduction to classical & modern test theory. Holt, Rinehart, and Winston.
8. Hersh W.R., Crabtree M.K., et al. Factors associated with success in searching Medline and applying evidence to answer clinical questions. Journal of the American Medical Informatics Association 9 3 (2002) 283-293
9. Hull D. Using statistical testing in the evaluation of retrieval experiments. Proceedings of SIGIR 93 (1993), ACM, Pittsburg, PA
10. Hull D.A. Stemming algorithms: A case study for detailed evaluation. Journal of the American Society for Information Science 47 1 (1996) 70-84
11. Robertson, S. & Soboroff, I. (2001). The TREC 2001 filtering track report. TREC 2001. Gaithersburg, MD: Department of Commerce, National Institute of Standards and Technology.
12. Sakai T. On the reliability of information retrieval metrics based on graded relevance. Information Processing and Management 43 (2006) 531-548
13. Sanderson M., and Zobel J. Information retrieval system evaluation: Effort, sensitivity, and reliability. Proceedings of 28th ACM SIGIR conference on research and development in information retrieval (2005), ACM, Salvador, Brazil
14. Shavelson R.J., and Webb N.M. Generalizability theory: A primer (1991), Sage, Newbury Park, CA
15. Soboroff I. Do TREC web collections look like the Web?. SIGIR Forum 36 (2002) 23-31
16. Soboroff, I. (2004). Overview of the TREC 2004 novelty track. TREC 2004. Gaithersburg, MD.
17. Soboroff, I. & Harman, D. (2003). Overview of the TREC 2003 novelty track. TREC 2003. Gaithersburg, MD.
18. Voorhees E. Variations in relevance judgments and the measurement of retrieval effectiveness. Proceedings of 21st ACM SIGIR conference on research and development in information retrieval (1998), ACM Press, Melbourne pp. 315-323
19. Voorhees E.M. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management 36 (2000) 697-716
20. Voorhees E.M., and Buckley C. The effect of topic set size on retrieval experiment error. Proceedings of 25th ACM SIGIR conference on research and development in information retrieval (2002), ACM Press, Tampere pp. 316-323