Test theory for assessing IR test collections

Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Volumn , Issue , 2007, Pages 367-374

  Bodoff, David ^a   Li, Pu ^b  

^a UNIVERSITY OF HAIFA   (Israel)

^b 160 Jacobs Management Center   (United States)

Author keywords

Information retrieval; Test collections; Test theory

Indexed keywords

COMPUTER AIDED SOFTWARE ENGINEERING; DATA STRUCTURES; DATABASE SYSTEMS; RELIABILITY ANALYSIS;

DATA-DRIVEN METHOD; EDUCATIONAL TESTING; TEST COLLECTIONS; TEST THEORY;

INFORMATION RETRIEVAL;

EID: 36448947171     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1277741.1277805     Document Type: Article

References (11)

1. Zobel, J., How reliable are the results of large-scale information retrieval experiments? , in Proceedings of the 21st ACM SIGIR Conference on Research and Development In Information Retrieval. 1998, ACM: Melbourne.
2. Voorhees, E. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, in Proceedings of the 21st ACMSIGIR Conference on Research and Development In Information Retrieval. 1998. Melbourne: ACM Press.
3. Buckley, C and E. Voorhees. Evaluating Evaluation Measure Stability, in Proceedings of the 23rd ACM SIGIR conference on Research and Development in Information Retrieval. 2000. Athens, Greece: ACM Press.
4. Voorhees, E.M. and C Buckley. The Effect of Topic Set Size on Retrieval Experiment Error, in Proceedings of the 25th ACM SIGIR Conference on Research and Development In Information Retrieval. 2002. Tampere: ACM Press.
5. Sanderson, M. and J. Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, in Proceedings of the 28th ACM SIGIR Conference on Research and Development in Information Retrieval. 2005, ACM: Salvador, Brazil.
6. Banks, D., P. Over, and N.F. Zhang, Blind men and elephants: six approaches to TREC data. Information Retrieval, 1999. 1(1-2): p. 7-34.
7. Lange, R., et al, A probabilistic Rasch analysis of question answering evaluation, in Proceedings of Human Language Technology conference North American chapter of the Association for Computational Linguistics annual meeting, HLT/NAACL 2004.2004, Association for Computational Linguistics: Boston.
8. Crocker, L. and J. Algina, Introduction to Classical & Modern Test Theory. 1986: Holt, Rinehart, and Winston.
9. Brennan, R.L., Generalizability Theory. Statistics for Social Science and Public Policy, ed. S.E.D. Lievesley and J.R. Feinberg. 2001: Springer-Verlag.
10. Shavelson, R.J. and N.M. Webb, Generalizability Theory: A Primer. 1991, Newbury Park, CA: Sage.
11. Gao, X. and L. Brennan, Variability of Estimated Variance Components and Related Statistics in a Performance Assessment. Applied Measurement in Education, 2001. 14(2): p. 191-203.