On per-topic variance in IR evaluation

SIGIR'12 - Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2012, Pages 891-900

  Robertson, Stephen E ^a   Kanoulas, Evangelos ^b  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

evaluation; information retrieval; mixed effects model; significance testing; simulation; statistical precision

Indexed keywords

DOCUMENT COLLECTION; EVALUATION; LARGE POPULATION; MODEL METHOD; SIGNIFICANCE TESTING; SIMULATION; STATISTICAL PRECISION; STATISTICAL SIGNIFICANCE;

INFORMATION RETRIEVAL;

COMPUTER SIMULATION;

EID: 84866617782     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2348283.2348402     Document Type: Conference Paper

References (11)

1. David Banks, Paul Over, and Nien-Fan Zhang. Blind men and elephants: Six approaches to TREC data. Inf. Retr., 1(1-2):7-34, 1999.
2. Ben Carterette, Evangelos Kanoulas, and Emine Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In Proceedings of the 20th ACM Conference on Information and Knowledge Management, CIKM 2011, pages 611-620. ACM, 2011.
3. Gordon V. Cormack and Thomas R. Lynam. Statistical precision of information retrieval evaluation. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 533-540. ACM, 2006.
4. David Hawking and Stephen E. Robertson. On collection size and retrieval effectiveness. Inf. Retr., 6(1):99-105, 2003.
5. Philipp Jarnet. Statistics-KernelEstimation. http://search.cpan.org/ ~janert/Statistics-KernelEstimation/.
6. Evangelos Kanoulas, Virgiliu Pavlu, Keshi Dai, and Javed A. Aslam. Modeling the score distributions of relevant and non-relevant documents. In Proceedings of Second International Conference on the Theory of Information Retrieval, ICTIR 2009, pages 152-163. Springer, 2009.
7. Jose C. Pinheiro and Douglas M. Bates. Mixed-Effects Models in S and S-PLUS. Statistics and computing. Springer-Verlag New York, Inc., 2000.
8. Stephen Robertson. On smoothing average precision. In Proceedings of the 34th European Conference on IR Research, ECIR 2012, pages 158-169. Springer, 2012.
9. Stephen E. Robertson. On document populations and measures of ir effectiveness. In Studies in theory of information retrieval (Proceedings of ICTIR 2007), pages 9-12, 2007.
10. Tetsuya Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 525-532. ACM, 2006.
11. Vishwa Vinay, Ingemar J. Cox, Natasa Milic-Frayling, and Kenneth R. Wood. On ranking the effectiveness of searches. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 398-404. ACM, 2006.