Designing test collections for comparing many systems

CIKM 2014 - Proceedings of the 2014 ACM International Conference on Information and Knowledge Management

Volumn , Issue , 2014, Pages 61-70

  Sakai, Tetsuya ^a  

^a WASEDA UNIVERSITY   (Japan)

Author keywords

Effect sizes; Evaluation; Evaluation measures; Power; Sample sizes; Statistical significance; Test collections; Variances

Indexed keywords

BUDGET CONTROL; COST BENEFIT ANALYSIS; DESIGN; ERROR ANALYSIS; KNOWLEDGE MANAGEMENT; SAMPLING; SEARCH ENGINES;

EFFECT SIZE; EVALUATION; EVALUATION MEASURES; POWER; SAMPLE SIZES; STATISTICAL SIGNIFICANCE; TEST COLLECTION; VARIANCES;

ANALYSIS OF VARIANCE (ANOVA);

EID: 84925449110     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2661829.2661893     Document Type: Conference Paper

References (27)

1. D. Bodoff and P. Li. Test theory for assessing IR test collections. In Proceedings of ACM SIGIR 2007, pages 367-374, 2007.
2. B. Carterette. Model-based inference about IR systems. In ICTIR 2011 (LNCS 6931), pages 101-112, 2011.
3. B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan. Evaluation over thousands of queries. In Proceedings of ACM SIGIR 2008, pages 651-658, 2008.
4. B. Carterette and M. D. Smucker. Hypothesis testing with incomplete relevance judgments. In Proceedings of ACM CIKM 2007, pages 643-652, 2007.
5. O. Chapelle, S. Ji, C. Liao, E. Velipasaoglu, L. Lai, and S.-L. Wu. Intent-based diversification of web search results: Metrics and algorithms. Information Retrieval, 14(6): 572-592, 2011.
6. C. L. A. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of ACM WSDM 2011, pages 75-84, 2011.
7. C. L. A. Clarke, N. Craswell, I. Soboroff, and E. M. Voorhees. Overview of the TREC 2011 web track. In Proceedings of TREC 2011, 2012.
8. C. L. A. Clarke, N. Craswell, and E. M. Voorhees. Overview of the TREC 2012 web track. In Proceedings of TREC 2012, 2013.
9. J. Cohen. Statistical Power Analysis for the Behavioral Sciences (Second Edition). Lawrence Erlbaum Associates, 1988.
10. G. Cumming. Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. Routledge, 2012.
11. P. D. Ellis. The Essential Guide to Effect Sizes. Cambridge University Press, 2010.
12. F. Fidler, C. Geoff, B. Mark, and T. Neil. Statistical reform in medicine, psychology and ecology. The Journal of Socio-Economics, 33: 615-630, 2004.
13. R. E. Kass and A. E. Raftery. Bayes factors. Journal of the American Statistical Association, 90(430): 773-795, 1995.
14. P. R. Killeen. An alternative to null hypothesis significance tests. Psychological Science, 16: 345-353, 2005.
15. Y. Nagata. How to Design the Sample Size (in Japanese). Asakura Shoten, 2003.
16. M. J. Nelson. Statistical power and effect size in information retrieval experiments. In Proceedings of CAIS/ASCI'98, pages 393-400, 1998.
17. M. Okubo and K. Okada. Psychological Statistics to Tell Your Story: Effect Size, Confidence Interval (in Japanese). Keiso Shobo, 2012.
18. I. Ounis, C. Macdonald, J. Lin, and I. Soboroff. Overview of the TREC-2011 microblog track. In Proceedings of TREC 2011, 2012.
19. T. Sakai. Metrics, statistics, tests. In PROMISE Winter School 2013: Bridging between Information Retrieval and Databases (LNCS 8173), pages 116-163, 2014.
20. T. Sakai. Statistical reform in information retrieval? SIGIR Forum, 48(1): 3-12, 2014.
21. T. Sakai and R. Song. Evaluating diversified search results using per-intent graded relevance. In Proceedings of ACM SIGIR 2011, pages 1043-1042, 2011.
22. T. Sakai and R. Song. Diversified search evaluation: Lessons from the NTCIR-9 INTENT task. Information Retrieval, 16(4): 504-529, 2013.
23. J. Urbano, M. Marrero, and D. Martín. On the measurement of test collection reliability. In Proceedings of ACM SIGIR 2013, pages 393-402, 2013.
24. E. M. Voorhees. Overview of the TREC 2003 robust retrieval track. In Proceeings of TREC 2003, 2004.
25. E. M. Voorhees. Overview of the TREC 2004 robust retrieval track. In Proceeings of TREC 2004, 2005.
26. E. M. Voorhees. Topic set size redux. In Proceedings of ACM SIGIR 2009, pages 806-807, 2009.
27. W. Webber, A. Moffat, and J. Zobel. Statistical power in retrieval experimentation. In Proceedings of ACM CIKM 2008, pages 571-580, 2008.