A general linear mixed models approach to study system component effects

SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2016, Pages 25-34

  Ferro, Nicola ^a   Silvello, Gianmaria ^a  

^a UNIVERSITY OF PADOVA   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION RETRIEVAL;

AD HOC DATUM; COMPONENT EFFECT; EVALUATION MEASURES; LINEAR MIXED MODELS; SYSTEM COMPONENTS; SYSTEM DEVELOPERS;

BIT ERROR RATE;

EID: 84980376576     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2911451.2911530     Document Type: Conference Paper

References (28)

1. D. Banks, P. Over, and N.-F. Zhang. Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval, 1:7-34, May 1999.
2. L. Boytsov, A. Belova, and P. Westfall. Deciding on an Adjustment for Multiplicity in IR Experiments. In SIGIR 2013, pp. 403-412, 2013.
3. S. Büttcher, C. L. A. Clarke, and G. V. Cormack. Information Retrieval: Implementing and Evaluating Search Engines. The MIT Press, USA, 2010.
4. B. A. Carterette. Multiple Testing in Statistical Analysis of Systems-Based Information Retrieval Experiments. ACM TOIS, 30(1):4:1-4:34, 2012.
5. E. Di Buccio, G. M. Di Nunzio, N. Ferro, D. K. Harman, M. Maistro, and G. Silvello. Unfolding Off-the-shelf IR Systems for Reproducibility. In SIGIR RIGOR 2015, 2015.
6. N. Ferro, R. Berendsen, A. Hanbury, M. Lupu, V. Petras, M. de Rijke, and G. Silvello. PROMISE Retreat Report - Prospects and Opportunities for Information Access Evaluation. SIGIR Forum, 46(2):60-84, 2012.
7. N. Ferro and D. Harman. CLEF 2009: Grid@CLEF Pilot Track Overview. In CLEF 2009, pp. 552-565. LNCS 6241, 2010.
8. A. Hanbury and H. Müller. Automated Component-Level Evaluation: Present and Future. In CLEF 2010, pp. 124-135. LNCS 6360, 2010.
9. D. A. Hull. Using Statistical Testing in the Evaluation of Retrieval Experiments. In SIGIR 1993, pp. 329-338, 1993.
10. G. K. Jayasinghe, W. Webber, M. Sanderson, L. S. Dharmasena, and J. S. Culpepper. Statistical comparisons of non-deterministic IR systems using two dimensional variance. IPM, 51(5):677-694, 2015.
11. J. Lin, M. Crane, A. Trotman, J. Callan, I. Chattopadhyaya, J. Foley, G. Ingersoll, C. Macdonald, and S. Vigna. Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge. In ECIR 2016, pp. 357-368. LNCS 9626, 2016.
12. C. Macdonald, R. McCreadie, R. L. T. Santos, and I. Ounis. From Puppy to Maturity: Experiences in Developing Terrier. OSIR at SIGIR, pp. 60-63, 2012.
13. S. Maxwell and H. D. Delaney. Designing Experiments and Analyzing Data. A Model Comparison Perspective. Lawrence Erlbaum Associates, 2nd ed, 2004.
14. K. R. Murphy and B. Myors. Statistical power analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests (2nd ed.). Lawrence Erlbaum, 2004.
15. S. Olejnik and J. Algina. Generalized Eta and Omega Squared Statistics: Measures of Effect Size for Some Common Research Designs. Psychological Methods, 8(4):434-447, 2003.
16. S. E. Robertson. The methodology of information retrieval experiment. In Information Retrieval Experiment, pp. 9-31. Butterworths, 1981.
17. S. E. Robertson and E. Kanoulas. On Per-topic Variance in IR Evaluation. In SIGIR 2012, pp. 891-900, 2012.
18. A. Rutherford. ANOVA and ANCOVA. A GLM Approach. John Wiley & Sons, 2nd ed, 2011.
19. T. Sakai. Evaluating Evaluation Metrics based on the Bootstrap. In SIGIR 2006, pp. 525-532, 2006.
20. T. Sakai. Statistical reform in information retrieval? SIGIR Forum, 48(1):3-12, 2014.
21. J. Savoy. Statistical Inference in Retrieval Effectiveness Evaluation. IPM, 33(44):495-512, 1997.
22. M. D. Smucker, J. Allan, and B. A. Carterette. A Comparison of Statistical Significance Tests for Information Retrieval Evaluation. In CIKM 2007, pp. 623-632, 2007.
23. J. M. Tague-Sutcliffe and J. Blustein. A Statistical Analysis of the TREC-3 Data. In Overview of TREC-3, pp. 385-398. NIST, SP 500-225, 1994.
24. A. Trotman, C. L. A. Clarke, I. Ounis, J. S. Culpepper, M.-A. Cartright, and S. Geva. Open Source Information Retrieval: a Report on the SIGIR 2012 Workshop. ACM SIGIR Forum, 46(2):95-101, 2012.
25. L. Wang, P. N. Bennett, and K. Collins-Thompson. Robust Ranking Models via Risk-Sensitive Optimization. In SIGIR 2012, pp. 761-770, 2012.
26. W. Webber, A. Moffat, and J. Zobel. Score Standardization for Inter-Collection Comparison of Retrieval Systems. In SIGIR 2008, pp. 51-58, 2008.
27. W. J. Wilbur. Non-parametric significance tests of retrieval performance comparisons. J. Inf. Science, 20(4):270-284, 1994.
28. P. Zhang, S. Dawei, J. Wang, and Y. Hou. Bias-variance analysis in estimating true query model for information retrieval. IPM, 50(1):199-217, 2014.