Statistical comparisons of non-deterministic IR systems using two dimensional variance

Information Processing and Management

Volumn 51, Issue 5, 2015, Pages 677-694

  Jayasinghe, Gaya K ^a   Webber, William ^b   Sanderson, Mark ^a   Dharmasena, Lasitha S ^c   Culpepper, J Shane ^a  

^a RMIT UNIVERSITY   (Australia)

^b William Webber Consulting   (Australia)

^c DEAKIN UNIVERSITY   (Australia)

Author keywords

Distributed IR; Information retrieval evaluation; Non determinism; Personalised web search; Randomization; Statistical analysis

Indexed keywords

APPROXIMATION ALGORITHMS; INFORMATION RETRIEVAL; STATISTICAL METHODS; WORLD WIDE WEB;

DISTRIBUTED INFORMATION RETRIEVAL; NON DETERMINISM; NONDETERMINISTIC SYSTEMS; RANDOMIZATION; STATISTICAL COMPARISONS; STATISTICAL SIGNIFICANCE; TWO-DIMENSIONAL VARIANCE; WEB SEARCHES;

SEARCH ENGINES;

EID: 84934952068     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2015.06.005     Document Type: Article

References (39)

1. S. Ahn, S.H. Park, and K.H. Lee How to demonstrate similarity by using noninferiority and equivalence statistical testing in radiology research Radiology 267 2 2013 328 338
2. R. Aly, D. Hiemstra, and T. Demeester Taily: Shard selection using the tail of score distributions Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval (SIGIR '13) 2013 ACM Dublin, Ireland 673 682
3. C. Andrieu, N. De Freitas, A. Doucet, and M.I. Jordan An introduction to MCMC for machine learning Machine Learning 50 1-2 2003 5 43
4. R.H. Baayen, D.J. Davidson, and D.M. Bates Mixed-effects modeling with crossed random effects for subjects and items Journal of Memory and Language 59 4 2008 390 412
5. J. Callan Distributed information retrieval Advances in Information Retrieval 2002 127 150
6. J. Callan, and M. Connell Query-based sampling of text databases ACM Transactions on Information Systems (TOIS) 19 2 2001 97 130
7. J. Callan, M. Connell, and A. Du Automatic discovery of language models for text databases Proceedings of the 1999 ACM SIGMOD international conference on management of data (SIGMOD '99) Vol. 28 1999 ACM Philadelphia, Pennsylvania, USA 479 490
8. Callan, J. P., Lu, Z., & Croft, W. B. (1995). Searching distributed collections with inference networks. In Proceedings of the 18th international ACM SIGIR conference on research and development in information retrieval (SIGIR '95), Seattle, Washington, USA, July 1995 (pp. 21-28).
9. B. Carterette Model-based inference about IR systems Advances in information retrieval theory 2011 Springer 101 112
10. B. Carterette, E. Kanoulas, and E. Yilmaz Simulating simple user behavior for system effectiveness evaluation Proceedings of the 20th ACM international conference on information and knowledge management (CIKM '11) 2011 ACM 611 620
11. M. Chen, and Q. Shao Monte carlo estimation of Bayesian credible and HPD intervals Journal of Computational and Graphical Statistics 8 1 1999 69 92
12. J. Cohen The earth is round (p <.05) American Psychologist 49 12 1994 997
13. J. Cohen Statistical power analysis for the behavioral sciences 2013 Routledge Academic
14. B. Efron Bootstrap methods: Another look at the jackknife The Annals of Statistics 1979 1 26
15. B. Efron The jackknife, the bootstrap and other resampling plans Vol. 38 1982 SIAM
16. B. Efron, and R. Tibshirani An introduction to the bootstrap Vol. 57 1993 Chapman & Hall/CRC
17. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and et al. Placing search in context: The concept revisited Proceedings of the 10th international conference on World Wide Web (WWW '01) 2001 ACM Hong Kong 406 414
18. R.A. Fisher The design of experiments 1935 Oliver & Boyd. Edinburgh and London
19. G.K. Jayasinghe, W. Webber, M. Sanderson, L.S. Dharmasena, and J.S. Culpepper Evaluating non-deterministic retrieval systems Proceedings of the 37th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '14) 2014 ACM Gold Coast, Queensland, Australia 911 914
20. A. Kulkarni, and J. Callan Document allocation policies for selective searching of distributed indexes Proceedings of the 19th ACM international conference on information and knowledge management (CIKM '10) 2010 ACM Toronto, ON, Canada 449 458
21. A. Kulkarni, A.S. Tigelaar, D. Hiemstra, and J. Callan Shard ranking and cutoff estimation for topically partitioned collections Proceedings of the 21st ACM international conference on information and knowledge management (CIKM '12) 2012 ACM Maui, Hawaii, USA 555 564
22. S. Lawrence Context in web search IEEE Bulletin of the Technical Committee on Data Engineering 23 3 2000 25 32
23. F. Liu, C. Yu, and W. Meng Personalized web search for improving retrieval effectiveness IEEE Transactions on Knowledge and Data Engineering 16 1 2004 28 40
24. D. Metzler, and W.B. Croft A markov random field model for term dependencies Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '05) 2005 ACM Salvador, Brazil 472 479
25. S.E. Robertson, and E. Kanoulas On per-topic variance in IR evaluation Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval (SIGIR '12) 2012 ACM Portland, Oregon, USA 891 900
26. T. Sakai Evaluating evaluation metrics based on the bootstrap Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '06) 2006 ACM 525 532
27. T. Sakai Alternatives to Bpref Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '07) 2007 ACM 71 78
28. Sakai, T. (2014). Designing test collections for comparing many systems. In Proceedings of the 23rd ACM international conference on information and knowledge management (CIKM '14) (pp. 61-70).
29. J. Savoy Statistical inference in retrieval effectiveness evaluation Information Processing & Management 33 4 1997 495 512
30. M. Shokouhi Central-rank-based collection selection in uncooperative distributed information retrieval Advances in information retrieval 2007 Springer 160 172
31. L. Si, and J. Callan Relevant document distribution estimation method for resource selection Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '03) 2003 ACM 298 305
32. L. Si, and J. Callan Unified utility maximization framework for resource selection Proceedings of the 13th ACM international conference on information and knowledge management (CIKM '04) 2004 ACM Washington, D.C., USA 32 41
33. L. Si, and J. Callan Modeling search engine effectiveness for federated search Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '05) 2005 ACM Salvador, Brazil 83 90
34. P. Thomas, and M. Shokouhi SUSHI: Scoring scaled samples for server selection Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval (SIGIR '09) 2009 ACM Boston, MA, USA 419 426
35. E.M. Voorhees, and C. Buckley The effect of topic set size on retrieval experiment error Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '02) 2002 ACM Tampere, Finland 316 323
36. E. Walker, and A.S. Nowacki Understanding equivalence and noninferiority testing Journal of General Internal Medicine 26 2 2011 192 196
37. F. Wilcoxon Individual comparisons by ranking methods Biometrics Bulletin 1 6 1945 80 83
38. J. Xu, and J. Callan Effective retrieval with distributed collections Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '98) 1998 ACM Melbourne, Australia 112 120
39. J. Xu, and W.B. Croft Cluster-based language models for distributed retrieval Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval (SIGIR '99) 1999 ACM Berkeley, California, USA 254 261