Reproducibility challenges in information retrieval evaluation

Journal of Data and Information Quality

Volumn 8, Issue 2, 2017, Pages

  Ferro, Nicola ^a  

^a UNIVERSITY OF PADOVA   (Italy)

Author keywords

Reproducibility

Indexed keywords

EID: 85009270128     PISSN: 19361955     EISSN: 19361963     Source Type: Journal    
DOI: 10.1145/3020206     Document Type: Article

References (32)

1. M. Agosti, E. Di Buccio, N. Ferro, I. Masiero, S. Peruzzo, and G. Silvello. 2012a. DIRECTions: Design and specification of an IR evaluation infrastructure. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF'12). LNCS 7488, 88-99.
2. M. Agosti, G. M. Di Nunzio, and N. Ferro. 2007. Scientific data of an evaluation campaign: Do we properly deal with them? In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF'06). LNCS 4730, 11-20.
3. M. Agosti, N. Ferro, and C. Thanos. 2012b. DESIRE 2011 workshop on data infrastructures for supporting information retrieval evaluation. SIGIR Forum 46, 1, 51-55.
4. J. Allan, B. A. Carterette, J. A. Aslam, V. Pavlu, B. Dachev, and E. Kanoulas. 2008. Million query track 2007 overview. In Proceedings of the 2007 Text REtrieval Conference (TREC'07). NIST SP 500-274.
5. O. Alonso and S. Mizzaro. 2012. Using crowdsourcing for TREC relevance assessment. Information Processing & Management 48, 6, 1053-1066.
6. E. Amigó, A. Corujo, J. Gonzalo, E. Meij, and M. de Rijke. 2012. Overview of RepLab 2012: Evaluating online reputation management systems. In CLEF 2012 Working Notes. Retrieved from http://ceur-ws.org/Vol-1178/.
7. J. Arguello, M. Crane, F. Diaz, J. Lin, and A. Trotman. 2015. Report on the SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). SIGIR Forum 49, 2, 107-116.
8. J. Callan and A. Moffat. 2012. Panel on use of proprietary data. SIGIR Forum 46, 2, 10-18.
9. W. B. Croft, D. Metzler, and T. Strohman. 2009. Search Engines: Information Retrieval in Practice. Pearson.
10. E. Di Buccio, G. M. Di Nunzio, N. Ferro, D. K. Harman, M. Maistro, and G. Silvello. 2015. Unfolding off-the-shelf IR systems for reproducibility. In Proceedings of the SIGIR Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR'15).
11. M. Dussin and N. Ferro. 2009. Managing the knowledge creation process of large-scale evaluation campaigns. In Proceedings of the European Conference on Digital Libraries (ECDL'09). LNCS 5714, 63-74.
12. N. Ferro. 2014. CLEF 15th birthday: Past, present, and future. SIGIR Forum 48, 2, 31-55.
13. N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, and G. Silvello (Eds.). 2016a. Proceedings of the 38th European Conference on IR Research (ECIR'16). LNCS 9626.
14. N. Ferro, N. Fuhr, K. Järvelin, N. Kando, M. Lippold, and J. Zobel. 2016b. Increasing reproducibility in IR: Findings from the Dagstuhl seminar on "reproducibility of data-oriented experiments in e-science". SIGIR Forum 50, 1, 68-82.
15. N. Ferro, A. Hanbury, H. Müller, and G. Santucci. 2011. Harnessing the scientific data produced by the experimental evaluation of search engines and information access systems. In Proceedings of the International Conference on Computational Science (ICCS'11). Procedia Computer Science, 740-749.
16. N. Ferro and G. Silvello. 2015. Rank-biased precision reloaded: Reproducibility and generalization. In Proceedings of the 37th European Conference on Information Retrieval (ECIR'15). LNCS 9022, 768-780.
17. J. Freire, N. Fuhr, and A. Rauber (Eds.). 2016. Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science. Dagstuhl Reports 6, 1.
18. N. Fuhr, A. Rauber, G. Kazai, and A. Hanbury (Eds.). 2015. Proceedings of the 37th European Conference on IR Research (ECIR'15). LNCS 9022.
19. A. Hanbury, H. Müller, K. Balog, T. Brodt, G. V. Cormack, I. Eggel, T. Gollub, F. Hopfgartner, J. Kalpathy-Cramer, N. Kando, A. Krithara, J. Lin, S. Mercer, and M. Potthast. 2015. Evaluation-as-a-service: Overview and outlook. CoRR abs/1512.07454.
20. D. K. Harman. 2011. Information Retrieval Evaluation. Morgan & Claypool Publishers.
21. D. K. Harman and E. M. Voorhees (Eds.). 2005. TREC. Experiment and Evaluation in Information Retrieval. MIT Press.
22. A. Lipani, F. Piroi, L. Andersson, and A. Hanbury. 2014. An information retrieval ontology for information retrieval nanopublications. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF'14). LNCS 8685, 44-49.
23. M. Lupu and A. Hanbury. 2013. Patent retrieval. Foundations and Trends in Information Retrieval 7, 1 (2013), 1-97.
24. M. Potthast, T. Gollub, F. Rangel Pardo, P. Rosso, E. Stamatatos, and B. Stein. 2014. Improving the reproducibility of PAN's shared tasks: Plagiarism detection, author identification, and author profiling. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF'14). LNCS 8685, 268-299.
25. T. Sakai. 2006. Evaluating evaluation metrics based on the bootstrap. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'06). 525-532.
26. M. Sanderson. 2010. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval 4, 4 (2010), 247-375.
27. G. Silvello, G. Bordea, N. Ferro, P. Buitelaar, and T. Bogers. 2016. Semantic representation and enrichment of information retrieval experimental data. International Journal on Digital Libraries. DOI:http://dx.doi.org/10.1007/s00799-016-0172-8
28. G. Silvello and N. Ferro. 2016. "Data citation is coming". Introduction to the special issue on data citation. Bulletin of IEEE Technical Committee on Digital Libraries (IEEE-TCDL) 12, 1, 1-5.
29. E. M. Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management 36, 5, 697-716.
30. E. M. Voorhees, S. Rajput, and I. Soboroff. 2016. Promoting repeatability through open runs. In Proceedings of the 7th International Workshop on Evaluating Information Access (EVIA'16). 17-20.
31. J. Zobel. 1998. How reliable are the results of large-scale information retrieval experiments. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'98). 307-314.
32. J. Zobel, W. Webber, M. Sanderson, and A. Moffat. 2011. Principles for robust evaluation infrastructure. In Proceedings of the Workshop on Data Infrastructures for Supporting Information Retrieval Evaluation (DESIRE'11). ACM, 3-6.