Building reliable and reusable test collections for image retrieval: The wikipedia task at ImageCLEF

IEEE Multimedia

Volumn 19, Issue 3, 2012, Pages 24-33

  Tsikrika, Theodora ^a   Kludas, Jana ^b   Popescu, Adrian ^c  

^a UNIVERSITY OF APPLIED SCIENCES WESTERN SWITZERLAND   (Switzerland)

^b AALTO UNIVERSITY   (Finland)

^c DIF   (France)

Author keywords

benchmarks; ground truth creation; image retrieval evaluation; ImageCLEF; incomplete judgments; measurement error; multimedia; ranking stability; test collections; topic development

Indexed keywords

BENCHMARKS; GROUND TRUTH; IMAGECLEF; INCOMPLETE JUDGMENTS; MULTIMEDIA; RETRIEVAL EVALUATION; TEST COLLECTION; TOPIC DEVELOPMENT;

MEASUREMENT ERRORS; WEBSITES;

IMAGE RETRIEVAL;

EID: 84866085633     PISSN: 1070986X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MMUL.2012.17     Document Type: Article

References (16)

1. M. Sanderson, "Test Collection Based Evaluation of Information Retrieval Systems," Foundations and Trends in Information Retrieval, vol. 4, no. 4, 2010, pp. 247-375.
2. H. Müller et al., eds., ImageCLEF: Experimental Evaluation in Visual Information Retrieval, Springer, 2010.
3. T. Tsikrika and J. Kludas, "Overview of the Wikipedia MM task at ImageCLEF 2008," Evaluating Systems for Multilingual and Multimodal Information Access: Proc. 9th Workshop of the Cross-Language Evaluation Forum (CLEF 2008), Revised Selected Papers, LNCS, Springer, 2009, pp. 539-550.
4. T. Tsikrika and J. Kludas, "Overview of the Wikipedia MM task at ImageCLEF 2009," Multilingual Information Access Evaluation Vol. II Multimedia Experiments: Proc. 10th Workshop of the Cross-Language Evaluation Forum (CLEF 2009), Revised Selected Papers, LNCS, Springer, 2010, pp. 60-71.
5. A. Popescu, T. Tsikrika, and J. Kludas, "Overview of the Wikipedia Retrieval Task at ImageCLEF 2010," Working Notes for the CLEF 2010 Workshop, 2010; http://clef2010.org/resources/proceedings/clef2010labs- submission-124.pdf.
6. T. Tsikrika, A. Popescu, and J. Kludas, "Overview of the Wikipedia Image Retrieval Task at Image-CLEF 2011," Working Notes for the CLEF 2011 Labs and Workshop, 2011; http://clef2011.org/resources/proceedings/Overview- ImageCLEF-Wikipeida-Retrieval-Clef2011.pdf.
7. E.M. Voorhees and C. Buckley, "The Effect of Topic Set Size on Retrieval Experiment Error," Proc. 25th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2002, pp. 316-323.
8. C. Buckley and E.M. Voorhees, "Retrieval Evaluation with Incomplete Information," Proc. 27th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2004, pp. 25-32.
9. J. Zobel, "How Reliable Are the Results of Large-Scale Information Retrieval Experiments?" Proc. 21st ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 1998, pp. 307-314.
10. S. Büttcher et al., "Reliable Information Retrieval Evaluation with Incomplete and Biased Judgments," Proc. 30th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2007, pp. 63-70. (Pubitemid 350164945)
11. S. Pal, M. Mitra, and J. Kamps, "Evaluation Effort, Reliability and Reusability in XML Retrieval," J. Am. Soc. for Information Science and Technology, vol. 62, no. 2, 2011, pp. 375-394.
12. T. Westerveld and R. van Zwol, "The INEX 2006 Multimedia Track," Advances in XML Information Retrieval: Proc. 5th Int'l Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2006), Revised Selected Papers, LNCS, Springer, 2007, pp. 331-344.
13. C. Buckley and E.M. Voorhees, "Evaluating Evaluation Measure Stability," Proc. 23rd ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2000, pp. 33-40.
14. J.A. Aslam, E. Yilmaz, and V. Pavlu, "The Maximum Entropy Method for Analyzing Retrieval Measures," Proc. 28th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2005, pp. 27-34.
15. E. Yilmaz and S. Robertson, "On the Choice of Effectiveness Measures for Learning to Rank," Information Retrieval, vol. 13, no. 3, 2010, pp. 271-290.
16. E. Yilmaz, J.A. Aslam, and S. Robertson, "A New Rank Correlation Coefficient for Information Retrieval," Proc. 31st ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2008, pp. 587-594.