Recent developments in the evaluation of information retrieval systems: Moving towards diversity and practical relevance

Informatica (Ljubljana)

Volumn 32, Issue 1, 2008, Pages 27-38

  Mandl, Thomas ^a  

^a UNIVERSITY OF HILDESHEIM   (Germany)

Author keywords

Evaluation; Information retrieval

Indexed keywords

COMPUTER NETWORKS; INFORMATION ANALYSIS; INFORMATION RETRIEVAL SYSTEMS; INFORMATION SCIENCE; INFORMATION SERVICES; INFORMATION SYSTEMS; MANAGEMENT INFORMATION SYSTEMS; MULTITASKING; SEARCH ENGINES;

CORPORA (CO); MULTI MEDIA OBJECTS; RETRIEVAL APPLICATIONS; RETRIEVAL SYSTEMS; SCIENTIFIC DOCUMENTS; STRUCTURED DATA; SYSTEM PERFORMANCES; USAGE SCENARIOS; WEB PAGES;

INFORMATION RETRIEVAL;

EID: 44649180547     PISSN: 03505596     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (45)

1. Allen, James (2004). HARD Track Overview in TREC 2004 High Accuracy Retrieval from Documents. In: Buckland, Lori; Voorhees, Ellen (Eds.). The Thirteenth Text Retrieval Conference (TREC 2004) NIST Special Publication: SP 500-261. http://trec.nist.gov/pubs/trec 13/tl3_proceedings.ht ml
2. Artemenko, Olga; Mandl, Thomas; Shramko, Margaryta; Womser-Hacker, Christa (2006). Evaluation of a Language Identification System for Mono- and Multi-lingual Text Documents. In: Proceedings of 2006 ACM SAC Symposium on Applied Computing (SAC) April, 23-27, 2006, Dijon, France, pp. 859-860.
3. Baeza-Yates, Ricardo; Ribeiro-Neto, Berthier (1999). Retrieval Evaluation. In: Baeza-Yates, R.; Ribeiro-Neto, B. (eds.): Modern Information Retrieval. Addison-Wesley. pp. 73-97.
4. Bollmann, Peter (1984). Two Axioms for Evaluation Measures in Information Retrieval. In: Proceedings of the Third Joint BCS/ACM Symposium on Research and Development in Information Retrieval (SIGIR 1984) Cambridge, 2-6 July 1984. pp. 233-245
5. Braschler, Martin; Peters, Carol (2004). Cross-Language Evaluation Forum: Objectives, Results, Achievements. In: Information Retrieval no. 7. pp. 7-31.
6. Broder, Andrei (2002). A taxonomy of web search. In: ACM SIGIR Forum vol. 36(2) pp. 3-10.
7. Buckley, Chris; Voorhees, Ellen (2005): Retrieval System Evaluation. In: TREC: Experiment and Evaluation in Information Retrieval. Cambridge & London: MIT Press. pp. 53-75.
8. Clarke, Charles; Craswell, Nick; Soboroff, Ian (2004). Overview of the TREC 2004 Terabyte Track. In: Buckland, Lori; Voorhees, Ellen (Eds.). The Thirteenth Text Retrieval Conference (TREC 2004) NIST Special Publication: SP 500-261. http://trec.nist.gov/pubs/trecl3/tl3_proceedings.ht ml
9. Cleverdon, Cyril (1997). The Cranfield Tests on Index Language Devices. In: Sparck-Jones, Karen; Willett, Peter (Eds.): Readings in Information Retrieval. Morgan Kaufman, pp. 47-59.
10. Clough, Paul; Müller, Henning; Deselaers, Thomas; Grubinger, Michael; Lehmann, Thomas; Jensen, Jeffery; Hersh, William (2005). The CLEF 2005 Cross-Language Image Retrieval Track. In: Working Notes of the 6th Workshop of the Cross-Language Evaluation Forum, CLEF. Sep. 2005, Vienna, Austria, http://www.clef-campaign.org/
11. Cormack, Gordon (2006). TREC 2006 Spam Track Overview. In: Voorhees & Buckland (2006)
12. Craswell, Nick; Hawking, David; Wilkinson, Ross; Wu, Mingfang (2004). Overview of the TREC 2003 Web Track. In: Proceedings Text Retrieval Conference (TREC). http://trec.nist.gov/pubs/trecl2/tl2_proceedings.ht ml
13. Craswell, Nick; Hawking, David (2004). Overview of the TREC-2004 Web Track. In: Voorhees & Buckland 2004.
14. Downie, Stephan (2003). Toward the Scientific Evaluation of Music Information Retrieval Systems. In: Intl Symposium on Music Information Retrieval (ISMIR) Washington, D.C., & Baltimore, USA. http://ismir2003.ismir.net/papers/Downie.PDF
15. th International Conference on Music Information Retrieval (ISMIR) London, UK, 11-15 Sept. pp. 320-323.
16. Eguchi, Koji; Oyama, Keizo; Aizawa, Akiko; Ishikawa, Haruko (2004). Overview of the Informational Retrieval Task at NTCIR-4 WEB. In: NTCIR Workshop 4 Meeting Working Notes. http://research.nii.ac.jp/ntcir-ws4/ NTCIR4-WN/index.html
17. Fuhr, Norbert (2003). Initiative for the Evaluation of XML Retrieval (INEX): INEX 2003 Workshop Proceedings, Dagstuhl, Germany, December 15-17. http://purl.oclc.org/NET/duett-07012004-093151
18. th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain, Revised Selected Papers. Berlin et al.: Springer [LNCS 4730] pp. 852-876.
19. th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria, Revised Selected Papers. Berlin et al.: Springer [LNCS 4022] pp. 251-262.
20. th annual international conference on Research and development in information retrieval (SIGIR). pp. 528-529.
21. Hersh, William; Bhuptiraju, Ravi; Ross, Laura; Johnson, Phoebe; Cohen, Aaron; Kraemer, Dale (2004). TREC 2004 Genomics Track Overview. In: Buckland, Lori; Voorhees, Ellen (Eds.). The Thirteenth Text Retrieval Conference (TREC 2004) NIST Special Publication: SP 500-261. http://trec.nist.gov/pubs/trec 13/t13_proceedings.html
22. Hersh, William; Cohen, Aaron; Roberts, Phoebe; Rekapalli, Hari Krishna (2006). TREC 2006 Genomics Track Overview. In: Voorhees & Buckland (2006)
23. Kando, Noriko and Evans, David (2007). Proceedings of the Sixth NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering, and Cross-Lingual Information Access. National Institute of Informatics, Tokyo, Japan. http://research.nii.ac.jp/ ntcir/workshop/OnlineProce edings6/NTCIR/index.html
24. th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway. Revised Selected Papers. Springer: LNCS 3237. pp. 376-390
25. th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria, Revised Selected Papers Berlin: Springer [LNCS 4022] pp. 222-225.
26. Mandl, Thomas; Womser-Hacker, Christa (2005). The Effect of Named Entities on Effectiveness in Cross-Language Information Retrieval Evaluation. In: Proceedings ACM SAC Symposium on Applied Computing (SAC). Santa Fe, New Mexico, USA. March 13.-17. pp. 1059-1064.
27. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, Revised Selected Papers. Berlin et al.: Springer [LNCS] to appear. Preprint at: http://www.clef-campaign.org
28. Mandl, Thomas; Womser-Hacker, Christa; Ferro, Nicola; Di Nunzio, Giorgio (2008). How Robust are Multilingual Information Retrieval Systems? In: Proceedings ACM Symposium on Applied Computing (SAC) Fortaleza, Brazil, pp. 1132-1136.
29. McNamee, Paul; Mayfield, James (2004). Character N-Gram Tokenization for European Language Text Retrieval. In: Information Retrieval, vol. 7 (1/2). pp. 73-98.
30. Mittal, Ankush (2006). An Overview of Multimedia Content-Based Retrieval Strategies. In: Informatica 30. pp. 347-356.
31. th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain, Revised Selected Papers. Berlin et al.: Springer [LNCS 4730] pp. 21-34.
32. th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, Revised Selected Papers. Berlin et al.: Springer [LNCS] to appear. Preprint: http://www.clef-campaign.org
33. th ACM International Conference on Information and Knowledge Management (CIKM) Arlington, Virginia, USA. pp. 872-877
34. Sanderson, Mark; Zobel, Justin (2005). Information retrieval system evaluation: effort, sensitivity, and reliability. In: Proceedings of the 28th annual international A CM SIGIR conference on Research and development in information retrieval (SIGIR 2005) Salvador, Brazil. ACM Press, pp. 162 - 169.
35. th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria, Revised Selected Papers. Berlin et al.: Springer [LNCS 4022] pp. 810-824.
36. Sigurbjörnsson, Börkur; Kamps, Jaap; de Rijke, Maarten (2005). Blueprint of a Cross-Lingual Web Retrieval Collection. In: Journal of Digital Information Management, vol. 3 (1) pp. 9-13.
37. Smeaton, Alan (2005). Large Scale Evaluations of Multimedia Information Retrieval: The TRECVid Experience. In: CIVR 2005 - International Conference on Image and Video Retrieval, Springer: LNCS 3569, pp 11-17.
38. th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), Seattle, Washington, USA, August 6-11. ACM Press, pp. 11-18.
39. Oard, Douglas W.; Wang, Jianqiang; Jones, Gareth; White, Ryen; Pecina, Pavel; Soergel, Dagobert; Huang, Xiaoli; Shafran, Izhak (2006). Overview of the CLEF-2006 Cross-Language Speech Retrieval Track. In: Nardi, Alessandro; Peters, Carol; Vicedo, José Luis (Eds.): CLEF 2006 Working Notes. http://www.clef-campaign.org/2006/working notes
40. Ounis, Iadh; Rijke, Maarten; Macdonald, Craig; Mishne, Gilad; Soboroff, Ian (2006). Overview of the TREC-2006 Blog Track. In: Voorhees & Buckland (2006)
41. Oyama, Keizo; Ishida, Emi; Kando, Noriko (2002) (eds.). NTCIR Workshop 3: Proceedings of the Third NTCIR Workshop on research in Information Retrieval, Automatic Text Summarization and Question Answering (Sept 2001-Oct 2002) http://research.nii.ac.jp/ntcir/workshop/OnlineProce edings3/index.html
42. th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria, Revised Selected Papers. Berlin et al.: Springer [LNCS 4022] pp. 307-331.
43. Voorhees, Ellen; Buckland, Lori (2006) (eds.). The Fifteenth Text REtrieval Conference Proceedings (TREC 2006) NIST Special Publication SP 500-272. National Institute of Standards and Technology. Gaithersburg, Maryland. Nov. 2006. http://trec.nist.gov/pubs/trec15/
44. Voorhees, Ellen (2005). The TREC robust retrieval track. In: ACM SIGIR Forum 39 (1) pp. 11-20.
45. Voorhees, Ellen (2006). Overview of TREC 2006. In: Voorhees & Buckland (2006)