On effectiveness measures and relevance functions in ranking INEX systems

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 3689 LNCS, Issue , 2005, Pages 312-327

  Vu, Huyen Trang ^a   Gallinari, Patrick ^a  

^a UNIVERSITÉ PIERRE ET MARIE CURIE   (France)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER PROGRAMMING LANGUAGES; FUNCTIONS; PARAMETER ESTIMATION; Q FACTOR MEASUREMENT; XML;

INEX; Q-MEASURE [18]; REASONABLENESS;

INFORMATION RETRIEVAL SYSTEMS;

EID: 33646155934     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/11562382_24     Document Type: Conference Paper

References (26)

1. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In ACM SIGIR'00, pages 33 - 40, Athens, Greece, 2000. ACM Press.
2. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Sanderson et al. [19], pages 25-32.
3. W. S. Cooper. Expected Search Length: A Single Measure of Retrieval Effectiveness Based on the Weak Ordering Action of Retrieval Systems. American Documentation, 19(1):30-41, 1968.
4. A. C. Davison and D. V. Hinkley. Bootstrap Methods and Their Application. Cambridge University Press, 1997.
5. A. P. de Vries, G. Kazai, and M. Lalmas. Evaluation metrics 2004. In INEX 2004 Workshop Pre-Proceedings, pages 249-250, 2004. Available at http://inex.is.Informatik.uni-duisburg.de:2004/pdf/INEX2004PreProceedings.pdf.
6. A. P. de Vries, G. Kazai, and M. Lalmas. Tolerance to Irrelevance: A User-effort Oriented Evaluation of Retrieval Systems without Predefined Retrieval Unit. In RIAO 2004, pages 463-473, Avignon, France, Apr. 2004.
7. D. Hawking and S. Robertson. On collection size and retrieval effectiveness. Information Retrieval, 6(1):99-105, 2003.
8. D. A. Hull, P. Kantor, and K. Ng. Advanced approaches to the statistical analysis of TREC information retrieval experiments. Technical report, 1997. Unpublished, contact the first author for a copy: hull@clairvoyancecorp.com.
9. G. Kazai, M. Lalmas, and A. P. de Vries. The overlap problem in content-oriented XML retrieval evaluation. In Sanderson et al. [19], pages 72-79.
10. G. Kazai, M. Lalmas, and A. P. de Vries. Reliability Tests for the XCG and inex-2002 Metric. In N. Fuhr, M. Lalmas, S. Malik, and Z. Szlávik, editors, Advances in XML Information Retrieval. Third Workshop of the INitiative for the Evaluation of XML Retrieval INEX 2004, LNCS 3493, pages 60-72, 2004.
11. G. Kazai, M. Lalmas, N. Fuhr, and N. Gövert. A report on the first year of the INitiative for the evaluation of XML retrieval (INEX'02). Journal of the American Society for Information Science and Technology (JASIST), 55(6):551-556, 2004.
12. J. Kekäläinen and K. Järvelin. Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology (JASIST), 53(13): 1120-1129, 2002.
13. W. Kraaij. Variations on Language Modeling for Information Retrieval. PhD thesis, University of Twente, 2004.
14. V. D. Mea and S. Mizzaro. Measuring retrieval effectiveness: a new proposal and a first experimental validation. Journal of the American Society for Information Science and Technology (JASIST), 55(6):530-543, 2004.
15. S. H. Myaeng, D.-H. Jang, M.-S. Kim, and Z.-C. Zhoo. A Flexible Model for Retrieval of SGML documents. In SIGIR '98, pages 138-140, Melbourne, Australia, Aug. 1998.
16. V. V. Raghavan, G. S. Jung, and P. Bollmann. A critical investigation of recall and precision as measures of retrieval system performance. ACM Transactions on Information Systems, 7(3):205-229, 1989.
17. T. Sakai. New Performance metrics based on Multigrade Relevance: Their Application to Question Answering. In NTCIR-4 Proceedings, 2004.
18. T. Sakai. Ranking the NTCIR Systems Based on Multigrade Relevance. In S. H. Myaeng, M. Zhou, K.-F. Wong, and H.-J. Zhang, editors, Information Retrieval Technology, Asia Information Retrieval Symposium, AIRS 2004, Revised Selected Papers, pages 251-262, 2004.
19. M. Sanderson, K. Järvelin, J. Allan, and P. Bruza, editors. SIGIR 2004: Proc. of the 27th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Sheffield, UK, July 25-29, 2004, 2004.
20. M. Sanderson and J. Zobel. Information retrieval system evaluation: Effort, sensitivity, and reliability. In ACM SIGIR'05, 2005. To appear.
21. J. Savoy. Statistical inference in retrieval effectiveness evaluation. Info. Process. Management, 33(4):495-512, 1997.
22. I. Soboroff. On evaluating web search with very few relevant documents. In Sanderson et al. [19], pages 530-531.
23. J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In Proceedings of TREC-3, NIST Special Publication 500-225, pages 385-398, Apr. 1995.
24. C. J. Van Rijsbergen. Information Retrieval. Butterworths, 1979.
25. E. M. Voorhees. The TREC robust retrieval track. SIGIR Forum. 39(1):11-20, 2005.
26. E. M. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In ACM SIGIR'02, pages 316-323. ACM Press, Aug. 2002.