Evaluation of system measures for incomplete relevance judgment in IR

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 4027 LNAI, Issue , 2006, Pages 245-256

  Wu, Shengli ^a   McClean, Sally ^a  

^a UNIVERSITY OF ULSTER   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DATA REDUCTION; NUMERICAL METHODS; QUERY LANGUAGES;

GRADED RELEVANCE JUDGMENT; LEAST SENSITIVE MEASURES; NORMALIZED DISCOUNT CUMULATIVE GAIN;

INFORMATION RETRIEVAL;

EID: 33746191446     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/11766254_21     Document Type: Conference Paper

References (17)

1. Aslam, J. A. and Yilmaz, E.: A geometric interpretation and analysis of R-precision. In Proceedings of ACM CIKM'2005, pages 664-671, Bremen, Germany, October-November.
2. Aslam, J. A. and Yilmaz, E. and Pavlu, V.: The maximum entropy method for analyzing retrieval measures. In Proceedings of ACM SIGIR'2005, pages 27-34, Salvador, Brazil.
3. Barry, C. L.: User-defined relevance criteria: an exploratory study. Journal of the American Society for Information Science, 45(3): 149-159, 1994.
4. Bodoff, D. and Robertson, S.: A new united probabilistic model. Journal of the American Society for Information Science and Technology, 55(6):471-487, 2004.
5. Buckley, C. and Voorhees, E. M.: Evaluating evaluation measure stability. In Proceedings of ACM SIGIR'2000, pages 33-40, Athens, Greece.
6. Buckley, C. and Voorhees, E. M.: Retrieval evaluation with incomplete information. In Proceedings of ACM SIGIR'2004, pages 25-32, Sheffield, United Kingdom.
7. Järvelin, K. and Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):442-446, 2002.
8. Sparck Jones, K. and van Rijisbergen, C.: Report on the need for and provision of an "ideal" information retrieval test collection. Technical report, British library research and development report 5266, Computer laboratory, University of Cambridge, Cambridge, UK, 1975.
9. Kekäläinen, J.: Binary and graded relevance in IR evaluations - comparison of the efforts on ranking of IR systems. Information Processing & Management, 41(5):1019-1033, 2005.
10. Lee, C. and Lee, G. G.: Probabilistic information retrieval model for a dependency structured indexing system. Information Processing & Management, 41(2):161-175, 2005.
11. Saracevic, T.: Relevance: A review of and a framework for thinking on the notion in information science. Journal of the American Society for Information Science, 26(6):321-343. 1975.
12. Schamber, L. and Eisenberg, M. B. and Nilan, M. S.: A re-examination of relevance: toward a dynamic, situational definition. Information Processing & Management, 26(6):755-776, 1990.
13. Sanderson, M. and Zobel, J.: Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of ACM SIGIR'2005, pages 162-169, Salvador, Brazil.
14. Voorhees, E. M. and Buckley C.: The effect of topic set size on retrieval experiment error. In Proceedings of ACM SIGIR'2002, pages 316-323, Tampere, Finland.
15. Voorhees, E. M. and Harman, D.: Overview of the sixth text retrieval conference (trec-6). Information Processing & Management, 36(1):3-35:2000.
16. Xu, Y. and Benaroch, M.: Information retrieval with a hybrid automatic query expansion and data fusion procedure. Information Retrieval, 8(1):41-65, 2005.
17. Zobel, J.: How reliable are the results of large-scale information retrieval experiments. In Proceedings of ACM SIGIR'1998, pages 307-314, Melbourne, Australia.