Evaluation of retrieval effectiveness with incomplete relevance data: Theoretical and experimental comparison of three measures

Information Processing and Management

Volumn 44, Issue 1, 2008, Pages 212-225

  Ahlgren, Per ^a   Grönqvist, Leif ^b  

^a UNIVERSITY OF BORÅS   (Sweden)

^b LINNAEUS UNIVERSITY   (Sweden)

Author keywords

Cranfield; Incomplete judgments; Rank effectiveness; Retrieval effectiveness

Indexed keywords

DATA ACQUISITION; PROBLEM SOLVING;

INCOMPLETE JUDGMENTS; RANK EFFECTIVENESS; RETRIEVAL EFFECTIVENESS;

INFORMATION RETRIEVAL;

EID: 35548951792     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2007.01.011     Document Type: Article

References (16)

1. Ahlgren, P., & Grönqvist, L. (2006a). Retrieval evaluation with incomplete relevance data: a comparative study of three measures (poster abstract). In Proceedings of the 15th ACM international conference on information and knowledge management (pp. 872-873).
2. Ahlgren, P., & Grönqvist, L. (2006b). Measuring retrieval effectiveness with incomplete relevance data. In InSCit2006, Current research in information sciences and technologies: Multidisciplinary approaches to global information systems, Vol. I (pp. 74-78).
3. Buckley, C., & Voorhees, E. M. (2000). Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval (pp. 33-40).
4. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (pp. 25-32).
5. Cleverdon, C. W. (1991). The significance of the Cranfield tests on index languages. In Proceedings of the 14th annual international ACM SIGIR conference on research and development in information retrieval (pp. 3-12).
6. Grönqvist, L. (2005). Evaluating latent semantic vector models with synonym tests and document retrieval. In ELECTRA workshop: Methodologies and evaluation of lexical cohesion techniques in real-world applications beyond bag of words, in association with ACM SIGIR.
7. Hull, D. (1993). Using statistical testing in the evaluation of retrieval experiments. In Proceedings of the 16th annual international ACM SIGIR conference on research and development in information retrieval (pp. 329-338).
8. Järvelin K., and Kekäläinen J. Cumulated gain-based evaluation of IR techniques. ACM Transactions of Information Systems 20 4 (2002) 422-446
9. Keen E.M. Presenting results of experimental retrieval comparisons. Information Processing and Management 28 4 (1992) 491-502
10. Kendall M., and Gibbons J.D. Rank correlation methods. 5th ed. (1990), Edward Arnold
11. Sanderson, M., & Zobel, J. (2005). Information retrieval system evaluation: effort, sensitivity, and reliability. In Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (pp. 162-169).
12. Tague-Sutcliffe J. The pragmatics of information retrieval experimentation, revisited. Information Processing and Management 28 4 (1992) 467-490
13. Voorhees, E. M. (2002). The philosophy of information retrieval evaluation. In Evaluation of cross-language information retrieval systems. Proceedings of CLEF 2001 (pp. 355-370).
14. Voorhees, E. M. (2004). Overview of TREC 2003. In Proceedings of the twelfth text retrieval conference (TREC 2003) (pp. 1-13).
15. Voorhees, E. M., & Buckley, C. (2002). The effect of topic set size on retrieval experiment error. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (pp. 316-323).
16. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21th annual international ACM SIGIR conference on research and development in information retrieval (pp. 307-314).