Binary and graded relevance in IR evaluations-Comparison of the effects on ranking of IR systems

Information Processing and Management

Volumn 41, Issue 5, 2005, Pages 1019-1033

  Kekäläinen, Jaana ^a  

^a Department of Information Studies ^*  (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

CORRELATION METHODS; DATA PROCESSING; DATA REDUCTION; GRAPH THEORY; INDEXING (OF INFORMATION); TEXT PROCESSING;

BINARY PRECISION; CUMULATED GAIN; RANK CORRELATIONS; WEIGHTING SCHEMES;

INFORMATION RETRIEVAL;

EID: 16244362331     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2005.01.004     Document Type: Article

References (30)

1. C.L. Barry User-defined relevance criteria: an exploratory study Journal of the American Society for Information Science 45 3 1994 149 159
2. P. Borlund Experimental components for the evaluation of interactive information retrieval systems Journal of Documentation 56 1 2000 71 90
3. Borlund, P., & Ingwersen, P. (1998). Measures of relative relevance and ranked half-life: performance indicators for interactive IR. In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, & J. Zobel (Eds.), Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp. 324-331
4. R. Burgin Variations in relevance judgements and the evaluation of retrieval performance Information Processing and Management 28 5 1992 619 627
5. Buckley, C., & Voorhees, E. M. (2000). Evaluating evaluation measure stability. In N. J. Belkin, P. Ingwersen, & M.-K. Leong (Eds.), Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval (pp. 33-40). ACM, New York
6. W.J. Conover Practical nonparametric Statistics 2nd ed. 1980 Wiley New Yok
7. W.S. Cooper A Definition of Relevance for Information Retrieval Information Storage and Retrieval 7 1 1971 19 37
8. E. Cosijn, and P. Ingwersen Dimensions of relevance Information Processing and Management 36 2000 533 550
9. C.A. Cuadra, and R.V. Katter Experimental studies of relevance judgments: final report Project summary Vol. I 1967 System Development Corporation Santa Monica
10. Hull, D. (1993). Using statistical testing in the evaluation of retrieval experiments. In R. Korfhage, E. M. Rasmussen & P. Willett (Eds.), Proceedings of the 16th international conference on research and development in information retrieval (pp. 349-338). ACM, New York, NY
11. Järvelin, K., & Kekäläinen, J. (2000). IR evaluation methods for highly relevant documents. In N. J. Belkin, P. Ingwersen, & M.-K. Leong (Eds.), Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval (pp. 41-48). ACM, New York
12. K. Järvelin, and J. Kekäläinen Cumulated gain-based evaluation of IR techniques ACM Transactions on Information Systems 20 4 2002 442 446
13. E.M. Keen Presenting results of experimental retrieval comparisons Information Processing and Management 28 4 1992 491 501
14. Kekäläinen, J., & Järvelin, K. (2002). Evaluating information retrieval systems under the challenges of interaction and multidimensional dynamic relevance. In H. Bruce, R. Fidel, P. Ingwersen, & P. (Eds.) Proceedings of the CoLIS 4 conference (pp. 253-270). Greenwood Village, Libraries Unlimited
15. R.R. Korfhage Information storage and retrieval 1997 Wiley & Sons New York
16. M.E. Lesk, and G. Salton Relevance assessments and retrieval system evaluation Information Storage and Retrieval 4 4 1968 343 359
17. S.M. Pollack Measures for the comparison of information retrieval systems American Documentation 19 4 1968 387 397
18. S.E. Robertson The probability ranking principle in IR Journal of Documentation 33 4 1977 294 304
19. S.E. Robertson, and N.J. Belkin Ranking in principle Journal of Documentation 34 2 1978 93 100
20. Sakai, T. (2003). Average gain ratio: a simple retrieval performance measure for evaluation with multiple relevance levels. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (pp. 417-418). ACM, New York
21. T. Saracevic Relevance: A review of and framework for the thinking on the notion in information science Journal of the American Society for Information Science 26 6 1975 321 343
22. Saracevic, T. (1996). Relevance reconsidered '96. In P. Ingwersen & N. O. Pors (Eds.), Proceedings of the second international conference on conceptions of library and information science: integration in perspective (pp. 201-218). The Royal School of Librarianship, Copenhagen
23. Schamber, L. (1994). Relevance and information behavior. In M. E. Williams (Ed.), Annual review of information science and technology (pp. 3-48), Vol. 29, Information Today, Medford
24. L. Schamber, M.B. Eisenberg, and M.S. Nilan A re-examination of relevance: toward a dynamic, situational definition Information Processing and Management 26 6 1990 755 776
25. Sormunen, E. (2002). Liberal relevance criteria of TREC-Counting on negligible documents? In M. Beaulieu, R. Baeza-Yates, & S. H. Myaeng (Eds.), Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval (pp. 324-330). ACM, New York
26. R. Tang, W.M. Shaw, and J.L. Vevea Towards the identification of the optimal number of relevance categories Journal of the American Society for Information Science 50 3 1999 254 264
27. Voorhees, E. (1998). Variations in relevance judgements and the measurement of retrieval effectiveness. In: W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, & J. Zobel (Eds.), Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval (pp. 315-323). ACM, New York
28. E. Voorhees Variations in relevance judgements and the measurement of retrieval effectiveness Information Processing and Management 36 2000 697 716
29. Voorhees, E. (2001). Evaluation by highly relevant documents. In W. B. Croft, D. J. Harper, D. H. Kraft, & J. Zobel (Eds.), Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (pp. 74-82). ACM, New York
30. Voorhees, E., & Harman, D. (1999). Overview of the eight Text REtrieval Conference (TREC-8). In E. Voorhees & D. Harman (Eds.), Proceedings of the eight text retrieval conference (TREC-8), NIST Special Publication 500-246 (pp. 1-24). Retrieved September 30, 2003. Available from 〈http://trec.nist.gov/pubs/trec8/papers/overview_8.pdf〉.