Using graded relevance assessments in IR evaluation

Journal of the American Society for Information Science and Technology

Volumn 53, Issue 13, 2002, Pages 1120-1129

  Kekäläinen, Jaana ^a   Järvelin, Kalervo ^a  

^a TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

CALCULATIONS; DATA STRUCTURES; DATABASE SYSTEMS; INFORMATION RETRIEVAL SYSTEMS; INFORMATION SCIENCE; TEXT PROCESSING;

INFORMATION OVERLOAD; MULTIPLE GRADE RELEVANCE ASSESSMENTS; QUERY STRUCTURES;

INFORMATION RETRIEVAL;

EID: 0036851939     PISSN: 15322882     EISSN: None     Source Type: Journal    
DOI: 10.1002/asi.10137     Document Type: Article

References (39)

1. Allan, J.; Callan, J., Croft, B.; Ballesteros, L., Broglio, J., Xu, J., & Shu, H. (1997) Inquery at TREC 5. In E.M. Voorhees & D.K. Harman (Eds.), Information technology: The Fifth Text Retrieval Conference (TREC-5) (pp. 119-132). Gaithersburg, MD: National Institute of Standards and Technology.
2. Barry, C.L. (1994) User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science, 45 (3), 149-159.
3. Barry, C.L., & Schamber, L. (1998). Users' criteria for relevance evaluation: A cross-situational comparison. Information Processing & Management, 34(2/3), 219-236.
4. Bateman, J. (1998). Changes in relevance criteria: A longitudinal study. In Proceedings of the 61st American Society for Information Science annual meeting 35 (pp. 23-32).
5. Blair, D.C., & Maron, M.E. (1985). An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM, 28(3), 289-299.
6. Borlund, P. (2000a). Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation, 56(1), 71-90.
7. Borlund, P. (2000b). Evaluation of interactive information retrieval systems. Ph.D. dissertation. Åbo: Åbo University Press.
8. Cosijn, E., & Ingwersen, P. (2000). Dimensions of relevance. Information Processing and Management, 36, 533-550.
9. Conover, W.J. (1980). Practical nonparametric statistics (2nd ed.). New York: John Wiley & Sons.
10. Cuadra, C.A., & Katter, R.V. (1967). Experimental studies of relevance judgments: Final report. Vol. I: Project summary, Santa Monica, CA: System Development Corporation.
11. Eisenberg, M. (1988). Measuring relevance judgments. Information Processing and Management. 24(4), 373-389.
12. Green, R. (1995). The expression of conceptual syntagmatic relationships: A comparative survey. Journal of Documentation, 51(4), 315-338.
13. Hersh, W.R., & Hickman, D.H. (1995). An evaluation of interactive Boolean and natural language searching with an online medical textbook. Journal of the American Society for Information Science, 46(7), 478-489.
14. Ingwersen, P., & Willett, P. (1995). An introduction to algorithmic and cognitive approaches for information retrieval. Libri, 45, 160-177.
15. Järvelin, K., & Kek̈l̈inen, J. (2000). IR evaluation methods for retrieving highly relevant documents. In Belkin, N., Ingwersen, P., & Leong, M.-K. (Eds.), Proceedings of the 23th Annual International ACM SIGIR conference on research and development in information retrieval (pp. 41-48). New York: ACM Press.
16. Katter, R.V. (1968). The influence of scale form on relevance judgments. Information Storage and Retrieval, 4(1), 1-11.
17. Keen, E.M. (1991). The use of term position devices in ranked output experiments. Journal of Documentation, 47(1), 1-22.
18. Kekäläinen, J. (1999). The effects of query complexity, expansion and structure on retrieval performance in probabilistic text retrieval. Ph.D. dissertation, Department of Information Studies, University of Tampere [On-line]. Available: http://www.info.uta.fi/tutkimus/fire/archive/QCES.pdf
19. Kekäläinen, J. & J̈rvelin, K. (1998). The impact of query structure and query expansion on retrieval performance. In W.B. Croft, A. Moffat, C.J. van Rijsbergen, R. Wilkinson, & J. Zobel (Eds.) Proceedings of the 21st Annual International ACM SIGIR conference on research and development in information retrieval (pp. 130-137). New York: ACM.
20. Kekäläinen, J. & Järvelin, K. (2000). The co-effects of query structure and expansion on retrieval performance in probabilistic text retrieval. Information Retrieval, 1(4), 329-344.
21. Losee, R.M. (1998). Text retrieval and filtering: Analytic models of performance. Boston: Kluwer Academic Publishers.
22. Rajashekar, T.B., & Croft, W.B. (1995) Combining automatic and manual index representations in probabilistic retrieval. Journal of the American Society for Information Science, 46(4), 272-283.
23. Radecki, T. (Ed.). (1988). Special issue on the potential for improvements in commercial document retrieval systems. Information Processing and Management, 24(3), 219-371.
24. Rees, A.M., & Schultz, D.G. (1967). A field experimental approach to the study of relevance assessments in relation to document searching. Cleveland: Case Western Reserve University.
25. Saracevic, T. (1996). Relevance reconsidered '96. In P. Ingwersen & N.O. Pors (Eds.), Proceedings of the second international conference on conceptions of library and information science: Integration in perspective (pp. 201-218). Copenhagen: The Royal School of Librarianship.
26. Saracevic, T. Kantor, P. Chamis, A., & Trivison, D. (1988). A study of information seeking and retrieving. I. Background and methodology. Journal of the American Society for Information Science, 39(3), 161-176.
27. Schamber, L. (1994). Relevance and Information Behavior. In: M.E. Williams (Ed.), Annual review of information science and technology 29. Medford. NJ: Information Today (pp. 3-48).
28. Schamber, L., Eisenberg, M., & Nilan, M. (1990). A re-examination of relevance: Toward a dynamic, situational definition. Information Processing and Management, 26(6), 755-776.
29. Smithson, S. (1994). Information retrieval evaluation in practice: A case study approach. Information Processing & Management, 30(2), 205-221.
30. Sormunen, E. (2000). A method for measuring wide range performance of Boolean queries in full-text databases. Ph.D. dissertation. Department of Information Studies, University of Tampere [On-line]. Available: http://acta.uta.fi/pdf/951-44-4732-8.pdf.
31. Sormunen, E. (2002). Liberal relevance criteria of TREC-Counting on negligible documents? In M. Beaulieu, R. Baeza-Yates, S.H. Myaeng, & K. Järvelin (Eds.). Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM SIGIR '02), Tampere, Finland, August 11-15, 2002 (pp. 324-330). New York: ACM Press.
32. Spink, A., Greisdorf, H., & Bateman, J. (1998). From highly relevant to non-relevant: Examining different regions of relevance. Information Processing & Management, 34(5), 599-622.
33. Tang, R., Shaw, W.M., & Vevea, J.L. (1999). Towards the identification of the optimal number of relevance categories. Journal of the American Society for Information Science, 50(3), 254-264.
34. TREC (2001). TREC homepage, data-English relevance judgements [On-line]. Available: http://trec.nist.gov/data/reljudge_eng.html
35. Turtle, H.R. (1990). Inference networks for document retrieval. Ph.D. dissertation. Computer and information Science Department, University of Massachusetts.
36. Vakkari, P. (2001). Changes in search tactics and relevance judgements in preparing a research proposal: A summary of findings of a longitudinal study. Information Retrieval, 4(3/4), 295-310.
37. Vakkari, P., & Hakala, N. (2000). Changes in relevance criteria and rroblem stages in task performance. Journal of Documentation, 56(5), 540-562.
38. Wang, P., & Soergel, D. (1998). A cognitive model of document use during a research project. Study I. Document selection. Journal of the American Society for Information Science, 49(2), 115-133.
39. Voorhees, E. (2001). Evaluation by highly relevant documents. In W.B. Croft, D.J. Harper, D.H. Kraft, & J. Zobel (Eds.). Proceedings of the 24th annual International ACM SIGIR conference on research and development in information retrieval (pp. 74-82). New York: ACM Press.