Variations in relevance assessments and the measurement of retrieval effectiveness

Journal of the American Society for Information Science

Volumn 47, Issue 1, 1996, Pages 37-49

  Harter, Stephen P ^a  

^a INDIANA UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0001769424     PISSN: 00028231     EISSN: None     Source Type: Journal    
DOI: 10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3     Document Type: Article

References (54)

1. Barry, C. L. (1993). The identification of user criteria and document characteristics: Beyond the topical approach to information retrieval. Unpublished doctoral dissertation, Syracuse University.
2. Bellardo, T. (1985). An investigation of online searcher traits and their relationship to search outcome. Journal of the American Society for Information Science, 36, 241-250.
3. Borgman, C. L. (1989). All users of information retrieval systems are not created equal: An exploration into individual differences. Information Processing and Management, 25, 237-252.
4. Burgin, R. (1992). Variations in relevance judgments and the evaluation of retrieval performance. Information Processing and Management, 28, 619-627.
5. Cleverdon, C. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19, 173-193.
6. Cleverdon, C. W. (1970). The effect of variations in relevance assessments in comparative experimental tests of index languages (Cranfield Library Report No. 3). Cranfield, UK: Cranfield Institute of Technology.
7. Cleverdon, C. W. (1990). Questions of relevance. In D. Bawden (Ed.), User-oriented evaluation of information systems and services (pp. 165-171). Brookfield, VT: Gower.
8. Cleverdon, C., Mills, J., & Keen, M. (1966). Vol. 1: Design. Vol. 2: Results. Factors determining the performance of indexing systems. Cranfield, UK: Aslib Cranfield Research Project.
9. Cuadra, C. A., & Katter, R. V. (1967). Opening the black box of "relevance." Journal of Documentation, 23, 291-303.
10. Doyle, L. B. (1963). Is relevance an adequate criterion for retrieval system evaluation? In H. P. Luhn (Ed.), Automation and scientific communication, short papers. Pt. 2 (pp. 199-200). Washington, DC: American Documentation Institute.
11. Ellis, D. (1984). Theory and explanation in information retrieval research. Journal of Information Science, 8, 25-38.
12. Fenichel, C. H. (1981). Online searching: Measures that discriminate among users with different types of experiences. Journal of the American Society for Information Science, 32, 23-32.
13. Giulano, V. E., & Jones, P. E. (1966). Study and test of a methodology for laboratory evaluation of message retrieval systems. Interim report. Prepared under contract No. AF 19(628)-4067 by Arthur D. Little, Inc., Cambridge, MA.
14. Gull, C. D. (1956). Seven years of work on the organization of materials in the special library. American Documentation, 7, 320-329.
15. Harman, D. (1992). Evaluation issues in information retrieval. Information Processing and Management, 28, 439-440.
16. Harman, D. K. (1993). The first text retrieval conference (TREC-1). Washington, DC: National Institute of Standards and Technology. (NIST special publication No. 500-207).
17. Harman, D. K. (1994). The second text retrieval conference (TREC-2). Washington, DC: National Institute of Standards and Technology. (NIST special publication No. 500-215).
18. Harter, S. P. (1971). The Cranfield II relevance assessments: A critical evaluation. Library Quarterly, 41, 229-243.
19. Harter, S. P. (1990). Search term combinations and retrieval overlap: A proposed methodology and case study. Journal of the American Society for Information Science, 41, 132-146.
20. Harter, S. P. (1992). Psychological relevance and information science. Journal of the American Society for Information Science, 43, 602-615.
21. Information Processing and Management. (1992). 28(4).
22. Jackson, W. J. (1981). ONTAP-ERIC: A critical view. Online Review 5, 335-338.
23. Journal of the American Society for Information Science. (1994). 41(3).
24. Katzer, J., McGill, M. J., Tessier, J. A., Frakes, W., & DasGupta, P. (1982). A study of the overlap among document representations. Information Technology: Research and Development, 2, 261-274.
25. Kazhdan, T. V. (1979). Effects of subjective expert evaluation of relevance on the performance parameters of a document-based information-retrieval system. Nauchno-Tekhnicheskava Informatsiya, Seriya 2, 13, 21-24.
26. Kent, A. (1974). Unsolvable problems. In A. Debons (Ed.), Information science: Search for identity (Proceedings of the 1972 NATO Advanced Study Institute in Information Science held at Seven Springs, Champion, PA, August 12-20, 1972, pp. 299-311). New York: Marcel Dekker, Inc.
27. Lancaster, F. W. (1968). Evaluation of the MEDLARS demand search service. Washington, DC: National Library of Medicine.
28. Lesk, M. E., & Salton, G. (1968a). Relevance assessments and retrieval system evaluation. Information Storage and Retrieval, 4, 343-359.
29. Lesk, M. E., & Salton, G. (1968b). Relevance assessments and retrieval system evaluation. In G. Salton (Ed.), Information storage and retrieval. Scientific rep. No. IRS-14. Ithaca, NY: Department of Computer Science, Cornell University.
30. Markey, K. (1984). Interindexer consistency tests: A literature review and report of a test of consistency in indexing visual materials. Library and Information Science Research, 6, 155-177.
31. McCain, K. W. (1989). Descriptor and citation retrieval in the medical behavioral sciences literature: Retrieval overlaps and novelty distribution. Journal of the American Sociel for Information Science, 40, 110-114.
32. Meadow, C. (1985). Relevance? Journal of the American Society for Information Science, 36, 354-355.
33. Park, T. K. (1992). The nature of relevance in information retrieval: An empirical study. Unpublished doctoral dissertation, Indiana University.
34. Park, T. K. (1993). The nature of relevance in information retrieval: An empirical study. Library Quarterly, 63, 318-351.
35. Rees, A. M. (1965). The evaluation of retrieval systems. Report CSL: TR-5. Comparative Systems Laboratory, Center for Documentation and Communication Research, Western Reserve University, Cleveland, OH, July, 21, 1965. Paper presented at the 2nd Annual Conerence on Technical Information Center Administration organized by Drexel Institute of Technology, Philadelphia, PA, June 14-17, 1965.
36. Rees, A. M., Schultz, D. G., Baumanis, G., Marcus, S., Rothenberg, L., Saracevic, T., Stern, M., & Zull, C. (1967). A field experimental approach to the study of relevance assessments in relation to document searching. Final report to the National Science Foundation. (NSF contract No. C-423, Case Western Reserve University, Cleveland, OH)
37. Robertson, S. E., & Hancock-Beaulieu, M. M. (1992). On the evaluation of IR systems. Information Processing and Management, 28, 457-466.
38. Salton, G. (1992). The state of retrieval system evaluation. Information Processing and Management, 28, 441-449.
39. Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26, 321-343.
40. Saracevic, T. (1991). Individual differences in organizing, searching and retrieving information. Proceedings of the Annual Meeting of the American Society for Information Science, 28, 82-86.
41. Saracevic, T., & Kantor, P. (1988). A study of information seeking and retrieving. III. Searchers, searches and overlap. Journal of the American Society for Information Science, 39, 197-216.
42. Schamber, L. (1991). Users' criteria for evaluation in multimedia information seeking and use situations. Unpublished doctoral dissertation, Syracuse University.
43. Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3-48.
44. Schamber, L., Eisenberg, M. B., & Nilan, M. S. (1990). A re-examination of relevance: Toward a dynamic, situational definition. Information Processing and Management, 26, 755-776.
45. Shaw, W. M., Jr., Wood, J. B., Wood, R. E., & Tibbo, H. R. (1991). The cystic fibrosis database: Content and research opportunities. Library and Information Science Research, 13, 347-366.
46. Smeaton, A. (1993, November 5). Report on TREC-2 Conference. [online] IR Digest, Vol. X, No. 42, Issue 186. Available via anonymous FTP via host dla.ucop.edu in directory pub/irl/1993.
47. Smithson, S. (1994). Information retrieval evaluation in practice: A case study approach. Information Processing and Management, 30, 205-221.
48. Swanson, D. R. (1971). Some unexplained aspects of the Cranfield tests of indexing performance factors. Library Quarterly, 41, 223-228.
49. Swanson, D. R. (1977). Information retrieval as a trial-and-error process. Library Quarterly, 47, 128-148.
50. Swanson, D. R. (1986). Subjective versus objective relevance in bibliographic retrieval systems. Library Quarterly, 56, 389-398.
51. Swanson, D. R. (1988). Historical note: Information retrieval and the future of an illusion. Journal of the American Society for Information Science, 39, 92-98.
52. Tague-Sutcliffe, J. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing and Management, 28, 467-490.
53. Wang, P. (1994). A cognitive model of document selection of real users of IR systems. Unpublished doctoral dissertation, University of Maryland.
54. Wilson, P. (1973). Situational relevance. Information Storage and Retrieval, 9, 457-469.