Liberal relevance criteria of TREC - Counting on negligible documents?

SIGIR Forum (ACM Special Interest Group on Information Retrieval)

Volumn , Issue , 2002, Pages 324-330

  Sormunen, Eero ^a  

^a TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

Author keywords

Experiment; Graded relevance; Relevance assessments; Test collections; TREC

Indexed keywords

DATA ACQUISITION; LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; SORTING; TEXT PROCESSING; WORLD WIDE WEB;

CORPUS; GRADED RELEVANCE; NEGLIGIBLE DOCUMENTS; RELEVANCE ASSESSMENT; TEST COLLECTIONS;

INFORMATION RETRIEVAL;

EID: 0036989640     PISSN: 01635840     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (14)

1. Blair, D.C. and Maron, M.E. An evaluation of retrieval effectiveness for a full-text document retrieval system. Commun. ACM 28, 3, 289-299.
2. Burgin, R. Variations on relevance judgments and the evaluation of retrieval performance. Information Processing & Management 28, 5, 619-627.
3. Gordon, M. and Pathak, P. Finding information on the world wide web: The retrieval effectiveness of search engines. Information Processing and Management, 35, 2, 141 - 180.
4. Hawking, D. Overview of the TREC-9 Web Track. http://trec.nist.gov/pubs/trec9/papers/web9.pdf [Cited 2 January 2002].
5. Järvelin, K. & Kekäläinen, J. IR evaluation methods for highly relevant documents. In: Belkin, N.J., Ingwersen, P. and Leong, M.-K. (eds.) Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York 2000, 41-48.
6. Saracevic, T., Kantor, P. et al. A study of information seeking and retrieving. I Background and methodology. Journal of the American Society for Information Science 39, 3, 161-176.
7. Sormunen, E. A method for measuring wide range performance of Boolean queries in full-text databases. Doctoral Thesis. University of Tampere. Acta Electronica Universitatis Tamperensis 34, URL: http://acta.uta.fi/teos.phtml?3786, Tampere 2000.
8. Sormunen, E., Kekäläinen, J., Koivisto, J. & Järvelin, K. Document text characteristics affect the ranking of the most relevant documents by expanded structured queries. Journal of Documentation 57, 3, 358-374.
9. TREC homepage, Data - English relevance judgements. Available at: http://trec.nist.gov/data/reljudge_eng.html [Cited 31 December 2001].
10. Voorhees, E. Variations in relevance judgments and the measurement of retrieval effectiveness. In: Croft, W.B. et al. (eds.), Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, August 1998). ACM Press, New York., 315-323.
11. Voorhees, E. Evaluation by highly relevant documents. In: Croft, W.B. et al. (eds.). Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New Orleans, September 2001). ACM Press, New York, 74-82.
12. Voorhees, E. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management 36, (2000), 697-716.
13. Voorhees, E. & Harman, D. Overview of the Seventh Text Retrieval Conference (TREC-7). 1999. http://trec.nist.gov/pubs/trec7/papers/overview_7.pdf.gz [Cited 2 January 2002].
14. Zobel, J. How reliable are the results of large-scale information retrieval experiments? In: Croft, W.B. et al. (eds.), Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, August 1998). ACM Press, New York.