Pooling for a large-scale test collection: An analysis of the search results from the first NTCIR Workshop

Information Retrieval

Volumn 5, Issue 1, 2002, Pages 41-59

  Kuriyama, Kazuko ^a   Kando, Noriko ^a   Nozue, Toshihiko ^b   Eguchi, Koji ^a  

^a NATIONAL INSTITUTE OF INFORMATICS   (Japan)

^b AOYAMA GAKUIN UNIVERSITY   (Japan)

Author keywords

Evaluation methodology; Exhaustiveness of pooling; IR system; Relevance assessment; Test collection

Indexed keywords

EID: 4043095423     PISSN: 13864564     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1012778807438     Document Type: Article

References (12)

1. Buckley C and Voorhees E (1999) Tutorial: Theory and practice in text retrieval system evaluation. In: Tutorial in ACM-SIGIR'99. Beikeley, CA, USA. pp. 1-109.
2. Cormack GV, Palmer CR and Clarke CLA (1998) Efficient construction of large test collections. In: Proceedings of the ACM-SIGIR'98. Melbourne, Australia, pp. 282-289.
3. Gilbert G and Sparck Jones K (1979) Statistical bases of relevance assessment for the 'Ideal' information retrieval test collection. BL R&D Report 5481. Cambridge, England.
4. Kageura K, Koyama T, Yoshioka M, Takasu A, Nozue T and Tsuji K (1997) NACSIS corpus project for IR and terminological research. In: Proceedings of the Natural Language Processing Pacific Rim Symposium 1997, Phuket, Thailand, pp. 493-496.
5. Kando N, Kunyama K and Nozue T (1999a) NTCIR-1 (NACSIS Test Collection for Information Retrieval Systems-1). Its Policy and Practice. IPSJ SIG Notes, 99-FI-53-5:33-40. (In Japanese).
6. Kando N, Kuriyama K, Nozue T, Eguchi K, Kato H and Hidaka S (1999b) Overview of IR tasks at the first NTCIR workshop. In: Proceedings of the NTCIR Workshop 1. Tokyo, Japan, pp. 11-14.
7. Kando N and Nozue T (1999), Eds. NTCIR Workshop 1: Proceedings of the First NTCIR Workshop on Retrieval in Japanese Text Retrieval and Term Recognition, Tokyo, Japan, http://research.nii.ac.jp/ntcir/ workshop/OnlineProceedings/(visited March 24th. 2001).
8. Kuriyama K, Eguchi K, Nozue T and Kando N (1999) NACSIS test collection for information retrieval systems-1 (1): Analysis of the pooling and the relevance assessments. In: Proceedings of the IPSJ Annual Meeting, Morioka, Japan, pp. 3,105-106 (In Japanese).
9. NTCIR (NACSIS Test Collection for IR Systems) Project, http://research.nii.ac.jp/ntcir/ (visited March 24th, 2001).
10. Voothees EM (1998) Variations in relevance judgments and the measurement of retrieval effectiveness. In: Proceedings of the ACM-SIGIR '98. Melbourne, Australia, pp 315-332.
11. Voorhees EM and Harman D (2000). Eds. The Eighth Text REtrieval Conference (TREC-8). NIST Special Publication 500-246, Maryland, U.S.A , Text REtrieval Conference (TREC). http://trec.nist.gov/(visited March 20th, 2001).
12. Zobel J (1998) How reliable are the results of large scale information retrieval experiments? In: Proceedings of the ACM-SIGIR'98, Melbourne. Australia, pp. 307-314.