Examining the robustness of evaluation metrics for patent retrieval with incomplete relevance judgements

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6360 LNCS, Issue , 2010, Pages 82-93

  Magdy, Walid ^a   Jones, Gareth J F ^a  

^a DUBLIN CITY UNIVERSITY   (Ireland)

Author keywords

[No Author keywords available]

Indexed keywords

DATA SETS; EVALUATION METRICS; KEY ISSUES; ONE DIMENSION; PATENT RETRIEVAL; PATENT SEARCH; RELEVANCE JUDGEMENT; RETRIEVAL TECHNIQUES; SIGNIFICANT FACTORS; STANDARD EVALUATIONS; STANDARD MAP;

INFORMATION RETRIEVAL; LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; STANDARDS;

PATENTS AND INVENTIONS;

EID: 78049400575     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-15998-5_10     Document Type: Conference Paper

References (18)

1. Aslam, J. A., Yilmaz, E.: Estimating average precision with incomplete and imperfect judgments. In: Proceedings of the 15th ACM international conference on Information and knowledge management CIKM, Arlington, Virginia, USA, pp. 102-111 (2006)
2. Baeza-Yates, J., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)
3. Bompad, T., Chang, C.-C., Chen, J., Kumar, R., Shenoy, R.: On the robustness of relevance measures with incomplete judgements. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands. ACM, New York (2007)
4. Buckley, C., Voorhees, E.: Evaluating evaluation measure stability. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 33-40. ACM, New York (2000)
5. Buckley, C., Voorhees, E. M.: Retrieval evaluation with incomplete information. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, South Yorkshire, UK, pp. 25-32 (2004)
6. Buckley, C., Dimmick, D., Soboroff, I., Voorhees, E.: Bias and the limits of pooling. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA, pp. 619-620. ACM, New York (2006)
7. Fujii, A., Iwayama, M., Kando, N.: Overview of patent retrieval task at NTCIR-4. In: Proceedings of the Fourth NTCIR Workshop on Evaluation of Information Retrieval, Automatic Text Summarization and Question Answering, Tokyo, Japan (2004)
8. Fujii, A., Iwayama, M., Kando, N.: Overview of the patent retrieval task at the NTCIR-6 workshop. In: Proceedings of the 6th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Crosslingual Information Access, Tokyo, Japan, pp. 359-365 (2007)
9. Graf, E., Azzopardi, L.: A methodology for building a patent test collection for prior art search. In: Proceedings of The Second International Workshop on Evaluating Information Access (EVIA 2008), Tokyo, Japan (2008)
10. Iwayama, M., Fujii, A., Kando, N., Takano, A.: Overview of patent retrieval task at NTCIR-3. In: Proceedings of the 3rd NTCIR Workshop on Evaluation of Information Retrieval, Automatic Text Summarization and Question Answering, Tokyo, Japan (2003)
11. Kendall, M.: A new measure of rank correlation. Biometrika 30 (1/2), 81-93 (1938)
12. Leong, M. K.: Patent Data for IR Research and Evaluation. In: Proceedings of the 2nd NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-lingual Information Access, Tokyo, Japan, pp. 359-365 (2001)
13. Magdy, W., Jones, G. J. F.: PRES: a score metric for evaluating recall-oriented information retrieval applications. In: Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva, Switzerland. ACM, New York (2010)
14. Van Rijsbergen, C. J.: Information Retrieval, 2nd edn. Butterworths (1979)
15. Robertson, S. E.: The parametric description of the retrieval tests. Part 2: Overall measures. Journal of Documentation 25(2), 93-107 (1969)
16. Roda, G., Tait, J., Piroi, F., Zenz, V.: CLEF-IP 2009: Retrieval experiments in the Intellectual Property domain. In: CLEF 2009 Working Notes, Corfu, Greece (2009)
17. Voorhees, E. M.: Special Issue: The Sixth Text REtrieval Conference (TREC-6). Information Processing and Management, 36 (1) (2000)
18. Voorhees, E. M.: Evaluation by highly relevant documents. In: Proceedings of the 24th ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, U. S. A., pp. 74-82 (2001)