PRES: A score metric for evaluating recall-oriented information retrieval applications

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 611-618

  Magdy, Walid ^a   Jones, Gareth J F ^a  

^a TRINITY COLLEGE DUBLIN   (Ireland)

Author keywords

Evaluation metric; Patent retrieval; PRES; Recall oriented information retrieval

Indexed keywords

EVALUATION MEASURES; EVALUATION METRICS; PATENT RETRIEVAL; PERSPECTIVE TAKING; PRECISION AND RECALL; PRES; PRIMARY EVALUATION; RECALL-ORIENTED INFORMATION RETRIEVAL; RELEVANT DOCUMENTS; RETRIEVAL APPLICATIONS; RETRIEVAL EFFECTIVENESS;

NATURAL LANGUAGE PROCESSING SYSTEMS; PATENTS AND INVENTIONS;

INFORMATION RETRIEVAL;

EID: 77956042082     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835551     Document Type: Conference Paper

References (32)

1. Ali M. S., Consens, M. P., Kazai, G., and Lalmas, M. Structural relevance: A common basis for the evaluation of structured document retrieval. In Proceedings of CIKM '08, pages 1153-1162, 2008.
2. Aslam J. A., and E. Yilmaz. Estimating average precision with incomplete and imperfect judgments. In Proceedings of CIKM' 06, page102-111, 2006.
3. Azzopardi L., de Rijke, M., and K. Balog. Building simulated queries for known-item topics: an analysis using six european languages. In Proceedings of SIGIR '07, pages 455-462, 2007.
4. Azzopardi, L. and Vinay, V. Retrievability. An evaluation measure for higher order information access tasks. In Proccedings of CIKM '08, pages 1425-1426, 2008.
5. Baeza-Yates, J., and Ribeiro-Neto, B. Modern Information Retrieval. Addison Wesley, 1999.
6. Bashir, S., and Rauber A. Analyzing Document Retrievability in Patent Retrieval Settings. In Proceedings of Database and Expert Systems Applications (DEXA 2009), pages 753-760, 2009.
7. Buckley, C., and Voorhees, E. M. Evaluating Evaluation Measure Stability. In Proceedings of SIGIR 2000, pages 33-40, 2000.
8. Buckley, C., Dimmick, D., Soboroff, I., and E. Voorhees. Bias and the limits of pooling. In Proceedings of SIGIR '06, pages 619-620, 2006.
9. Buckley, C., and Voorhees, E. M. Retrieval evaluation with incomplete information. In Proceedings of SIGIR '04, pages 25-32, 2004.
10. Carterette, B., Bennett, P. N. Chickering, D. M., and Dumais, S. T. Here or There: Preference Judgments for Relevance. In Proceedings of ECIR '08, pages 16-27, 2008.
11. Cleverdon, C. The Cranfield Tests on Index Language Devices. In: Sparck Jones, K. and Willett, P. (eds.). Readings in Information Retrieval, pages 47-59, Morgan Kaufmann, 1997.
12. Hull, D. Using statistical testing in the evaluation of retrieval experiments. In Proceedings of SIGIR '93, pages 329-338, 1993.
13. Fujii, A., Iwayama, M., and Kando, N. Overview of Patent Retrieval Task at NTCIR-4. In Proceedings of the 4th NTCIR Workshop, 2004.
14. Graf, E., and Azzopardi, L. A methodology for building a patent test collection for prior art search. In Proceedings of the 2nd EVIA Workshop, pages 60-71, 2008.
15. Jordan, C., Watters, C., and Gao, Q. Using controlled query generation to evaluate blind relevance feedback algorithms. In Proceedings of JCDL '06, pages 286-295, 2006.
16. Kamps, J., Pehcevski, J., Kazai, G., Lalmas, M., and Robertson, S. INEX 2007 evaluation measures. In Proceedings of INEX '07, pages 24-33, 2007.
17. Kendall, M. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
18. Mandl, T. Recent developments in the evaluation of information retrieval systems: moving toward diversity and practical applications. Informatica, 32:27-38, 2008.
19. Moffat, A., and Zobel, J. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27(1):1-27, 2008.
20. Oard, D. W., Hedin, B., Tomlinson, S., and Baron, J. R. Overview of the TREC 2008 legal track. In Proceedings of TREC 2008, 2008.
21. van Rijsbergen, C. J. Information Retrieval, 2nd edition. Butterworths, 1979.
22. Robertson S. E. The parametric description of the retrieval tests. Part 2: Overall measures. Journal of Documentation, 25(2):93-107, 1969.
23. Robertson, S. A new interpretation of average precision. In Proceedings of SIGIR '08, pages 689-690, 2008.
24. Rocchio J. Performance indices for document retrieval systems. In Information storage and retrieval, Computation Laboratory of Harvard University, Cambridge, MA, 1964.
25. Roda G., Tait J., Piroi F., and Zenz V. CLEF-IP 2009: retrieval experiments in the Intellectual Property domain. In Proceedings of CLEF '09, 2009.
26. Tague J., Nelson, M., and Wu, H. Problems in the simulation of bibliographic retrieval systems. In Proceeding of SIGIR '81, pages 66-71, 1981.
27. Tomlinson S., Oard, D. W., Baron, J. R., and Thompson, P. Overview of the TREC 2007 Legal Track. In Proceedings of TREC 2007, 2007.
28. Voorhees, E. M., and Tice, D. M. The TREC-8 Question Answering Track Evaluation. In Proceedings of TREC 1999, pages 77-82, 1999.
29. Voorhees, E. M. The Philosophy of Information Retrieval Evaluation. In Evaluation of Cross-Language Information Retrieval System, Proceedings of CLEF '02, pages 355-370, 2002.
30. Voorhees, E. M. The TREC robust retrieval track. In SIGIR Forum 39(1):11-20, 2005.
31. Xue, X., and Croft W. B. Automatic Query Generation for Patent Search. In Proceedings of CIKM'09, pages 2037-2040, 2009.
32. Zhu, J., and Tait, J. A proposal for chemical information retrieval evaluation. In In Proceedings of the 1st ACM Workshop on Patent Information Retrieval at CIKM '08, pages 15-18, 2008.