Estimating average precision with incomplete and imperfect judgments

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2006, Pages 102-111

  Yilmaz, Emine ^a   Aslam, Javed A ^a  

^a Northeastern University   (United States)

Author keywords

Average precision; Bpref; Evaluation; Incomplete judgments; Sampling

Indexed keywords

AVERAGE PRECISION; IMPERFECT JUDGMENTS; TREC DATA;

APPROXIMATION THEORY; DATA REDUCTION; ROBUST CONTROL;

INFORMATION RETRIEVAL SYSTEMS;

EID: 34547632535     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1183614.1183633     Document Type: Conference Paper

References (17)

1. J. Allan. HARD track overview in TREC 2004: High accuracy retrieval from documents. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2004.
2. J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch, pooling, and system evaluation. In O. Frieder, J. Hammer, S. Quershi, and L. Seligman, editors, Proceedings of the Twelfth International Conference on Information and Knowledge Management, pages 484-491. ACM Press, November 2003.
3. J. A. Aslam and R. Savell. On the effectiveness of evaluating retrieval systems in the absence of relevance judgments. In J. Callan, G. Cormack, C. Clarke, D. Hawking, and A. Smeaton, editors, Proceedings of the 26th Annual International ACM SIGIB Conference on Research and Development in Information Retrieval, pages 361-362. ACM Press, July 2003.
4. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, editors, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 27-34. ACM Press, August 2005.
5. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 33-40. ACM Press, 2000.
6. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR, Conference on Research and Development in Information Retrieval, pages 25-32, New York, NY, USA, 2004. ACM Press.
7. S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. In Proceedings of the Thirty-Fourth Annual Meeting of the Association for Computational Linguistics, pages 310-318, San Francisco, 1996. Morgan Kaufmann Publishers.
8. C. L. A. Clarke, F. Scholer, and I. Soboroff. The TREC 2005 terabyte track. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005), 2005.
9. C. Cleverdon. The cranfield tests on index language devices. In Readings in Information Retrieval, pages 47-59. Morgan Kaufmann, 1997.
10. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Croft et al. [11], pages 282-289.
11. W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors. Proceedings of the 21th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, Aug. 1998. ACM Press, New York.
12. D. Harman. Overview of the third text REtreival conference (TREC-3). In D. Harman, editor, Overview of the Third Text Retrieval Conference (TREC-3), pages 1-19, Gaithersburg, MD, USA, Apr. 1995. U.S. Government Printing Office, Washington D.C.
13. D. Hawking and S. Robertson. On collection size and retrieval effectiveness. Information Retrieval, 6(1):99-105, 2003.
14. R. Nuray and F. Can. Automatic ranking of retrieval systems in imperfect environments. In Proceedings of the 26th Annual International ACM SIGIR, Conference on Research and Development in Informaion Retrieval, pages 379-380. ACM Press, 2003.
15. I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 66-73, New Orleans, Louisiana, USA, Sept. 2001. ACM Press, New York.
16. E. M. Voorhees. Evaluation by highly relevant documents. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74-82. ACM Press, 2001.
17. J. Zobel. How reliable are the results of large-scale retrieval experiments? In Croft et al. [11], pages 307-314.