A statistical method for evaluation using incomplete judgments

Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn 2006, Issue , 2006, Pages 541-548

  Aslam, Javed A ^a   Pavlu, Virgil ^a   Yilmaz, Emine ^a  

^a Northeastern University   (United States)

Author keywords

Average Precision; Evaluation; Sampling

Indexed keywords

AVERAGE PRECISION; LARGE SCALE RETRIEVAL EVALUATION; RANDOM SAMPLING; STANDARD PERFORMANCE MEASURES; EVALUATION; INCOMPLETE JUDGMENTS;

CORRELATION THEORY; DATA REDUCTION; FUNCTION EVALUATION; LARGE SCALE SYSTEMS; RANDOM PROCESSES; STATISTICAL METHODS; BENCHMARKING; PROBLEM SOLVING; QUERY PROCESSING; SAMPLING;

INFORMATION RETRIEVAL; SYSTEMS ANALYSIS;

EID: 33750288965     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (12)

1. E. C. Anderson. Monte carlo methods and importance sampling. Lecture Notes for Statistical Genetics, October 1999.
2. J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch. pooling, and system evaluation. In O. Frieder, J. Hammer, S. Quershi, and L. Seligman, editors, Proceedings of the Twelfth International Conference on Information and Knowledge Management, pages 484-491. ACM Press, November 2003.
3. J. A. Aslam, V. Pavlu, and E. Yilmaz. Measure-based metasearch. In G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, editors, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 571-572. ACM Press, August 2005.
4. J. A. Aslam, V. Pavlu, and E. Yilmaz. A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments. In Proceedings of the 22nd ICML Workshop on Learning with Partially Classified Training Data, August 2005. Copyright held by authors.
5. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 25-32, New York, NY, USA, 2004. ACM Press.
6. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Croft et al. [7], pages 282-289.
7. W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors. Proceedings of the 21th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, Aug. 1998. ACM Press, New York.
8. D. Harman. Overview of the third text REtreival conference (TREC-3). In D. Harman, editor, Overview of the Third Text REtrieval Conference (TREC-3), pages 1-19, Gaithersburg, MD, USA, Apr. 1995. U.S. Government Printing Office, Washington D.C.
9. P. Kantor, M.-H. Kim, U. Ibraev, and K. Atasoy. Estimating the number of relevant documents in enormous collections. In D. Cfd, editor, Proceedings of tthe 62nd Annual Meeting of the American Sociaty for Information Science, volume 36, pages 507 - 514, 1999.
10. J. A. Rice. Mathematical Statistics and Data Analysis. Wadsworth and Brooks/Cole, 1988.
11. E. M. Voorhees and D. Harman. Overview of the seventh text retrieval conference (TREC-7). In Proceedings of the Seventh Text REtrieval Conference (TREC-7), pages 1-24, 1999.
12. J. Zobel. How reliable are the results of large-scale retrieval experiments? In Croft et al. [7], pages 307-314.