Robust test collections for retrieval evaluation

Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'07

Volumn , Issue , 2007, Pages 55-62

  Carterette, Ben ^a  

^a University of Massachusetts Amherst   (United States)

Author keywords

Evaluation; Information retrieval; Reusability; Test collections

Indexed keywords

EVALUATIONS; RELEVANCE JUDGMENTS; ROBUST TEST COLLECTIONS; TEST COLLECTIONS;

COST FUNCTIONS; INFORMATION SYSTEMS; REUSABILITY; SET THEORY;

INFORMATION RETRIEVAL;

EID: 36448969717     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1277741.1277754     Document Type: Article

References (21)

1. J. Aslam and M. Montague. Models for Metasearch. In Proceedings of SIGIR, pages 275-285, 2001.
2. J. Aslam and R. Savell. On the effectiveness of evaluating retrieval systems in the absence of relevance judgments. In Proceedings of SIGIR, pages 361-362, 2003.
3. J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch, pooling, and system evaluation. In Proceedings of CIKM, pages 484-491, 2003.
4. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Proceedings of SIGIR, pages 541-548, 2006.
5. A. L. Berger, S. D. Pietra, and V. J. D. Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39-71, 1996.
6. D. J. Blower. An easy derivation of logistic regression from the bayesian and maximum entropy perspective. In Proceedings of the 23rd International Workship on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, pages 30-43, 2004.
7. B. Carterette and J. Allan. Research methodology in studies of assessor effort for retrieval evaluation. In Proceedings of RIAO, 2007.
8. B. Carterette, J. Allan, and R. K. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of SIGIR, pages 268-275, 2006.
9. B. Carterette and D. I. Petkova. Learning a ranking from pairwise preferences. In Proceedings of SIGIR, 2006.
10. R. T. Clemen and R. L. Winkler. Unanimity and compromise among probability forecasters. Management Science, 36(7):767-779, July 1990.
11. G. V. Cormack, C. R. Palmer, and C. L. Clarke. Efficient Construction of Large Test Collections. In Proceedings of SIGIR, pages 282-289, 1998.
12. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis. Chapman & Hall/CRC, 2004.
13. E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge University Press, 2003.
14. R. Manmatha and H. Sever. A Formal Approach to Score Normalization for Metasearch. In Proceedings of HLT, pages 88-93, 2002.
15. I. J. Myung, S. Ramamoorti, and J. Andrew D. Baily. Maximum entropy aggregation of expert predictions. Management Science, 42(10):1420-1436, October 1996.
16. J. Piatt. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods, pages 61-74, 2000.
17. M. Sanderson and H. Joho. Forming test collections with no system pooling. In Proceedings of the 27th Annual International A CM SIGIR Conference on Research and Development in Information Retrieval, pages 33-40, 2004.
18. I. Soboroff. Dynamic test collections: measuring search effectiveness on the live web. In Proceedings of SIGIR, pages 276-283, 2006.
19. K. Sparck Jones and C. J. van Rijsbergen. Information Retrieval Test Collections. Journal of Documentation, 32(1):59-75, 1976.
20. E. M. Voorhees and D. K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
21. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of SIGIR, pages 307-314, 1998.