Bias and the limits of pooling for large collections

Information Retrieval

Volumn 10, Issue 6, 2007, Pages 491-508

  Buckley, Chris ^a   Dimmick, Darrin ^b   Soboroff, Ian ^b   Voorhees, Ellen ^b  

^a Sabir Research Inc   (United States)

^b NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

Pooling; Sampling bias; Test collections

Indexed keywords

EID: 35548987507     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-007-9032-x     Document Type: Article

References (24)

1. Allan, J. (2006). HARD track overview in TREC 2005: High Accuracy Retrieval from Documents. In Proceedings of the Fourteenth Text Retrieval Conference (TREC 2005).
2. Aslam, J. A., Pavlu, V., & Yilmaz, E. (2005). A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments. In Proceedings of the 22nd ICML Workshop on Learning with Partially Classified Training Data (pp. 57-66).
3. Aslam, J. A., Pavlu, V., & Yilmaz, E. (2006). A statistical method for system evaluation using incomplete judgments. In Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006) (pp. 541-548).
4. Blair, D. C., & Maron, M. E. (1985). An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM, 28(3), 289-299.
5. Buckley, C. (2006). Looking at limits and tradeoffs: Sabir Research at TREC 2005. In Proceedings of the Fourteenth Text REtrieval Conference (TREC 2005).
6. Buckley, C., & Voorhees, E. M. (2004). Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on research and development in information retrieval (pp. 25-32).
7. Buckley, C., & Voorhees, E. M. (2005). Retrieval system evaluation. In TREC: Experiment and Evaluation in Information Retrieval (pp. 53-75). MIT Press.
8. Carterette, B., Allan, J., & Sitaraman, R. (2006). Minimal test collections for retrieval evaluation. In Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006) (pp. 268-275).
9. Cleverdon, C. W. (1967). The Cranfield tests on index language devices. In Aslib Proceedings, 19 (pp. 173-192). Reprinted in Readings in Information Retrieval, K. Sparck Jones & C. van Rijsbergen (Eds.), Morgan Kaufmann, 1997.
10. Reprinted in Readings in Information Retrieval, K. Sparck Jones & C. van Rijsbergen (Eds.), Morgan Kaufmann, 1997.
11. Cormack, G. V., Palmer, C. R., & Clarke C. L. A. (1998). Efficient construction of large test collections. In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, & J. Zobel (Eds.), Proceedings of the 21st Annual International ACM SIGIR Conference on research and development in information retrieval (pp. 282-289). Melbourne, Australia, ACM Press, New York.
12. Hawking, D. (2001). Overview of the TREC-9 web track. In E. M. Voorhees & D. K. Harman (Eds.), Proceedings of the Ninth Text REtrieval Conference (TREC-9) (pp. 87-105).
13. Hawking, D., & Robertson, S. (2003). On collection size and retrieval effectiveness. Information Retrieval, 6(1), 99-150.
14. Madigan, D., Vardi, Y., & Weissman, I. (2006). Extreme value theory applied to document retrieval from large collections. Information Retrieval, 9(3), 273-294.
15. Sanderson, M., & Joho, H. (2004). Forming test collections with no system pooling. In Proceedings of the Twenty-Seventh Annual International ACM SIGIR Conference on research and development in information retrieval (SIGIR 2004) (pp. 33-40).
16. Soboroff, I., & Robertson, S. (2003). Building a filtering test collection for TREC 2002. In Proceedings of SIGIR 2003 the Twenty-sixth Annual Conference on research and development in information retrieval (pp. 243-250).
17. Sparck Jones, K., & van Rijsbergen, C. (1975). Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge.
18. Tague-Sutcliffe, J. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing and Management, 28(4), 467-490.
19. Voorhees, E. M. (2006). Overview of the TREC 2005 robust retrieval track. In Proceedings of the Fourteenth Text Retrieval Conference (TREC 2005).
20. Voorhees, E. M., & Buckley, C. (2002). The effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 316-323).
21. Voorhees, E. M., & Harman, D. K. (2000). Overview of the eighth Text REtrieval Conference (TREC-8). In E. M. Voorhees & D. K. Harman (Eds.), Proceedings of the Eighth Text REtrieval Conference (TREC-8)(pp. 1-24). NIST Special Publication 500-246.
22. Voorhees, E. M., & Harman, D. K. (2002). Overview of TREC 2001. In Proceedings of TREC 2001 (pp. 1-15). NIST Special Publication 500-250.
23. Voorhees, E. M., & Harman, D. K. (Eds.). (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.
24. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, & J. Zobel (Eds.), Proceedings of the 21st Annual International ACM SIGIR Conference on research and development in information retrieval (pp. 307-314). Melbourne, Australia, ACM Press, New York.