On test collections for adaptive information retrieval

Information Processing and Management

Volumn 44, Issue 6, 2008, Pages 1879-1885

  Voorhees, Ellen M ^a  

^a NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

Cranfield; Retrieval evaluation; Test collections

Indexed keywords

EXPERIMENTS; INFORMATION SERVICES; PLANNING; SEARCH ENGINES; STRATEGIC PLANNING;

ADAPTIVE INFORMATION RETRIEVAL; CORE COMPETENCIES; CRANFIELD; META-ANALYSIS; RETRIEVAL EVALUATION; SOURCES OF VARIABILITY; SYSTEM EFFECTIVENESS; TEST COLLECTIONS;

ABSTRACTING;

EID: 53849148992     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2007.12.011     Document Type: Article

References (22)

1. Allan, J. (2005). HARD track overview in TREC 2004: High accuracy retrieval from documents. In Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004 (pp. 25-35). NIST Special Publication 500-261.
2. Banks D., Over P., and Zhang N.-F. Blind men and elephants: Six approaches to TREC data. Information Retrieval 1 (1999) 7-34
3. Buckley, C., Dimmick, D., Soboroff, I., & Voorhees, E. (2006). Bias and the limits of pooling. In Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006) (pp. 619-620).
4. Buckley, C., & Voorhees, E. M. (2000). Evaluating evaluation measure stability. In N. Belkin, P. Ingwersen, & M. Leong (Eds.), Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 33-40).
5. Buckley C., and Voorhees E.M. Retrieval system evaluation. TREC: Experiment and evaluation in information retrieval (2005), MIT Press 53-75
6. Cleverdon, C. W. (1991). The significance of the Cranfield tests on index languages. In Proceedings of the Fourteenth Annual International ACM/SIGIR Conference on Reserach and Development in Information Retrieval (pp. 3-12).
7. Cuadra C.A., and Katter R.V. Opening the black box of relevance. Journal of Documentation 23 4 (1967) 291-303
8. Harter S.P. Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science 47 1 (1996) 37-49
9. Hersh, W., Turpin, A., Price, S., Chan, B., Kraemer, & D., Sacherek, L., et al. (2000). Do batch and user evaluations give the same results? In N. Belkin, P. Ingwersen, & M. Leong (Eds.), Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 17-24).
10. Ingwersen P., Ruthven I., and Belkin N. First international symposium on information interaction in context. ACM SIGIR Forum 41 1 (2007) 117-119
11. Lagergren E., and Over P. Comparing interactive information retrieval systems across sites: The TREC-6 interactive track matrix experiment. In: Croft W.B., Moffat A., van Rijsbergen C., Wilkinson R., and Zobel J. (Eds). Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (1998), ACM Press, New York, Melbourne, Australia 164-172 Aug.
12. Robertson S.E. On sample sizes for non-matched-pair IR experiments. Information Processing and Management 26 6 (1990) 739-753
13. Sakai, T. (2006). Evaluating evaluation metrics based on the bootstrap. In Proceedings of the Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006) (pp. 525-532).
14. Sanderson, M., & Zobel, J. (2005). Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of the Twenty-Eighth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005) (pp. 162-169).
15. Schamber L. Relevance and information behavior. Annual Review of Information Science and Technology 29 (1994) 3-48
16. Sparck Jones K. Automatic language and information processing: Rethinking evaluation. Natural Language Engineering 7 1 (2001) 29-46
17. Sparck Jones K., and van Rijsbergen C. Information retrieval test collections. Journal of Documentation 32 1 (1976) 59-75
18. Tague-Sutcliffe J. The pragmatics of information retrieval experimentation, revisited. Information Processing and Management 28 4 (1992) 467-490
19. Taube M. A note on the pseudomathematics of relevance. American Documentation 16 2 (1965) 69-72
20. Turpin, A. H., & Hersh, W. (2001). Why batch and user evaluations do not give the same results. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 225-231).
21. Voorhees, E. M., & Buckley, C. (2002). The effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 316-323).
22. White R.W., Muresan G., and Marchionini G. Workshop on evaluating exploratory search systems. ACM SIGIR Forum 40 2 (2006) 52-60