The reusability of a diversified search test collection

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 7675 LNCS, Issue , 2012, Pages 26-38

  Sakai, Tetsuya ^a   Dou, Zhicheng ^a   Song, Ruihua ^a   Kando, Noriko ^b  

^a MICROSOFT RESEARCH ASIA   (China)

^b NATIONAL INSTITUTE OF INFORMATICS   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

AD HOC IR; CHINESE DOCUMENTS; EVALUATION METRICS; LEAVE-ONE-OUT; PARTICIPATING TEAMS; RELEVANCE ASSESSMENTS; SEARCH RESULTS; TEST COLLECTION;

INFORMATION RETRIEVAL; INFRARED DEVICES; REUSABILITY; SEARCH ENGINES; TESTING;

LAKES;

EID: 84871603799     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-35341-3_3     Document Type: Conference Paper

References (22)

1. Buckley, C., Dimmick, D., Soboroff, I., Voorhees, E.M.: Bias and the limits of pooling for large collections. Information Retrieval 10(6), 491-508 (2007)
2. Buckley, C., Voorhees, E.M.: Retrieval evaluation with incomplete information. In: Proceedings of ACM SIGIR 2004, pp. 25-32 (2004)
3. Büttcher, S., Clarke, C.L., Yeung, P.C., Soboroff, I.: Reliable information retrieval evaluation with incomplete and biased judgements. In: ACM SIGIR 2007 Proceedings, pp. 63-70 (2007)
4. Carterette, B.: Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM TOIS 30(1) (2012)
5. Carterette, B., Gabrilovich, E., Josifovski, V., Metzler, D.: Measuring the reusability of test collections. In: Proceedings of WSDM 2012, pp. 231-240 (2010)
6. Carterette, B., Kanoulas, E., Pavlu, V., Fang, H.: Reusable test collections through experimental design. In: Proceedings of ACM SIGIR 2010, pp. 547-554 (2010)
7. Carterette, B., Pavlu, V., Kanoulas, E., Aslam, J.A., Allan, J.: Evaluation over thousands of queries. In: Proceedings of ACM SIGIR 2008, pp. 651-658 (2008)
8. Clarke, C.L., Craswell, N., Soboroff, I.: Overview of the TREC 2009 web track. In: Proceedings of TREC 2009 (2009)
9. Clarke, C.L., Craswell, N., Soboroff, I., Voorhees, E.: Overview of the TREC 2011 web track. In: Proceedings of TREC 2011 (2012)
10. Sakai, T.: Alternatives to bpref. In: Proceedings of ACM SIGIR 2007, pp. 71-78 (2007)
11. Sakai, T.: Comparing metrics across TREC and NTCIR: The robustness to pool depth bias. In: Proceedings of ACM SIGIR 2008, pp. 691-692 (2008)
12. Sakai, T.: Comparing metrics across TREC and NTCIR: The robustness to system bias. In: Proceedings of ACM CIKM 2008, pp. 581-590 (2008)
13. Sakai, T., Kando, N.: Are popular documents more likely to be relevant? a dive into the ACLIA IR4QA pools. In: Proceedings of EVIA 2008, pp. 8-9 (2008)
14. Sakai, T., Kando, N.: On information retrieval metrics designed for evaluation with incomplete relevance assessments. Information Retrieval 11, 447-470 (2008)
15. Sakai, T., Song, R.: Diversified search evaluation: Lessons from the NTCIR-9 INTENT task. Information Retrieval (to appear)
16. Sakai, T., Song, R.: Evaluating diversified search results using per-intent graded relevance. In: Proceedings of ACM SIGIR 2011 (2011)
17. Song, R., Zhang, M., Sakai, T., Kato, M.P., Liu, Y., Sugimoto, M., Wang, Q., Orii, N.: Overview of the NTCIR-9 INTENT task. In: Proceedings of NTCIR-9, pp. 82-105 (2011)
18. Voorhees, E.M.: The Philosophy of Information Retrieval Evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 355-370. Springer, Heidelberg (2002)
19. Webber, W., Park, L.A.: Score adjustment for correction of pooling bias. In: Proceedings of ACM SIGIR 2009, pp. 444-451 (2009)
20. Yilmaz, E., Aslam, J., Robertson, S.: A new rank correlation coefficient for information retrieval. In: Proceedings of ACM SIGIR 2008, pp. 587-594 (2008)
21. Yilmaz, E., Aslam, J.A.: Estimating average precision with incomplete and imperfect judgments. In: ACM CIKM 2006 Proceedings, pp. 102-111 (2006)
22. Zobel, J.: How reliable are the results of large-scale information retrieval experiments? In: Proceedings of ACM SIGIR 1998, pp. 307-314 (1998)