Selecting a subset of queries for acquisition of further relevance judgements

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6931 LNCS, Issue , 2011, Pages 113-124

  Hosseini, Mehdi ^a   Cox, Ingemar J ^a   Milic Frayling, Natasa ^b   Vinay, Vishwa ^b   Sweeting, Trevor ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b MICROSOFT RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

BUDGET ALLOCATION; PARTICIPATING SYSTEMS; RELATIVE PERFORMANCE; RELEVANCE ASSESSMENTS; RELEVANCE JUDGEMENT; RELEVANCE JUDGMENT; SEARCH SYSTEM; TEST COLLECTION;

BUDGET CONTROL; INFORMATION RETRIEVAL; LAKES; QUERY PROCESSING;

SEARCH ENGINES;

EID: 80052988568     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-23318-0_12     Document Type: Conference Paper

References (14)

1. Allan, J., Carterette, B., Aslam, J.A., Pavlu, V., Dachev, B., Kanoulas, E.: TREC 2007 million query track. Notebook Proceedings of TREC 2007. TREC (2007)
2. Aslam, J.A., Pavlu, V., Yilmaz, E.: A statistical method for system evaluation using incomplete judgments. In: SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 541-548. ACM, New York (2006)
3. Carterette, B., Allan, J., Sitaraman, R.: Minimal test collections for retrieval evaluation. In: SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 268-275. ACM, New York (2006)
4. Carterette, B., Gabrilovich, E., Josifovski, V., Metzler, D.: Measuring the reusability of test collections. In:WSDM2010: Proceedings of the Third ACMInternational Conference on Web Search and Data Mining, pp. 231-240. ACM, New York (2010)
5. Carterette, B., Kanoulas, E., Pavlu, V., Fang, H.: Reusable test collections through experimental design. In: SIGIR 2010: Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 547-554. ACM, New York (2010)
6. Carterette, B., Pavlu, V., Kanoulas, E., Aslam, J.A., Allan, J.: Evaluation over thousands of queries. In: SIGIR 2008: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 651-658. ACM, New York (2008)
7. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Annals of Statistics 32, 407-499 (2004)
8. Guiver, J., Mizzaro, S., Robertson, S.: A few good topics: Experiments in topic set reduction for retrieval evaluation. ACM Trans. Inf. Syst. 27(4) (2009)
9. Mizzaro, S., Robertson, S.: Hits hits trec: exploring ir evaluation results with network analysis. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2007, pp. 479-486. ACM, New York (2007)
10. Robertson, S.: On the contributions of topics to system evaluation. In: Clough, P., Foley, C., Gurrin, C., Jones, G.J.F., Kraaij, W., Lee, H., Mudoch, V. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 129-140. Springer, Heidelberg (2011)
11. Sparck Jones, K., Van Rijsbergen, K.: Information retrieval test collections. Journal of Documentation 32(1), 59-75 (1976)
12. Voorhees, E.M.: The philosophy of information retrieval evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 355-370. Springer, Heidelberg (2002)
13. Voorhees, E.M., Buckley, C.: The effect of topic set size on retrieval experiment error. In: SIGIR 2002: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 316-323. ACM, New York (2002)
14. Zobel, J.: How reliable are the results of large-scale information retrieval experiments? In: SIGIR 1998: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 307-314. ACM, New York (1998)