Evaluating new search engine configurations with pre-existing judgments and clicks

Proceedings of the 20th International Conference on World Wide Web, WWW 2011

Volumn , Issue , 2011, Pages 397-406

  Ozertem, Umut ^a   Jones, Rosie ^b   Dumoulin, Benoit ^c  

^a YAHOO RESEARCH   (United States)

^b AKAMAI TECHNOLOGIES   (United States)

^c MICROSOFT   (United States)

Author keywords

Relevance estimation; Relevance evaluation; User browsing models

Indexed keywords

DATA SETS; ENGINE CONFIGURATION; EVALUATION METHODS; ON-LINE TESTS; PERFORMANCE ESTIMATION; RANDOM SAMPLE; RELEVANCE EVALUATION; SEARCH MODELS; SEARCH RESULTS; SMOOTHING TECHNIQUES;

SEARCH ENGINES; WEB BROWSERS; WORLD WIDE WEB;

ESTIMATION;

EID: 84871081785     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1963405.1963463     Document Type: Conference Paper

References (16)

1. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 667-674, 2008.
2. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25-32, 2004.
3. B. Carterette, J. Allan, and R. K. Sitaraman. Minimal test collections for retrieval evaluation. In E. N. Efthimiadis, S. T. Dumais, D. Hawking, and K. Järvelin, editors, SIGIR, pages 268-275, 2006.
4. B. Carterette and R. Jones. Evaluating search engines by modeling the relationship between relevance and clicks. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 217-224. MIT Press, Cambridge, MA, 2008.
5. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In CIKM '09: Proceeding of the 18th ACM conference on Information and knowledge management, pages 621-630, 2009.
6. O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In WWW. 2009.
7. N. Craswell, O. Zoeter, M. J. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. In M. Najork, A. Z. Broder, and S. Chakrabarti, editors, WSDM, pages 87-94, 2008.
8. G. E. Dupret and B. Piwowarski. A user browsing model to predict search engine click data from past observations. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 331-338, New York, NY, USA, 2008. ACM.
9. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422-446, 2002.
10. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 154-161, New York, NY, USA, 2005. ACM Press.
11. T. Joachims and F. Radlinski. Search engines that learn from implicit feedback. IEEE Computer, 40(8):34-40, August 2007.
12. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., 27(1):1-27, 2008.
13. B. Piwowarski, G. Dupret, and R. Jones. Mining user web search activity with layered bayesian networks or how to capture a click in its context. In WSDM '09: Proceedings of the Second ACM International Conference on Web Search and Data Mining, pages 162-171, New York, NY, USA, 2009. ACM.
14. F. Radlinski and T. Joachims. Active exploration for learning rankings from clickthrough data. In ACM SIGKDD International Conference On Knowledge Discovery and Data Mining (KDD), 2007.
15. I. Soboroff. A comparison of pooled and sampled relevance judgments. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 785-786, New York, NY, USA, 2007. ACM.
16. A. Trotman, N. Phrao, and D. Jenkinson. Can we at least agree on something? In Proc. SIGIR Workshop on Focused Retrieval. 2007.