Model characterization curves for federated search using click-logs: Predicting user engagement metrics for the span of feasible operating points

Proceedings of the 20th International Conference on World Wide Web, WWW 2011

Volumn , Issue , 2011, Pages 67-76

  Ponnuswami, Ashok Kumar ^a   Pattabiraman, Kumaresh ^a   Brand, Desmond ^a   Kanungo, Tapas ^a  

^a MICROSOFT   (United States)

Author keywords

Metrics; Offline prediction; Online evaluation; Randomization; User engagement

Indexed keywords

METRICS; OFFLINE; ON-LINE EVALUATION; RANDOMIZATION; USER ENGAGEMENT;

AGGREGATES; ALGORITHMS; FORECASTING; WORLD WIDE WEB;

SEARCH ENGINES;

EID: 84871096702     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1963405.1963419     Document Type: Conference Paper

References (25)

1. J. Arguello, J. Callan, F. Diaz, and J. F. Crespo. Source of evidence for vertical selection. In Proc. of Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2009.
2. O. Chapelle and Ya Zhang. A dynamic bayesian network model for web search ranking. In Proc. of Intl. Conf. on World Wide Web, 2009.
3. K. Collins-Thompson. Accounting for stability in retrieval algorithms using risk-reward curves. In Proc. of SIGIR 2009 Workshop on the Future of Evaluation in Information Retrieval, pages 27-28, 2009.
4. K. Collins-Thompson. Reducing the risk of query expansion via robust constrained optimization. In Intl. Conf. on Information and Knowledge Management, pages 837-846, 2009.
5. N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. In Intl. Conf. on Web Search and Data Mining, 2008.
6. F. Diaz and J. Arguello. Adaptation of offline selection predictions in presense of user feedback. In Proc. of Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2009.
7. G. Dupret and C. Liao. A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine. In Intl. Conf. on Web Search and Data Mining, 2010.
8. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. CRC Press, 1993.
9. D. Fetterly, N. Craswell, and V. Vinay. The impact of crawl policy on web search effectiveness. In Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 58-587, 2009.
10. J. H. Friedman. Greedy function approximation: A graidient boosting machine. Annals of Statistics, 29:1189-1232, 2001.
11. K. Fukunaga. Statistical Pattern Recognition. Academic Press, 1990.
12. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Sringer-Verlag, New York, NY, 2001.
13. D. Hawking, N. Craswell, P. Thistiewaite, and D. Harman. Results and challenges in web search evaluation. Computer Networks, 31(11-16), 1999.
14. S. Ji, T. Moon, G. Dupret, C. Liao, and Z. Zheng. User behavior driven ranking without editorial judgments. In Proc. of Intl. Conf. on Information and Knowledge Management, 2010.
15. T. Joachims. A support vector method for multivariate performance measures. In Proc. Intl. Conference on Machine Learning, 2005.
16. T. Kanungo, M. Y. Jaisimha, J. Palmer, and R. M. Haralick. A quantitative methodology for analyzing the performance of detection algorithm. In Proc. of IEEE International Conference on Computer Vision, pages 247-252, 1993.
17. N. A. Macmillan and C. D. Creelman. Detection Theory. Lawerence Earlbaum Associates, Inc., 2005.
18. A. K. Ponnuswami, K. Pattabiraman, Q. Wu, R. Gilad-Bachrach, and T. Kanungo. On composition of a federated web search result page: Using online users to provide pairwise preference for heterogeneous verticals. In Proc. of Intl. Conf. on Web Search and Data Mining, 2011.
19. F. Radlinksi and T. Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proc. of AAAI Conf. on Artificial Intelligence, 2006.
20. F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. pages 239-248, 2005.
21. M. Shokouhi and L. Si. Federated information retrieval. In D. W. Oard and Editors F. Sebastiani, editors, Foundations and Trends in Information Retrieval. 2010.
22. L. Si and J. Callan. Modeling search engine effectiveness for federated search. In Proc. of Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2005.
23. H. L. Van Trees. Detection, Estimation, and Modulation Theory, Part 1. John Wiley and Sons, Inc., 2001.
24. E. M. Voorhees and D. K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
25. K. Wang, N. Gloy, and X. Li. Inferring search behaviors using partially observable Markov (pom) model. In Intl. Conf. on Web Search and Data Mining, 2010.