Active exploration for learning rankings from clickthrough data

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn , Issue , 2007, Pages 570-579

  Radlinski, Filip ^a   Joachims, Thorsten ^a  

^a Department of Computer Science and School of Operations Research and Information Engineering   (United States)

Author keywords

Active exploration; Clickthrough data; Learning to rank; Web search

Indexed keywords

DATA ACQUISITION; E-LEARNING; INFORMATION RETRIEVAL SYSTEMS; ONLINE SYSTEMS; PROBLEM SOLVING; SEARCH ENGINES;

ASSYNTHETIC DATA; CLICKTHROUGH DATA; LEARNING RANKINGS;

LEARNING SYSTEMS;

EID: 36849045755     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1281192.1281254     Document Type: Conference Paper

References (28)

1. E. Agichtein, E. Brill, S. Dumais, and R. Ragno. Learning user interaction models for predicting web search result preferences. In Proceedings of the A CM Conference on Research and Development in Information Retrieval (SIGIR), 2006.
2. R. A. Bradley and M. E. Terry. The rank analysis of incomplete block designs. 1. the method of paired comparisons. Biometrika, 39:324-345, 1952.
3. K. Brinker. Active learning of label ranking functions. In Proceedings of the International Conference on Machine Learning (ICML), 2004.
4. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of the International Conference on Machine Learning (ICML), 2005.
5. B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), 2006.
6. U. Chajewska, D. Koller, and R. Parr. Making rational decisions using adaptive utility elicitation. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2000.
7. W. Chu and Z. Ghahramani. Extensions of gaussian processes for ranking: Semi-supervised and active learning. In Proceedings of the NIPS 2005 Workshop on Learning to Rank, 2005.
8. W. Chu and Z. Ghahramani. Preference learning with gaussian processes. In Proceedings of the International Conference on Machine Learning (ICML), 2005.
9. O. Dekel, C. D. Manning, and Y. Singer. Log-linear models for label ranking. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS), 2003.
10. N. Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM Transactions on Information Science (TOIS), 7(3):183-204, 1989.
11. M. E. Glickman. Parameter estimation in large dynamic paired comparison experiments. Applied Statistics, 48:377-394, 1999.
12. M. E. Glickman. Bayesian optimal design of knockout tournaments, 2006. Under Review.
13. M. E. Glickman and S. T. Jensen. Adaptive paired comparison design. Journal of Statistical Planning and Inference, 127:279-293, 2005.
14. L. Granka. Eye tracking analysis of user behaviors in online search. Master's thesis, Cornell University, 2004.
15. L. Granka, T. Joachims, and G. Gay. Eye-tracking analysis of user behavior in WWW search. In Poster Abstract, Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR), 2004.
16. D. Hawking and N. Craswell. Overview of the TREC-2001 web track. Nov 2001.
17. R. Herbrich and T. Graepel. Trueskill™: A bayesian skill rating system. Technical Report MSR-TR--2006-80, Microsoft Research, 2006.
18. R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In A. S. et al., editor, Advances in Large Margin Classifiers, pages 115-132, 2000.
19. D. R. Hunter. MM algorithms for generalized bradley-terry models. The Annals of Statistics, 32(1):384-406, 2004.
20. T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD), 2002.
21. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2), April 2007.
22. R. Lin, T. A. Louis, S. M. Paddock, and G. Ridgeway. Loss function based ranking in two-stage hierarchical models. Bayesian Analysis, 1(4):915-946, 2006.
23. S. Pandey, S. Roy, C. Olston, J. Cho, and S. Chakrabarti. Shuffling a stacked deck: The case for partially randomized ranking of search engine results. In Proceedings of the International Conference on Very Large Data Bases (VLDB), 2005.
24. F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. In Proceedings of the A CM Conference on Knowledge Discovery and Data Mining (KDD), 2005.
25. F. Radlinski and T. Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2006.
26. D. Ryvkin. The predictive power of noisy elimination tournaments. Under review.
27. M. Saar-Tsechansky and F. Provost. Active sampling for class probability estimation and ranking. Mach. Learn., 54(2):153-178, 2004.
28. L. Torrey. An active learning approach to efficiently ranking retrieval engines. Technical Report TR-2003-449, Computer Science Dept., Dartmouth College, 2003.