Ranking from Pairs and triplets: Information quality, evaluation methods and query complexity

Proceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011

Volumn , Issue , 2011, Pages 105-114

  Radinsky, Kira ^a   Ailon, Nir ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Ranking evaluation; Ranking from pairs; Relevance feedback

Indexed keywords

ACTIVE-LEARNING ALGORITHM; EVALUATION METHOD; EVALUATION PHASE; EVALUATION PROCESS; GOOD STABILITY; INFORMATION QUALITY; LEARNING TO RANK; PAIR-WISE COMPARISON; QUADRATIC COSTS; QUADRATIC NUMBER; QUERY COMPLEXITY; RANKING EVALUATION; RANKING FROM PAIRS; RELATIVE POSITIONS; RELEVANCE FEEDBACK; RELEVANCE JUDGMENT; RELEVANCE SCORE; RETRIEVAL EVALUATION; SEARCH RESULTS; STATISTICAL LEARNING THEORY; THEORY AND PRACTICE; TRAINING PHASE;

DATA MINING; FEEDBACK; INFORMATION RETRIEVAL; LEARNING ALGORITHMS; SEARCH ENGINES; STABILITY; WORLD WIDE WEB;

QUALITY CONTROL;

EID: 79952387636     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1935826.1935850     Document Type: Conference Paper

References (36)

1. N. Ailon. A simple linear ranking algorithm using query dependent intercept variables. In ECIR '09: Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval, pages 685-690, Berlin, Heidelberg, 2009. Springer-Verlag.
2. N. Ailon and M. Mohri. An efficient reduction of ranking to classification. Journal of Machine Learning Research (In Press), 2011.
3. D. Ariely. Predictably Irrational: The Hidden Forces That Shape Our Decisions. HarperCollins, February 2008.
4. D. Ariely, G. Loewenstein, and D. Prelec. Coherent arbitrariness: Stable demand curves without stable preferences. Quarterly Journal of Economics, 118: 73-105, 2003. (Pubitemid 36263882)
5. M. F. Balcan, N. Bansal, A. Beygelzimer, D. Coppersmith, J. Langford, and G. B. Sorkin. Robust reductions from ranking to classification. Mach. Learn., 72(1-2): 139-153, 2008.
6. M. Bravennan and E. Mossel. Noisy sorting without resampling, pages 268-276, 2008.
7. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25-32, New York, NY, USA, 2004. ACM.
8. Z. Cao, T. Qi, T. Y. Liu, M.-F. Tsai, and H. Li. Learning to rank: From pairwise approach to listwise approach. In Microsoft Tech Report MSR-TR-2007-40, 2007.
9. J. Carbonell and J. Goldstein. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings Ann. Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval, 1998.
10. B. Carterette and P. N. Bennett. Evaluation measures for preference judgments. In Proceedings of SIGIR, 2008.
11. B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Here or there: Preference judgments for relevance. In Proceedings of the European Conference on Information Retrieval (ECIR), 2008.
12. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In CIKM '09: Proceeding of the 18th ACM conference on Information and knowledge management, pages 621-630, New York, NY, USA, 2009. ACM.
13. C. L. Clarke, N. Craswell, and I. Soboroff. Overview of the tree 2009 web track. Technical report, no date.
14. W. Cochran. Some methods for strengthening the common chi-square tests. Biometrics, 10: 417-451, 1954.
15. W. W. Cohen, R. E. Schapire, and Y. Singer. Learning to order things. In NIPS: Proceedings of the 1997 conference on Advances in neural information processing systems 10, pages 451-457, Cambridge, MA, USA, 1998. MIT Press.
16. V. Conitzer, A. Davenport, and J. Kalagnanam. Improved bounds for computing kemeny rankings. In AAAI 06: Proceedings of the 21st national conference on Artificial intelligence, pages 620-626. AAAI Press, 2006.
17. K. Crammer and Y. Singer. Pranking with ranking. In Advances in Neural Information Processing Systems 14, pages 641-647. MIT Press, 2001.
18. A. Das Sarma, A. Das Sarma, S. Gollapudi, and R. Panigrahy. Ranking mechanisms in twitter-like forums. In WSDM: Proceedings of the third ACM international conference on Web search and data mining, pages 21-30, 2010.
19. P. Diaconis and R. Graham. Spearman's footrule as a measure of disarray. J. of Royal Statistical Society, 39(2): 262-268, 1977.
20. R. Herbrich, T. Graepel, and K. Obermayer. Advances in Large Margin Classifiers. MIT Press, 200.
21. E. Hüllermeier, J. Fürnkranz, W. Cheng, and K. Brinker. Label ranking by learning pairwise preferences. Artif. Intell., 172(16-17): 1897-1916, 2008.
22. K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20: 422-446, 2002.
23. T. Joachims. Optimizing search engines using clickthrough data. In KDD '02: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142, New York, NY, USA, 2002. ACM.
24. C. Kenyon Mathieu and W. Schudy. How to rank with few errors. In STOC '07: Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 95-103, New York, NY, USA, 2007. ACM.
25. A. Kittur, H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proc. CHI 2008, ACM Pres, pages 453-456, 2008.
26. Y. Lan, T. Y. Liu, Z. Ma, and H. Li. Generalization analysis of listwise learning-to-rank algorithms. In ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning, pages 577-584, New York, NY, USA, 2009. ACM.
27. S. Mizzaro. A new measure of retrieval effectiveness (or: What's wrong with precision and recall. In In Proceedings of the International Workshop on Information Retrieval (IR '2001, pages 43-52, 2001.
28. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., 27: 1-27, 2008.
29. A. Oulasvirta, J. P. Hukkinen, and B. Schwartz. When more is less: the paradox of choice in search engine use. In SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 516-523, 2009.
30. J. Payne, J. Bettman, and E. Johnson. The adaptive decision maker. Massachusetts: Cambridge University Press, 1993.
31. C. Rudin. The P-Norm Push: A simple convex ranking algorithm that concentrates at the top of the list. Journal of Machine Learning Research, 10: 2233-2271, Oct 2009.
32. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. part iii: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58: 2126-2144, 2007.
33. P. Slovic, D. Griffin, and A. Tversky. Compatibility effects in judgment and choice. Insights in decision making, pages 5-27, 1990.
34. K. Train. Discrete Choice Methods with Simulation. Massachusetts: Cambridge University Press, 2003.
35. F. Xia, T. Y Liu, J. Wang, W Zhang, and H. Li. Listwise approach to learning to rank: theory and algorithm. In ICML '08: Proceedings of the 25th international conference on Machine learning, pages 1192-1199, New York, NY, USA, 2008. ACM.
36. J. Xu and H. Li. Adarank: a boosting algorithm for information retrieval. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 391-398, 2007.