Pairwise ranking aggregation in a crowdsourced setting

WSDM 2013 - Proceedings of the 6th ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2013, Pages 193-202

  Chen, Xi ^a   Bennett, Paul N ^b   Collins Thompson, Kevyn ^b   Horvitz, Eric ^b  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

crowdsourcing; pairwise preference; ranking

Indexed keywords

ACTIVE LEARNING; ACTIVE LEARNING STRATEGIES; BAYESIAN UPDATING; CROWDSOURCING; CURRENT MODELS; EXPLORATION EXPLOITATIONS; LEARNING PROBLEM; PAIR-WISE COMPARISON; PAIRWISE PREFERENCE; RANKING; RANKING AGGREGATION; REAL WORLD DATA; WEB SEARCHES;

DATA MINING; INFORMATION RETRIEVAL; WEBSITES;

UNCERTAINTY ANALYSIS;

EID: 84874248579     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2433396.2433420     Document Type: Conference Paper

References (30)

1. B. Carterette, P. Bennett, D. Chickering, and S. Dumai. Here or there: Preference judgments for relevance. In ECIR, 2008.
2. C.L.Mallows. Non-null ranking models. Biometrika, 44:114-130, 1957.
3. K. Collins-Thompson and J. Callan. A language modeling approach to predicting reading difficulty. In HLT, 2004.
4. R. Coulom. Computing elo ratings of move patterns in the game of go. Technical report, Université Charles de Gaulle, 2007.
5. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28:20-28, 1979.
6. D. Gleich and L. Lim. Rank aggregation via nuclear norm minimization. In KDD, 2011.
7. R. Herbrich, T. Minka, and T. Graepel. Trueskill(tm): A bayesian skill rating system. In NIPS, 2007.
8. T.-K. Huang, R. C. Weng, and C.-J. Lin. Generalized bradley-terry models and multiclass probability estimates. Journal of Machine Learning Research, 7:85-115, 2006. (Pubitemid 43100917)
9. E. Kamar, S. Hacker, and E. Horvitz. Combing human and machine intelligence in large-scale crowdsourcing. In AAMAS, 2012.
10. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, 2011.
11. K.W.Chang, C. Hsieh., T. Huang, T. Wu, R. Weng, and C.J.Lin. Large-scale ranking by sparse paired comparisons. Unpublished manuscript, 2012.
12. C. Lin and J. J. More. Newton's method for large bound-constrained optimization problems. SIAM Journal on Optimization, 9:1100-1127, 1999. (Pubitemid 129489319)
13. T. Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3:225-331, 2009.
14. L.L.Thurstone. The method of paired comparisons for social values. Journal of Abnormal and Social Psychology, 21:384-400, 1927.
15. L.R.Ford. Solution of a ranking problem from binary comparisons. American Math Monthly, 64:28-33, 1957.
16. R. Luce. Individual choice behavior: a theoretical analysis. Wiley, 1959.
17. J. Nocedal and S. Wright. Numerical Optimization. Springer, 2000.
18. T. Pfeiffer, X. A. Gao, A. Mao, Y. Chen, and D. G. Rand. Adaptive polling for information aggregation. In AAAI, 2012.
19. R. Plackett. The analysis of permutations. Applied Statistics, 24:193-302, 1975.
20. T. Qin, X. Geng, and T.Y.Liu. A new probabilistic model for rank aggregation. In NIPS, 2010.
21. R.A.Bradley and M. Terry. The rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324-345, 1952.
22. V. Raykar, S. Yu, L. H. Zhao, G. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
23. B. Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison, 2009.
24. K. Tsukida and M. R. Gupta. How to analyze paired comparison data. Technical report, University of Washington, 2011.
25. M. N. Volkovs and R. S. Zemel. A flexible generative model for preference aggregation. In WWW, 2012.
26. J. Wang, P. G. Ipeirotis, and F. Provost. Managing crowdsourcing workers. In The 2011 Winter Conference on Business Intelligence, 2011.
27. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, 2010.
28. R. C. Weng and C.-J. Lin. A bayesian approximation method for online ranking. Journal of Machine Learning Research, 12:267-300, 2011.
29. M. Woodroofe. Very weak expansions for sequentially designed experiments: linear models. The Annals of Statistics, 17:1087-1102, 1989.
30. Y. Yan, R. Rosales, G. Fung, and J. G. Dy. Active learning from crowds. In ICML, 2011.