Aggregating crowdsourced binary ratings

WWW 2013 - Proceedings of the 22nd International Conference on World Wide Web

Volumn , Issue , 2013, Pages 285-294

  Dalvi, Nilesh ^a   Dasgupta, Anirban ^b   Kumar, Ravi ^c   Rastogi, Vibhor ^c  

^a FACEBOOK   (United States)

^b YAHOO RESEARCH   (United States)

^c GOOGLE INC   (United States)

Author keywords

Crowdsourcing; Mechanical turk; Spectral methods

Indexed keywords

ERROR ANALYSIS; WORLD WIDE WEB;

ERROR RATE; MECHANICAL TURKS; REAL DATA SETS; SPECTRAL METHODS; STATE OF THE ART; THEORETICAL ERRORS; TRUTH VALUES;

CROWDSOURCING;

EID: 84893096721     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2488388.2488414     Document Type: Conference Paper

References (19)

1. S. Boyd and J. Dattorro. Alternating projections, 2003. www.stanford.edu/class/ee392o/altproj.pdf.
2. B. Carpenter. A hierarchical Bayesian model of crowdsourced relevance coding. In Prof. 12th TREC, 2011.
3. A. Dawid and A. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, pages 20-28, 1979.
4. O. Dekel and O. Shamir. Vox populi: Collecting high-quality labels from a crowd. In Proc. 22nd COLT, 2009.
5. A. Ghosh, S. Kale, and R. P. McAfee. Who moderates the moderators?: Crowdsourcing abuse detection in user-generated content. In Proc. 12th EC, pages 167-176, 2011.
6. A. Ghosh and R. P. McAfee. Crowdsourcing with endogenous entry. In Proc. 21st WWW, pages 999-1008, 2012.
7. N. Gillis and F. Glineur. Low-rank matrix approximation with weights or missing data is NP-hard. SIAM J. Matrix Analysis Applications, 32(4):1149-1165, 2011.
8. P. G. Ipeirotis and P. K. Paritosh. Managing crowdsourced human computation: a tutorial. In Proc. 20th WWW(Companion Volume), pages 287-288, 2011.
9. E. Kamar and E. Horvitz. Incentives for truthful reporting in crowdsourcing. In AAMAS, pages 1329-1330, 2012.
10. D. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In Proc. 25th NIPS, pages 1953-1961, 2011.
11. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. In Proc. 26th NIPS, 2012.
12. P. K. Paritosh, P. Ipeirotis, M. Cooper, and S. Suri. The computer is the new sewing machine: benefits and perils of crowdsourcing. In Proc. 20th WWW (Companion Volume), pages 325-326, 2011.
13. V. C. Raykar and S. Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. JMLR, 13:491-518, 2012.
14. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. JMLR, 11:1297-1322, 2010.
15. V. Sheng, F. Provost, and P. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proc. 14th KDD, pages 614-622, 2008.
16. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast-but is it good? Evaluating non-expert annotations for natural language tasks. In EMNLP, pages 254-263, 2008.
17. J. A. Tropp. User-friendly tail bounds for sums of random matrices. Found. Comput. Math., 12:389-434, 2012.
18. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Proc. 24th NIPS, pages 2424-2432, 2010.
19. D. Zhou, J. Platt, S. Basu, and Y. Mao. Learning from the wisdom of crowds by minimax entropy. In Proc. 26th NIPS, pages 2204-2212, 2012.