The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing

WWW 2014 - Proceedings of the 23rd International Conference on World Wide Web

Volumn , Issue , 2014, Pages 165-175

  Li, Hongwei ^a   Zhao, Bo ^b   Fuxman, Ariel ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Crowd Targeting; Crowdsourcing; Worker Quality

Indexed keywords

BUDGET CONTROL;

AUTOMATIC DISCOVERY; CROWD TARGETING; CROWDSOURCING PLATFORMS; DATA QUALITY; HIGH QUALITY; PRACTICAL PROBLEMS; QUALITY ESTIMATION; REAL DATA SETS;

CROWDSOURCING;

EID: 84909630641     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2566486.2568033     Document Type: Conference Paper

References (27)

1. Y. Bachrach, T. Graepel, T. Minka, and J. Guiver. How to grade a test without knowing the answers - a bayesian graphical model for adaptive crowdsourcing and aptitude testing. arXiv preprint arXiv:1206.6386, 2012.
2. A. P. Dawid and A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society., 28(1):20-28, 1979.
3. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological), pages 1-38, 1977.
4. S. Ertekin, H. Hirsh, and C. Rudin. Learning to predict the wisdom of crowds. arXiv preprint arXiv:1204.3611, 2012.
5. C.-J. Ho, S. Jabbari, and J. W. Vaughan. Adaptive task assignment for crowdsourced classification. In ICML, pages 534-542, 2013.
6. P. G. Ipeirotis. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2):16-21, 2010.
7. P. G. Ipeirotis. Demographics of mechanical turk. In NYU Digital Working Paper CeDER-10-01, 2010.
8. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, pages 64-67. ACM, 2010.
9. D. R. Karger, S. Oh, and D. Shah. Budget-optimal crowdsourcing using low-rank matrix approximations. In Communication, Control, and Computing (Allerton), 2011 49th Annual Allerton Conference on, pages 284-291. IEEE, 2011.
10. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.
11. G. Kazai, J. Kamps, and N. Milic-Frayling. The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In Proceedings of the 21st ACM Conference on Information and Knowledge Management (CIKM 2012). ACM Press, New York NY, 2012.
12. G. Kazai, J. Kamps, and N. Milic-Frayling. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information Retrieval, 16:138-178, 2013.
13. H. Li, B. Yu, and D. Zhou. Error Rate Analysis of Labeling by Crowdsourcing. In ICML Workshop: Machine Learning Meets Crowdsourcing. Atalanta, Georgia, USA., 2013.
14. H. Li, B. Yu, and D. Zhou. Error rate bounds in crowdsourcing models. arXiv preprint arXiv:1307.2674, 2013.
15. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. NIPS, 2012.
16. R. M. Mickey, O. J. Dunn, and V. Clark. Applied statistics: analysis of variance and regression. Wiley-Interscience, 2004.
17. T. Pfei?er, X. A. Gao, Y. Chen, A. Mao, and D. G. Rand. Adaptive polling for information aggregation. In AAAI, 2012.
18. V. C. Raykar, S. Yu, L. H. Zhao, C. Florin, L. Bogoni, and L. Moy. Learning From Crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
19. J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI, pages 2863-2872. ACM, 2010.
20. L. A. Schmidt. Crowdsourcing for human subjects research. Proceedings of Crowd Conf, 2010.
21. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi. Inferring Ground Truth from Subjective Labelling of Venus Images. In NIPS, 1995.
22. R. Snow, B. O. Connor, D. Jurafsky, A. Y. Ng, D. Labs, and C. St. Cheap and Fast - But is it Good ? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP, 2008.
23. L. STAHLE and S. Wold. Analysis of variance (anova). Chemometrics and intelligent laboratory systems, 6(4):259-272, 1989.
24. P. Welinder, S. Branson, S. Belongie, and P. Perona. The Multidimensional Wisdom of Crowds. In NIPS, 2010.
25. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose Vote Should Count More : Optimal Integration of Labels from Labelers of Unknown Expertise. In NIPS, pages 2035-2043, 2009.
26. Y. Yan, G. M. Fung, R. Rosales, and J. G. Dy. Active learning from crowds. In ICML, pages 1161-1168, 2011.
27. D. Zhou, J. Platt, S. Basu, and Y. Mao. Learning from the Wisdom of Crowds by Minimax Entropy. In NIPS, 2012.