Crowdsourcing high quality labels with a tight budget

WSDM 2016 - Proceedings of the 9th ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2016, Pages 237-246

  Li, Qi ^a   Ma, Fenglong ^a   Gao, Jing ^a   Su, Lu ^a   Quinn, Christopher J ^b  

^a UNIVERSITY AT BUFFALO   (United States)

^b PURDUE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLASSIFICATION (OF INFORMATION); CROWDSOURCING; DATA MINING; ECONOMIC AND SOCIAL EFFECTS; INFORMATION RETRIEVAL; MARKOV PROCESSES; WEBSITES; WORLD WIDE WEB;

BUDGET ALLOCATION; CROWDSOURCING PLATFORMS; HIGH QUALITY; LARGE DATASETS; MARKOV DECISION PROCESSES; QUALITY ASPECTS; QUALITY REQUIREMENTS; STATE OF THE ART;

BUDGET CONTROL;

EID: 84964378377     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2835776.2835797     Document Type: Conference Paper

References (41)

1. A. Agresti. A survey of exact inference for contingency tables. Statistical Science, pages 131-153, 1992.
2. B. I. Aydin, Y. S. Yilmaz, Y. Li, Q. Li, J. Gao, and M. Demirbas. Crowdsourcing for multiple-choice question answering. In Proc. of IAAI, 2014.
3. Y. Bachrach, T. Graepel, T. Minka, and J. Guiver. How to grade a test without knowing the answers|a Bayesian graphical model for adaptive crowdsourcing and aptitude testing. In Proc. of ICML, pages 1183-1190, 2012.
4. X. Chen, Q. Lin, and D. Zhou. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In Proc. of ICML, pages 64-72, 2013.
5. N. Dalvi, A. Dasgupta, R. Kumar, and V. Rastogi. Aggregating crowdsourced binary ratings. In Proc. of WWW, pages 285-294, 2013.
6. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, pages 20-28, 1979.
7. P. Donmez, J. G. Carbonell, and J. G. Schneider. A probabilistic framework to learn from multiple annotators with time-varying accuracy. In Proc. of SDM, pages 826-837, 2010.
8. S. Faradani, B. Hartmann, and P. G. Ipeirotis. What's the right price? Pricing tasks for finishing on time. Proc. of HCOMP, 2011.
9. P. G. Ipeirotis and E. Gabrilovich. Quizz: Targeted crowdsourcing with a billion (potential) users. In Proc. of WWW, pages 143-154, 2014.
10. H. Jin, L. Su, D. Chen, K. Nahrstedt, and J. Xu. Quality of information aware incentive mechanisms for mobile crowd sensing systems. In MobiHoc, pages 167-176, 2015.
11. D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1-24, 2014.
12. H. Li, B. Zhao, and A. Fuxman. The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In Proc. of WWW, pages 165-176, 2014.
13. Q. Li, Y. Li, J. Gao, L. Su, B. Zhao, M. Demirbas, W. Fan, and J. Han. A confidence-aware approach for truth discovery on long-tail data. PVLDB, 8(4), 2014.
14. Q. Li, Y. Li, J. Gao, B. Zhao, W. Fan, and J. Han. Resolving conicts in heterogeneous data by truth discovery and source reliability estimation. In Proc. of SIGMOD, pages 1187-1198, 2014.
15. Q. Liu, A. Ihler, and M. Steyvers. Scoring workers in crowdsourcing: How many control questions are enough? In NIPS, pages 1914-1922, 2013.
16. F. Ma, Y. Li, Q. Li, M. Qiu, J. Gao, S. Zhi, L. Su, B. Zhao, H. Ji, and J. Han. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In Proc. of KDD, pages 745-754, 2015.
17. C. Meng, W. Jiang, Y. Li, J. Gao, L. Su, H. Ding, and Y. Cheng. Truth discovery on crowd sensing of correlated entities. In Proc. of SenSys, pages 169-182, 2015.
18. M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
19. V. Raykar and P. Agrawal. Sequential crowdsourced labeling as an epsilon-greedy exploration in a markov decision process. In Proc. of AISTATS, pages 832-840, 2014.
20. V. C. Raykar and S. Yu. Ranking annotators for crowdsourced labeling tasks. In NIPS, pages 1809-1817, 2011.
21. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. The Journal of Machine Learning Research, 11:1297-1322, 2010.
22. J. Redi and I. Povoa. Crowdsourcing for rating image aesthetic appeal: Better a paid or a volunteer crowd? In Proc. of the International ACM Workshop on Crowdsourcing for Multimedia, pages 25-30, 2014.
23. S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Prentice-Hall, Egnlewood Cliffs, 1995.
24. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proc. of SIGKDD, pages 614-622, 2008.
25. Y. Singer and M. Mittal. Pricing mechanisms for crowdsourcing markets. In Proc. of WWW, pages 1157-1166, 2013.
26. A. Singla, I. Bogunovic, G. Bartók, A. Karbasi, and A. Krause. Near-optimally teaching the crowd to classify. In Proc. of ICML, pages 154-162, 2014.
27. A. Singla and A. Krause. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proc. of WWW, pages 1167-1178, 2013.
28. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast | but is it good? Evaluating non-expert annotations for natural language tasks. In Proc. of EMNLP, pages 254-263, 2008.
29. Y. Tian and J. Zhu. Learning from crowds in the presence of schools of thought. In Proc. of SIGKDD, pages 226-234, 2012.
30. L. Tran-Thanh, T. D. Huynh, A. Rosenfeld, S. D. Ramchurn, and N. R. Jennings. Budgetfix: Budget limited crowdsourcing for interdependent task allocation with quality guarantees. In Proc. of AAMAS, pages 477-484, 2014.
31. L. Tran-Thanh, M. Venanzi, A. Rogers, and N. R. Jennings. Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks. In Proc. of AAMAS, pages 901-908, 2013.
32. L. Von Ahn and L. Dabbish. Labeling images with a computer game. In Proc. of CHI, pages 319-326, 2004.
33. P. Welinder, S. Branson, P. Perona, and S. J. Belongie. The multidimensional wisdom of crowds. In NIPS, pages 2424-2432, 2010.
34. P. Welinder and P. Perona. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In CVPRW, pages 25-32, 2010.
35. J. Whitehill, T.-f. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043, 2009.
36. H. Wu, J. Corney, and M. Grant. Relationship between quality and payment in crowdsourced design. In Proc. of CSCWD, pages 499-504, 2014.
37. Y. Xiao, Y. Zhang, and M. van der Schaar. Socially-optimal design of crowdsourcing platforms with reputation update errors. In ICASSP, pages 5263-5267, 2013.
38. Q. Zhang, Y. Wen, X. Tian, X. Gan, and X. Wang. Incentivize crowd labeling under budget constraint. In Proc. of IEEE Infocom, 2015.
39. H. Zheng, D. Li, and W. Hou. Task design, motivation, and participation in crowdsourcing contests. International Journal of Electronic Commerce, 15(4):57-88, 2011.
40. D. Zhou, S. Basu, Y. Mao, and J. C. Platt. Learning from the wisdom of crowds by minimax entropy. In NIPS, pages 2195-2203, 2012.
41. Y. Zhou, X. Chen, and J. Li. Optimal PAC multiple arm identification with applications to crowdsourcing. In Proc. ICML, pages 217-225, 2014.