Achieving budget-optimality with adaptive schemes in crowdsourcing

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 4851-4859

  Khetan, Ashish ^a   Oh, Sewoong ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; ECONOMIC AND SOCIAL EFFECTS;

ADAPTIVE SCHEME; ADAPTIVITY; OPTIMALITY; PROBABILISTIC MODELING; TASK ASSIGNMENT; TRADE OFF;

BUDGET CONTROL;

EID: 85011336703     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. N. Alon and J. H. Spencer. The probabilistic method. John Wiley and Sons, 2004.
2. N. Dalvi, A. Dasgupta, R. Kumar, and V. Rastogi. Aggregating crowdsourced binary ratings. In Proceedings of the 22nd international conference on World Wide Web, pages 285-294, 2013.
3. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20-28, 1979.
4. A. Ghosh, S. Kale, and P. McAfee. Who moderates the moderators?: crowdsourcing abuse detection in user-generated content. In Proceedings of the 12th ACM conference on Electronic commerce, pages 167-176. ACM, 2011.
5. C. Ho, S. Jabbari, and J. W. Vaughan. Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 534-542, 2013.
6. R. Jin and Z. Ghahramani. Learning with multiple labels. In Advances in neural information processing systems, pages 921-928, 2003.
7. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, pages 1953-1961, 2011.
8. D. R. Karger, S. Oh, and D. Shah. Efficient crowdsourcing for multi-class labeling. In Proceedings of the ACM SIGMETRICS/international conference on Measurement and modeling of computer systems, pages 81-92, 2013.
9. D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62:1-24, 2014.
10. H. Li and B. Yu. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086, 2014.
11. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems 25, pages 701-709, 2012.
12. O. Maron and A. W. Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. Robotics Institute, page 263, 1993.
13. M. Mezard and A. Montanari. Information, physics, and computation. Oxford University Press, 2009.
14. J. Ok, S. Oh, J. Shin, and Y. Yi. Optimality of belief propagation for crowdsourced classification. In International Conference on Machine Learning, 2016.
15. N. B. Shah, S. Balakrishnan, and M. J. Wainwright. A permutation-based model for crowd labeling: Optimal estimation and robustness. arXiv preprint arXiv:1606.09632, 2016.
16. V. S Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD, pages 614-622. ACM, 2008.
17. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi. Inferring ground truth from subjective labelling of venus images. In NIPS, pages 1085-1092, 1995.
18. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems, pages 2424-2432, 2010.
19. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, Volume 22, pages 2035-2043, 2009.
20. D. Williams. Probability with martingales. Cambridge university press, 1991.
21. Y. Zhang, X. Chen, D. Zhou, and M. I. Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. In Advances in neural information processing systems, pages 1260-1268, 2014.
22. D. Zhou, Q. Liu, J. C. Platt, C. Meek, and N. B. Shah. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240, 2015.
23. D. Zhou, J. Platt, S. Basu, and Y. Mao. Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems 25, pages 2204-2212, 2012.