Exact exponent in optimal rates for crowdsourcing

33rd International Conference on Machine Learning, ICML 2016

Volumn 2, Issue , 2016, Pages 926-938

  Gao, Chao ^a   Lu, Yu ^a   Zhou, Dengyong ^b  

^a YALE UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CROWDSOURCING; ERRORS;

CHERNOFF INFORMATION; CONSISTENT ESTIMATORS; EM ALGORITHMS; ERROR EXPONENT; MACHINE LEARNING APPLICATIONS; MISCLASSIFICATION ERROR; OPTIMAL ERROR; UPPER AND LOWER BOUNDS;

LEARNING SYSTEMS;

EID: 84998579474     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. Amazon Mechanical Turk. https://www.mturk.com/mturk.
2. Anandkumar, Animashree, Ge, Rong, Hsu, Daniel, Kakade, Sham M, and Telgarsky, Matus. Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research, 15(1):2773-2832, 2014.
3. Bachrach, Yoram, Graepel, Thore, Minka, Tom, and Guiver, John. How to grade a test without knowing the answers - a Bayesian graphical model for adaptive crowdsourcing and aptitude testing. In Proceedings of the 29th International Conference on Machine Learning (ICML-12), pp. 1183-1190, 2012.
4. Berend, Daniel and Kontorovich, Aryeh. A finite sample analysis of the naive bayes classifier. Journal of Machine Learning Research, 16:1519-1545, 2015.
5. Cover, Thomas M and Thomas, Joy A. Elements of information theory. John Wiley & Sons, 2006.
6. Dalvi, N., Dasgupta, A., Kumar, R., and Rastogi, V. Aggregating crowdsourced binary ratings. In Proceedings of the 22nd international conference on World Wide Web, pp. 1220-1229, 2013.
7. Dawid, A. P. and Skene, A. M. Maximum likeihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, 28(1):20-28, 1979.
8. Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39(1):1-38, 1977.
9. Gao, Chao and Zhou, Dengyong. Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. arXiv preprint arXiv: 1310.5764, 2013.
10. Ghosh, Arpita, Kale, Satyen, and McAfee, Preston. Who moderates the moderators? Crowdsourcing abuse detection in user-generated content. In Proceedings of the 12th ACM conference on Electronic commerce, pp. 167-176, 2011.
11. Karger, David R, Oh, Sewoong, and Shah, Devavrat. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1-24, 2014.
12. Liu, Q., Peng, J., and Ihler, A. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems 25, pp. 701-709, 2012.
13. Neyman, J and Pearson, ES. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 231(694-706):289-337, 1933.
14. Parisi, Fabio, Strino, Francesco, Nadler, Boaz, and Kluger, Yuval. Ranking and combining multiple predictors without labeled data. Proceedings of the National Academy of Sciences, 111(4):1253-1258, 2014.
15. Raykar, V. C, Yu, S., Zhao, L. H., Valadez, G. H., Florin, C, Bogoni, L., and Moy, L. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
16. Tian, Tian and Zhu, Jun. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems 28, pp. 1612-1620, 2015.
17. Van der Vaart, Aad W. Asymptotic statistics. Cambridge university press, 2000.
18. Venanzi, Matteo, Guiver, John, Kazai, Gabriella, Kohli, Pushmeet, and Shokouhi, Milad. Community-based Bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web, pp. 155-164, 2014.
19. Welinder, Peter, Branson, Steve, Perona, Pietro, and Belongie, Serge J. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23, pp. 2424-2432, 2010.
20. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pp. 2035-2043, 2009.
21. Zhang, Yuchen, Chen, Xi, Zhou, Dengyong, and Jordan, Michael I. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. In Advances in neural information processing systems, pp. 1260-1268, 2014.
22. Zhou, D., Platt, J. C, Basu, S., and Mao, Y Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems 25, pp. 2204-2212, 2012.
23. Zhou, Dengyong, Liu, Qiang, Platt, John, and Meek, Christopher. Aggregating ordinal labels from crowds by minimax conditional entropy. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 262-270, 2014.