Optimality of Belief Propagation for crowdsourced classification

33rd International Conference on Machine Learning, ICML 2016

Volumn 2, Issue , 2016, Pages 803-818

  Ok, Jungseul ^a   On, Sewoong ^b   Shin, Jinwoo ^a   Yi, Yung ^a  

^a Korea Advanced Institute of Science and Technology (KAIST)   (South Korea)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; INFERENCE ENGINES; INFORMATION THEORY; LEARNING SYSTEMS;

BELIEF PROPAGATION; CANONICAL SCENARIOS; INFERENCE PROBLEM; LOWER BOUNDS; OPTIMALITY; STATE-OF-THE-ART ALGORITHMS; SUB-OPTIMAL PERFORMANCE;

BACKPROPAGATION;

EID: 84998610657     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Bollobás, B. Random graphs. Springer, 1998.
2. Bordenave, C, Lelarge, M., and Massoulié, L. Nonbacktracking spectrum of random graphs: community detection and non-regular ramanujan graphs. In Proceedings of IEEE FOCS, 2015.
3. Completely Automated Public Turing test to tell Computers and Humans Apart. Captcha. http://www.captcha.net/, 2000.
4. Dalvi, N., Dasgupta, A., Kumar, R., and Rastogi, V. Aggregating crowdsourced binary ratings. In Proceedings of WWW, 2013.
5. Dawid, A. P. and Skene, A. M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28, 1979.
6. Gao, C. and Zhou, D. Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. arXiv preprint arXiv: 1310.5764, 2013.
7. Ghosh, A., Kale, S., and McAfee, P. Who moderates the moderators?: Crowdsourcing abuse detection in usergenerated content. In Proceedings of ACM EC, 2011.
8. Hajek, B., Wu, Y., and Xu, J. Exact recovery threshold in the binary censored block model. In Proceedings of IEEE Information Theory Workshop, 2015.
9. Jin, R. and Ghahramani, Z. Learning with multiple labels. In Proceedings of NIPS, 2003.
10. Karger, D. R., Oh, S., and Shah, D. Iterative learning for reliable crowdsourcing systems. In Proceedings of NIPS, 2011.
11. Karger, D. R., Oh, S., and Shah, D. Efficient crowdsourcing for multi-class labeling. In Proceedings of ACM SIGMETRICS, 2013.
12. Karger, D. R., Oh, S., and Shah, D. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1-24, 2014.
13. Kudekar, Shrinivas, Richardson, Tom, and Urbanke, Rudiger L. Spatially coupled ensembles universally achieve capacity under belief propagation. IEEE Transactions on Information Theory, 59(12):7761-7813, 2013.
14. Li, H. and Yu, B. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv: 1411.4086, 2014.
15. Li, H., Yu, B., and Zhou, D. Error rate analysis of labeling by crowdsourcing. In Proceedings of ICML, 2013.
16. Littlestone, N. and Warmuth, M. K. The weighted majority algorithm. In Proceedings of IEEE FOCS, 1989.
17. Liu, Q., Peng, J., and Ihler, A. T. Variational inference for crowdsourcing. In Proceedings of NIPS, 2012.
18. Mossel, E., Neeman, J., and Sly, A. Belief propagation, robust reconstruction and optimal recovery of block models. In Proceedings of COLT, 2014.
19. Park, S. and Shin, J. Max-product belief propagation for linear programming: applications to combinatorial optimization. In Proceedings of UAI, 2015.
20. Pearl, J. Reverend bayes on inference engines: A distributed hierarchical approach. In Proceedings of AAAI, 1982.
21. Raykar, V. C, Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., Moy, L., and Blei, D. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
22. Saade, A., Lelarge, M., Krzakala, F., and Zdeborová, L. Spectral detection in the censored block model. In Proceedings of IEEE ISIT, pp. 1184-1188. IEEE, 2015.
23. Sheng, V. S., Provost, F, and Ipeirotis, P. G. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of ACM SIGKDD, 2008.
24. Smyth, P., Fayyad, U., Burl, M., Perona, P., and Baldi, P. Inferring ground truth from subjective labelling of venus images. In Proceedings of NIPS, 1995.
25. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP. Association for Computational Linguistics, 2008.
26. Welinder, P., Branson, S., Belongie, S., and Perona, P. The multidimensional wisdom of crowds. In Proceedings of NIPS, 2010.
27. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of NIPS, 2009.
28. Zhang, Y., Chen, X., Zhou, D., and Jordan, M. I. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. In Proceedings of NIPS, 2014.
29. Zhou, D., Platt, J., Basu, S., and Mao, Y. Learning from the wisdom of crowds by minimax entropy. In Proceedings of NIPS, 2012.
30. Zhou, D., Liu, Q., Platt, J. C, Meek, C, and Shah, N. B. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv: 1503.07240, 2015.