Efficient crowdsourcing for multi-class labeling

Performance Evaluation Review

Volumn 41, Issue 1 SPEC. ISS., 2013, Pages 81-92

  Karger, David R ^a   Oh, Sewoong ^b   Shah, Devavrat ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

Crowd sourcing; Low rank Matrices; Random Graphs

Indexed keywords

AMAZON'S MECHANICAL TURKS; CROWD-SOURCING; LOW RANK APPROXIMATIONS; LOW-RANK MATRICES; MULTI-CLASS LABELING; PROBABILISTIC MODELS; RANDOM GRAPHS; WEIGHTED ADJACENCY MATRIXES;

ALGORITHMS; COMPUTER SYSTEMS; GRAPH THEORY; IMAGE CLASSIFICATION; INFERENCE ENGINES;

REDUNDANCY;

EID: 84880241088     PISSN: 01635999     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2494232.2465761     Document Type: Conference Paper

References (28)

1. Casting Words. http://castingwords.com.
2. Crowd Flower. http://crowdflower.com.
3. Crowd Spring. http://www.crowdspring.com.
4. ESP game. http://www.espgame.org.
5. Soylent. http://projects.csail.mit.edu/soylent/.
6. M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology, UIST '11, pages 33-42, 2011.
7. M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM UIST, pages 313-322, New York, NY, USA, 2010.
8. B. Bollobás. Random Graphs. Cambridge University Press, Jan. 2001.
9. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28, 1979.
10. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):pp. 1-38, 1977.
11. U. Feige and E. Ofek. Spectral techniques applied to sparse random graphs. Random Struct. Algorithms, 27(2):251-275, 2005.
12. J. Friedman, J. Kahn, and E. Szemerédi. On the second eigenvalue in random regular graphs. In Proceedings of the Twenty-First Annual ACM Symposium on Theory of Computing, pages 587-598, Seattle, Washington, USA, may 1989. ACM.
13. A. Ghosh, S. Kale, and P. McAfee. Who moderates the moderators?: crowdsourcing abuse detection in user-generated content. In Proceedings of the 12th ACM conference on Electronic commerce, pages 167-176. ACM, 2011.
14. S. Holmes. Crowd counting a crowd. March 2011, Statistics Seminar, Stanford University.
15. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, pages 64-67. ACM, 2010.
16. R. Jin and Z. Ghahramani. Learning with multiple labels. In Advances in neural information processing systems, pages 921-928, 2003.
17. D. R. Karger, S. Oh, and D. Shah. Budget-optimal crowdsourcing using low-rank matrix approximations. In Proc. of the Allerton Conf. on Commun., Control and Computing, 2011.
18. D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowd sourcing systems. 2011. http://arxiv.org/abs/1110.3564.
19. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, pages 1953-1961, 2011.
20. R. H. Keshavan, A. Montanari, and S. Oh. Matrix completion from a few entries. IEEE Trans. Inform. Theory, 56(6):2980-2998, June 2010.
21. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems 25, pages 701-709, 2012.
22. V. C. Raykar and S. Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res., 13:491-518, mar 2012.
23. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. J. Mach. Learn. Res., 99:1297-1322, August 2010.
24. T. Richardson and R. Urbanke. Modern Coding Theory. Cambridge University Press, march 2008.
25. n,p. Combinatorica, 7:121-129, 1987.
26. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '08, pages 614-622. ACM, 2008.
27. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems, pages 2424-2432, 2010.
28. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, volume 22, pages 2035-2043, 2009.