Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention

IEEE Transactions on Knowledge and Data Engineering

Volumn 27, Issue 3, 2015, Pages 794-809

  Kurve, Aditya ^a   Miller, David J ^a   Kesidis, George ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

Crowdsourcing; ensemble classification; expectation maximization; inference; multicategory

Indexed keywords

AGGREGATES; DECISION MAKING; IMAGE SEGMENTATION; MAXIMUM PRINCIPLE; STOCHASTIC SYSTEMS;

CROWDSOURCING; ENSEMBLE CLASSIFICATION; EXPECTATION - MAXIMIZATIONS; INFERENCE; MULTICATEGORY;

STOCHASTIC MODELS;

EID: 84922928478     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2014.2327026     Document Type: Article

References (28)

1. P. Welinder, S. Branson, S. Belongie, and P. Perona, "The multidimensional wisdom of crowds," Adv. Neural Inf. Process. Syst., vol. 6, no. 7, p. 8, 2010.
2. M. Vukovic, "Crowdsourcing for enterprises," in Proc. IEEE World Conf. Services-I, 2009, pp. 686-692.
3. J. R. Douceur, "The sybil attack," in Proc. Revised Papers First Int. Workshop Peer-to-Peer Syst., 2002, pp. 251-260.
4. M. Feldman, C. Papadimitriou, J. Chuang, and I. Stoica, "Freeriding and whitewashing in peer-to-peer systems," in Proc. ACM SIGCOMM Workshop Practice Theory Incentives Netw. Syst., 2004, pp. 228-236.
5. A. Dawid, and A. Skene, "Maximum likelihood estimation of observer error-rates using the EM algorithm," Appl. Statist., vol. 28, pp. 20-28, 1979.
6. D. Karger, S. Oh, and D. Shah, "Iterative learning for reliable crowdsourcing systems," in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 1953-1961.
7. P. Welinder and P. Perona, "Online crowdsourcing: Rating annotators and obtaining cost-effective labels," in Proc. IEEE Conf Comput. Vis. Pattern Recog. Workshops, 2010, pp. 25-32.
8. V. C. Raykar, and S. Yu, "Eliminating spammers and ranking annotators for crowdsourced labeling tasks," J. Mach. Learn. Res., vol. 13, pp. 491-518, 2012.
9. X. L. Meng and D. Van Dyk, "The EM algorithman old folk-song sung to a fast new tune," J. Roy. Statist. Soc.: Series B (Statist. Methodol.), vol. 59, no. 3, pp. 511-567, 1997.
10. A. Dempster, N. Laird, and D. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Royal Statist. Soc. Series B (Methodol.), vol. 39, pp. 1-38, 1977.
11. K. Bache and M. Lichman, "UCI machine learning repository,"SchoolInf. Comput. Sci., Univ. California, Irvine, CA, USA, 2013.
12. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan, "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," Adv. Neural Inf. Process. Syst., vol. 22, pp. 2035-2043, 2009.
13. M. W. Graham, and D. J. Miller, "Unsupervised learning of parsimonious mixtures on large spaces with integrated feature and component selection," IEEE Trans. Signal Process., vol. 54, no. 4, pp. 1289-1303, Apr. 2006.
14. C. Wu, "On the convergence properties of the EM algorithm,"Ann. Statist., vol. 11, no. 1, pp. 95-103, 1983.
15. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Supplementary derivation for whose vote should count more. (2009). [Online]. Available: http://mplab.ucsd.edu/jake/supp.pdf
16. D. Karger, S. Oh, and D. Shah, "Efficient crowdsourcing for multiclass labeling," in Proc. ACM SIGMETRICS Int. Conf. Meas. Model. Comput. Syst., 2013, pp. 81-92.
17. R. E. Schapire, "The strength of weak learnability," Mach. Learn., vol. 5, no. 2, pp. 197-227, 1990.
18. D. Benbouzid, R. Busa-Fekete, N. Casagrande, F. Collin, and B. Kégl, "MULTIBOOST: A multi-purpose boosting package," J. Mach. Learn. Res., vol. 13, pp. 549-553, 2012.
19. G. Szarvas, R. Farkas, and A. Kocsor, "A multilingual named entity recognition system using boosting and c4. 5 decision tree learning algorithms," in Proc. 9th Int. Conf. Discovery Sci., 2006, pp. 267-278.
20. V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni, and L. Moy, "Learning from crowds," J. Mach. Learn. Res., vol. 11, pp. 1297-1322, 2010.
21. R. Jin, and Z. Ghahramani, "Learning with multiple labels," in Proc. Adv. Neural Inf. Process. Syst., 2002, pp. 897-904.
22. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi, "Inferring ground truth from subjective labelling of venus images," in Proc. Adv. Neural Inf. Process. Syst., 1995, pp. 1085-1092.
23. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Supplementary materials. (2009). [Online]. Available: http://mplab.ucsd.edu/~'jake/supp.pdf
24. D. Zhou, J. Platt, S. Basu, and Y. Mao, "Learning from the wisdom of crowds by minimax entropy," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 2204-2212.
25. A. Ghosh, S. Kale, and P. McAfee, "Who moderates the moderators?: Crowdsourcing abuse detection in user-generated content,"in Proc. 12th ACM Conf. Electron. Commerce, 2011, pp. 167-176.
26. P. G. Ipeirotis, F. Provost, and J. Wang, "Quality management on amazon mechanical turk," in Proc. ACM SIGKDD Workshop Human Comput., 2010, pp. 64-67.
27. D. R. Karger, S. Oh, and D. Shah, "Budget-optimal crowdsourcing using low-rank matrix approximations," in Proc. 49th IEEE Annu. Allerton Conf. Commun., Control, Comput., 2011, pp. 284-291.
28. R. M. Gray, J. C. Kieffer, and Y. Linde, "Locally optimal block quantizer design," Inf. Control, vol. 45, no. 2, pp. 178-198, 1980.