Leveraging Side Information to Improve Label Quality Control in Crowd-Sourcing

Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

Volumn , Issue , 2017, Pages 79-88

  Jin, Yuan ^a   Carman, Mark ^a   Kim, Dongwoo ^b   Xie, Lexing ^b  

^a MONASH UNIVERSITY   (Australia)

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CROWDSOURCING; FORECASTING;

CONTENT INFORMATION; CONTEXTUAL INFORMATION; CROWD SOURCING; ENCODINGS; ITEM DIFFICULTIES; LABELINGS; LOGISTICS FUNCTIONS; PERSONALITY TRAITS; SIDE INFORMATION; WORKERS';

QUALITY CONTROL;

EID: 85079540117     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Bachrach, Y.; Graepel, T.; Minka, T.; and Guiver, J. 2012. How to grade a test without knowing the answers-a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In Proceedings of the 29th International Conference on Machine Learning (ICML-12), 1183-1190.
2. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) 28(1):pp. 20-28.
3. Griffiths, T. L., and Steyvers, M. 2004. Finding scientific topics. Proceedings of the National academy of Sciences 101(suppl 1):5228-5235.
4. John, O. P.; Naumann, L. P.; and Soto, C. J. 2008. Paradigm shift to the integrative big five trait taxonomy. Handbook of personality: Theory and research 3:114-158.
5. Jung, H. J., and Lease, M. 2012. Improving quality of crowdsourced labels via probabilistic matrix factorization. In Proceedings of the 4th Human Computation Workshop (HCOMP) at AAAI, 101-106.
6. Kajino, H.; Tsuboi, Y.; and Kashima, H. 2012. A convex formulation for learning from crowds. In AAAI.
7. Kamar, E.; Kapoor, A.; and Horvitz, E. 2015. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing.
8. Kazai, G.; Kamps, J.; and Milic-Frayling, N. 2012. The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In Proceedings of the 21st ACM international conference on Information and knowledge management, 2583-2586. ACM.
9. Kazai, G.; Kamps, J.; and Milic-Frayling, N. 2013. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information retrieval 16(2):138-178.
10. Lease, M. 2011. On quality control and machine learning in crowdsourcing.
11. Li, H.; Zhao, B.; and Fuxman, A. 2014. The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing. In Proceedings of the 23rd international conference on World wide web, 165-176. ACM.
12. Liu, Q.; Peng, J.; and Ihler, A. T. 2012. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems, 692-700.
13. Ma, F.; Li, Y.; Li, Q.; Qiu, M.; Gao, J.; Zhi, S.; Su, L.; Zhao, B.; Ji, H.; and Han, J. 2015. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 745-754. ACM.
14. Moreno, P. G.; Teh, Y. W.; Perez-Cruz, F.; and Artés-Rodríguez, A. 2014. Bayesian nonparametric crowdsourcing. arXiv preprint arXiv:1407.5017.
15. Rasch, G. 1960. Probabilistic Models for Some Intelligence and Attainment Tests. MESA Press.
16. Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010a. Learning from crowds. Journal of Machine Learning Research 11:1297-1322.
17. Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010b. Learning from crowds. Journal of Machine Learning Research 11(Apr):1297-1322.
18. Ruvolo, P.; Whitehill, J.; and Movellan, J. R. 2013. Exploiting commonality and interaction effects in crowdsourcing tasks using latent factor models. In Neural Information Processing Systems. Workshop on Crowdsourcing: Theory, Algorithms and Applications.
19. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, 614-622. New York, NY, USA: ACM.
20. Venanzi, M.; Guiver, J.; Kazai, G.; Kohli, P.; and Shokouhi, M. 2014. Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web, WWW '14, 155-164. New York, NY, USA: ACM.
21. Venanzi, M.; Guiver, J.; Kohli, P.; and Jennings, N. R. 2016. Time-sensitive bayesian information aggregation for crowdsourcing systems. Journal of Artificial Intelligence Research 56:517-545.
22. Wauthier, F. L., and Jordan, M. I. 2011. Bayesian bias mitigation for crowdsourcing. In 25th Annual Conference on Neural Information Processing Systems 2011, 1800-1808.
23. Welinder, P.; Branson, S.; Perona, P.; and Belongie, S. J. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems, 2424-2432.
24. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. R. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In 23rd Annual Conference on Neural Information Processing Systems, NIPS'09, 2035-2043.