Accurate integration of crowdsourced labels using workers' self-reported confidence scores

IJCAI International Joint Conference on Artificial Intelligence

Volumn , Issue , 2013, Pages 2554-2560

  Oyama, Satoshi ^a   Baba, Yukino ^b   Sakurai, Yuko ^c   Kashima, Hisashi ^b  

^a HOKKAIDO UNIVERSITY   (Japan)

^b UNIVERSITY OF TOKYO   (Japan)

^c KYUSHU UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

CONFIDENCE SCORE; CROWDSOURCING; EXPECTATION-MAXIMIZATION ALGORITHMS; IMAGE LABELING; INTEGRATION PROCESS; PROBABILISTIC MODELS; QUESTION ANSWERING TASK; WORKERS';

ALGORITHMS; PROBABILITY DISTRIBUTIONS;

ARTIFICIAL INTELLIGENCE;

EID: 84896061022     PISSN: 10450823     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie. Visual Recognition with Humans in the Loop. In Proceedings of the 11th European Conference on Computer Vision (ECCV), 2010.
2. A. P. Dawid and A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statics), 28(1):20-28, 1979.
3. R. J. de Ayala. The Theory and Practice of Item Response Theory. The Guilford Press, 2009.
4. Bruno de Finetti. Methods for Discriminating Levels of Partial Knowledge Concerning a Test Item. British Journal of Mathematical and Statistical Psychology, 18(1):87-123, 1965.
5. Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider. Efficiently Learning the Accuracy of Labeling Sources for Selective Sampling. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2009.
6. Carsten Eickhoff and Arjen de Vries. How Crowdsourcable is Your Task. In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM), 2011.
7. Ryan Gomes, Peter Welinder, Andreas Krause, and Pietro Perona. Crowdclustering. In Advances in Neural Information Processing 24, 2011.
8. Panos Ipeirotis. How good are you, Turker?, 2009. http://www.behind-the- enemylines.com/2009/01/how-good-are-you-turker.html.
9. Hiroshi Kajino, Yuta Tsuboi, and Hisashi Kashima. A Convex Formulation for Learning from Crowds. In Proceedings of the 26th Conference on Artificial Intelligence (AAAI), 2012.
10. Kentaro Kato and Yiping Zhang. An Item Response Model for Probability Testing. In International Meeting of the Psychometric Society, 2010.
11. Gabriella Kazai. In Search of Quality in Crowdsourcing for Search Engine Evaluation. In Proceedings of the 33rd European Conference on Information Retrieval (ECIR), 2011.
12. Aniket Kittur, Ed H. Chi, and Bongwon Suh. Crowdsourcing User Studies with Mechanical Turk. In Proceedings of the 26th International Conference on Human Factors in Computing Systems (CHI), 2008.
13. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from Crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
14. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2008.
15. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. Inferring Ground Truth from Subjective Labelling of Venus Images. In Advances in Neural Information Processing Systems 7, 1995.
16. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and Fast-But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2008.
17. Peter Welinder, Steve Branson, Serge Belongie, and Pietro Perona. The Multidimensional Wisdom of Crowds. In Advances in Neural Information Processing Systems 23, 2010.
18. Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
19. Jacob Whitehill, Paul Ruvolo, TingfanWu, Jacob Bergsma, and Javier Movellan. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems 22, 2009.
20. Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer G. Dy. Modeling Annotator Expertise: Learning When Everybody Knows a Bit of Something. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
21. Yan Yan, Rómer Rosales, Glenn Fung, and Jennifer G. Dy. Active Learning from Crowds. In Proceedings of the 28th International Conference on Machine Learning (ICML), 2011.