Learning from multiple annotators with varying expertise

Machine Learning

Volumn 95, Issue 3, 2014, Pages 291-327

  Yan, Yan ^a   Rosales, Rómer ^b   Fung, Glenn ^c   Subramanian, Ramanathan ^d   Dy, Jennifer ^d  

^a YAHOO RESEARCH   (United States)

^b LINKEDIN CORPORATION   (United States)

^c SIEMENS CORPORATE RESEARCH   (United States)

^d Northeastern University   (United States)

Author keywords

Adversarial annotators; Classification; Crowdsourcing; Graphical models; Multiple labelers; Opinion aggregation

Indexed keywords

ARTIFICIAL INTELLIGENCE; CLASSIFICATION (OF INFORMATION);

ADVERSARIAL ANNOTATORS; CROWDSOURCING; GRAPHICAL MODEL; KNOWLEDGE SOURCES; MULTIPLE LABELERS; PRACTICAL PROBLEMS; PROBABILISTIC APPROACHES; STATISTICAL LEARNING;

SOCIAL SCIENCES COMPUTING;

EID: 84901497847     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-013-5412-1     Document Type: Article

References (33)

1. Asuncion, A., & Newman, D. (2007). UCI machine learning repository. URL http://www.ics.uci.edu/~mlearn/MLRepository.html.
2. Belkin, M., Matveeva, I., & Niyogi, P. (2004). Regularization and semi-supervised learning on large graphs. In Proceedings of the workshop on computational learning theory (pp. 624-638). San Mateo: Morgan Kaufmann.
3. Berkhin, P. (2002). Survey of clustering data mining techniques (on-line). URL http://www.ee.ucr.edu/~barth/EE242/clustering-survey.pdf.
4. Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Wortman, J. (2008). Learning bounds for domain adaptation. In Advances in neural information processing systems (Vol. 20, pp. 129-136). Cambridge: MIT Press.
5. Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. In International conference on machine learning (pp. 19-26). San Mateo: Morgan Kaufmann.
6. Corduneanu, A., & Jaakkola, T. S. (2005). Distributed information regularization on graphs. In Advances in neural information processing systems (Vol. 17, pp. 297-304). Cambridge: MIT Press.
7. Cover, T., & Thomas, J. (1991). Elements of information theory. New York: Wiley Interscience.
8. Crammer, K., Kearns, M., & Wortman, J. (2008). Learning from multiple sources. Journal of Machine Learning Research, 9, 1757-1774.
9. Dawid, A. P., & Skeene, A. M. (1979). Maximum likelihood estimation of observed error-rates using the EM algorithm. Applied Statistics, 28, 20-28.
10. Dekel, O., & Shamir, O. (2009). Good learners for evil teachers. In Proceedings of the 26th annual international conference on machine learning (pp. 233-240). New York: ACM.
11. Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood estimation from incomplete data. Journal of the Royal Statistical Society B 39(1).
12. Donmez, P., & Carbonell, J. G. (2008). Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In Conference on information and knowledge management (pp. 619-628). New York: ACM.
13. Frank, A., & Asuncion, A. (2010). UCI machine learning repository. University of California, Irvine, School of Information and Computer Sciences. URL http://archive.ics.uci.edu/ml.
14. Howe, J. (2008). Crowdsourcing: why the power of the crowd is driving the future of business, 1st edn. New York: Crown Publishing Group.
15. Hui, S. L., & Zhou, X. H. (1998). Evaluation of diagnostic tests without a gold standard. Statistical Methods in Medical Research, 7, 354-370.
16. Jin, R., & Ghahramani, Z. (2003). Learning with multiple labels. In Advances in neural information processing systems (pp. 897-904).
17. Kasneci, G., Gael, J. V., Stern, D., & Graepel, T. (2011). CoBayes: Bayesian knowledge corroboration with assessors of unknown areas of expertise. In Conference on web search and data mining (pp. 465-474).
18. Krishnapuram, B., Williams, D., Xue, Y., Hartemink, A., Carin, L., & Figueiredo, M. (2005). On semi-supervised classification. In Advances in neural information processing systems (pp. 721-728). Cambridge: MIT Press.
19. Nocedal, J., & Wright, S. (2003). Numerical optimization (2nd ed.). Berlin: Springer.
20. Paquet, U., Van Gael, J., Stern, D., Kasneci, G., Herbrich, R., & Graepel, T. (2010). Vuvuzelas and active learning for online classification. In NIPS workshop on computational social sciences and the wisdom of crowds.
21. Qazi, M., Fung, G., Krishnan, S., Rosales, R., Steck, H., Rao, B., Poldermans, D. D., & Chandrasekaran, D. (2007). Automated heart wall motion abnormality detection from ultrasound images using Bayesian networks. In Proceedings of the 20th international joint conference on artificial intelligence (pp. 519-525).
22. Raykar, V. C., & Yu, S. (2011). Ranking annotators for crowdsourced labeling tasks. In Advances in neural information processing systems (pp. 1809-1817). Cambridge: MIT Press.
23. Raykar, V. C., Yu, S., Zhao, L. H., Jerebko, A., Florin, C., Valadez, G. H., Bogoni, L., & Moy, L. (2009). Supervised learning from multiple experts: whom to trust when everyone lies a bit. In Proceedings of the 26th annual international conference on machine learning (pp. 889-896).
24. Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 614-622). New York: ACM.
25. Smyth, P., Fayyad, U., Burl, M., Perona, P., & Baldi, P. (1994). Inferring ground truth from subjective labeling of Venus images. In Advances in neural information processing systems (pp. 1085-1092).
26. Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing, association for computational linguistics, EMNLP '08, Stroudsburg, PA, USA (pp. 254-263).
27. Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. In CVPR workshop on Internet vision (pp. 1-8).
28. Spiegelhalter, D. J., & Stovin, P. (1983). An analysis of repeated biopsies following cardiac transplantation. Statistics in Medicine, 2(1), 33-40.
29. Welinder, P., Branson, S., Belongie, S., & Perona, P. (2011). The multidimensional wisdom of crowds. In Advances in neural information processing systems.
30. Yan, Y., Rosales, R., Fung, G., & Dy, J. (2010a). Modeling annotator expertise: learning when everybody knows a bit of something. In International conference on artificial intelligence and statistics (pp. 932-939).
31. Yan, Y., Rosales, R., Fung, G., & Dy, J. (2010b). Modeling multiple annotator expertise in the semi-supervised learning scenario. In Conference on uncertainty in artificial intelligence (pp. 674-682).
32. Zhou, D., Bousquet, O., Lal, T. N., Weston, J., & Schölkopf, B. (2004). Learning with local and global consistency. In Advances in neural information processing systems (Vol. 16, pp. 321-328). Cambridge: MIT Press.
33. Zhu, X. (2006). Semi-supervised learning literature survey (Technical Report TR-1530). Computer Sciences Department, University of Wisconsin Madison. URL http://www.cs.wisc.edu/~jerryzhu/pub/ssl-survey.pdf