Supervised learning from multiple experts: Whom to trust when everyone lies a bit

Proceedings of the 26th International Conference On Machine Learning, ICML 2009

Volumn , Issue , 2009, Pages 889-896

  Raykar, Vikas C ^a   Yu, Shipeng ^a   Zhao, Linda H ^b   Jerebko, Anna ^a   Florin, Charles ^a   Valadez, Gerardo Hermosillo ^a   Bogoni, Luca ^a   Moy, Linda ^c  

^a CAD and Knowledge Solutions   (United States)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

^c NEW YORK UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

GOLD STANDARDS; MAJORITY VOTING; PROBABILISTIC APPROACHES;

ALGORITHMS; LABELS; ROBOT LEARNING; SUPERVISED LEARNING;

EDUCATION;

EID: 71149084080     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (11)

1. Dawid, A. P., & Skeene, A. M. (1979). Maximum likei-hood estimation of observed error-rates using the EM algorithm. Applied Statistics, 28, 20-28.
2. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39, 1-38.
3. Frank, E., & Hall, M. (2001). A simple approach to ordinal classification. Lecture Notes in Computer Science, 145-156.
4. Hui, S. L., & Zhou, X. H. (1998). Evaluation of diagnostic tests without a gold standard. Statistical Methods in Medical Research, 7, 354-370.
5. Lugosi, G. (1992). Learning with an unreliable teacher. Pattern Recognition, 25, 79-87.
6. Neal, R. M., & Hinton, G. E. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models (pp. 355-368). Kluwer Academic Publishers.
7. Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get another label? Improving data quality and data mining using multiple, noisy labelers. Proceedings of the 14th ACM SIGK·DD International Conference on Knowledge Discovery and Data Mining (pp. 614-622).
8. Smyth, P. (1995). Learning with probabilistic supervision. In Computational learning theory and natural learning systems 3, 163-182. MIT Press.
9. Smyth, P., Fayyad, U., Burl, M., Perona, P., & Baldi, P. (1995). Inferring ground truth from subjective labelling of venus images. In Advances in neural information processing systems 7, 1085-1092.
10. Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254-263).
11. Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. Proceedings of the First IEEE Workshop on Internet Vision at CVPR 08 (pp. 1-8).