Whose vote should count more: Optimal integration of labels from labelers of unknown expertise

Advances in Neural Information Processing Systems 22 - Proceedings of the 2009 Conference

Volumn , Issue , 2009, Pages 2035-2043

  Whitehill, Jacob ^a   Ruvolo, Paul ^a   Wu, Tingfan ^a   Bergsma, Jacob ^a   Movellan, Javier ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION;

FACE DETECTOR; INTERNET-BASED SERVICES; LABELED IMAGES; LEARNING-BASED APPROACH; LOW-COSTS; MACHINE-LEARNING; MODERN MACHINES; OPTIMAL INTEGRATION; VERY LARGE DATABASE; VISION SYSTEMS;

LEARNING SYSTEMS;

EID: 77951951247     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (18)

1. Amazon. Mechanical turk. http://www.mturk.com.
2. W. H. Batchelder and A. K. Romney. Test theory without an answer key. Psychometrika, 53(1):71-92, 1988.
3. A. Birnbaum. Some latent trait models and their use in inferring an examinee's ability. Statistical theories of mental test scores, 1968.
4. N. Butko and J. Movellan. I-POMDP: An infomax model of eye movement. In Proceedings of the International Conference on Development and Learning, 2008.
5. A. Dawid and A. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20-28, 1979.
6. I. Gauthier and M. Tarr. Becoming a "greeble" expert: Exploring mechanisms for face recognition. Vision Research, 37(12), 1997.
7. V. Johnson. On bayesian analysis of multi-rater ordinal data: An application to automated essay grading. Journal of the American Statistical Association, 91:42-51, 1996.
8. G. Karabatsos and W. H. Batchelder. Markov chain estimation for test theory without an answer key. Psychometrika, 68(3):373-389, 2003.
9. Omron. OKAO vision brochure, July 2008.
10. G. Rasch. Probabilistic Models for Some Intelligence and Attainment Tests. Denmark, 1960.
11. S. Rogers, M. Girolami, and T. Polajnar. Semi-parametric analysis of multi-rater data. Statistics and Computing, 2009.
12. V. Sheng, F. Provost, and P. Ipeirotis. Get another label? improving data quality and data mining using multiple noisy labelers. In Knowledge Discovery and Data Mining, 2008.
13. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi. Inferring ground truth from subjective labelling of venus images. In Advances of Neural Information Processing Systems, 1994.
14. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods on Natural Language Processing, 2008.
15. S. Steinbach, V. Rabaud, and S. Belongie. Soylent grid: it's made of people! In International Conference on Computer Vision, 2007.
16. D. Turnbull, R. Liu, L. Barrington, and G. Lanckriet. A Game-based Approach for Collecting Semantic Annotations of Music. In 8th International Conference on Music Information Retrieval (ISMIR), 2007.
17. L. von Ahn and L. Dabbish. Labeling Images with A Computer Game. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319-326. ACM Press New York, NY, USA, 2004.
18. L. von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science, 321(5895):1465, 2008.