Programmatic gold: Targeted and scalable quality assurance in crowdsourcing

AAAI Workshop - Technical Report

Volumn WS-11-11, Issue , 2011, Pages 43-48

  Oleson, David ^a   Sorokin, Alexander ^a   Laughlin, Greg ^a   Hester, Vaughn ^a   Le, John ^a   Biewald, Lukas ^a  

^a CrowdFlower   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; DATA ANNOTATION; EFFECTIVE TOOL; GOLD STANDARDS; MANUAL ANNOTATION; MANUAL WORK; NOVEL PROCESS; OVERALL QUALITY; POST PROCESSING;

GOLD; QUALITY CONTROL;

QUALITY ASSURANCE;

EID: 80055047049     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (16)

1. Alonso, O.; Rose, D. E.; and Stewart, B. 2008. Crowdsourcing for relevance evaluation. SIGIR Forum 42(2):9-15.
2. 2005. Amazon mechanical turk. http://www.mturk.com/.
3. 2011. Crowdflower. http://www.crowdflower.com/.
4. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics 28(1):20-28.
5. Dow, S.; Kulkarni, A.; Bunge, B.; Nguyen, T.; Klemmer, S.; and Hartmann, B. 2011. Shepherding the crowd: managing and providing feedback to crowd workers. In Proceedings of the CHI Extended Abstracts on Human factors in computing systems, 1669-1674. New York, NY, USA: ACM.
6. Holmes, S.; Kapelner, A.; and Lee, P. P. 2009. An interactive java statistical image segmentation system: Gemident. Journal of Statistical Software 30(10):1-20.
7. Howe, J. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. New York, NY, USA: Crown Publishing Group, 1 edition.
8. Ipeirotis, P.; Provost, F.; and Wang, J. 2010. Quality management on amazon mechanical turk. In KDD-HCOMP '10.
9. Kittur, A.; Chi, E. H.; and Suh, B. 2008. Crowdsourcing user studies with mechanical turk. In CHI, 453-456.
10. Le, J.; Edmonds, A.; Hester, V.; and Biewald, L. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010).
11. Lehdonvirta, V., and Ernkvist, M. 2011. Knowledge map of the virtual economy. http://www.infodev.org/en/Publication.1056.html.
12. 2011. Microtask. http://www.microtask.com/.
13. P. Welinder, P. P. 2010. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In CVPR.
14. 2011. Samasource. http://www.samasource.org/.
15. Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 254-263. Association for Computational Linguistics.
16. Sorokin, A., and Forsyth, D. 2008. Utility data annotation with amazon mechanical turk. In First International Workshop on Internet Vision, CVPR 08.