On quality control and machine learning in crowdsourcing

AAAI Workshop - Technical Report

Volumn WS-11-11, Issue , 2011, Pages 97-102

  Lease, Matthew ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; DATA COLLECTION; DATA QUALITY CONTROL; DATA-DRIVEN; HUMAN COMPUTATION; POTENTIAL APPLICATIONS;

DATA ACQUISITION; LEARNING SYSTEMS;

QUALITY CONTROL;

EID: 80055052639     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Adar, E. 2011. Why I Hate Mechanical Turk Research (and Workshops). ACM Conference on Human Factors in Computing Systems (CHI) Workshop on Crowdsourcing and Human Computation.
2. Alonso, O.; Rose, D.; and Stewart, B. 2008. Crowdsourcing for relevance evaluation. ACM SIGIR Forum 42(2):9-15.
3. Bell, R., and Koren, Y. 2007. Lessons from the Netflix prize challenge. ACM SlGKDD Explorations Newsletter 9(2):75-79.
4. Blum, A., and Mitchell, T. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory (COLT), 92-100.
5. Brew, A., and Cunningham, P. 2010. The interaction between supervised learning and crowdsourcing. In NIPS Workshop on Computational Social Science and the Wisdom of the Crowds.
6. Davis, J.; Arderiu, J.; Lin, H.; Nevins, Z.; Schuon, S.; Gallo, O.; and Yang, M. 2010. The HPU. In Computer Vision and Pattern Recognition Workshops (CVPRW), 9-16.
7. Gashler, M.; Giraud-Carrier, C.; and Martinez, T. 2008. Decision tree ensemble: small heterogeneous is better than large homogeneous. In Machine Learning and Applications, 2008. ICMLA'08, Seventh International Conference on, 900-905. IEEE.
8. Halevy, A.; Norvig, P.; and Pereira, F. 2009. The unreasonable effectiveness of data. Intelligent Systems, IEEE 24(2):8-12.
9. Horowitz, D., and Kamvar, S. 2010. The anatomy of a large-scale social search engine. In Proceedings of WWW, 431-440. ACM.
10. Ipeirotis, P. G. 2011. The unreasonable effectiveness of Simplicity. February 6. http://behind-theenemy-lines.blogspot.com/2011/02/ unreasonableeffectiveness-of.html.
11. Kochhar, S.; Mazzocchi, S.; and Paritosh, P. 2010. The anatomy of a large-scale human computation engine. In Proceedings of the ACM SlGKDD Workshop on Human Computation, 10-17. ACM.
12. Linguistic Data Consortium, http://www.ldc.upenn.edu.
13. Little, G.; Chilton, L.; Goldman, M.; and Miller, R. 2009. Turkit: Tools for iterative tasks on mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, 29-30. ACM.
14. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the ACL, 311-318.
15. Ross, J.; Irani, L.; Silberman, M.; Zaldivar, A.; and Tomlinson, B. 2010. Who are the crowdworkers?: shifting demographics in mechanical turk. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, 2863-2872. ACM.
16. Settles, B. 2009. Active Learning Literature Survey. Technical Report 1648, University of Wisconsin-Madison Computer Sciences.
17. Smyth, P.; Fayyad, U.; Burl, M.; Perona, P.; and Baldi, P. 1995. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems 1085-1092.
18. Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. 2008. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 254-263.
19. Surowiecki, J. 2004. The Wisdom of Crowds.
20. von Ahn, L. 2005. Human Computation. Ph.D. Dissertation, Carnegie Mellon University. Tech. Report CMU-CS-05-193.
21. Voorhees, E. 2002. The philosophy of information retrieval evaluation. In Evaluation of cross-language information retrieval systems, 143-170.
22. Wallach, H., and Vaughan, J. W. 2010. Workshop on Computational Social Science and the Wisdom of Crowds. In NIPS.
23. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, 2035-2043.
24. Yan, T.; Kumar, V.; and Ganesan, D. 2010. CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones. In MobiSys, 77-90.