Evaluating the crowd with confidence

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn Part F128815, Issue , 2013, Pages 686-694

  Joglekar, Manas ^a   Garcia Molina, Hector ^a   Parameswaran, Aditya ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

Confidence; Crowdsourcing

Indexed keywords

CROWDSOURCING; DATA MINING;

CONFIDENCE; CONFIDENCE INTERVAL; ERROR RATE; REAL-WORLD DATASETS;

QUALITY CONTROL;

EID: 84906822235     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2487575.2487595     Document Type: Conference Paper

References (35)

1. Mechanical Turk. http://mturk.com.
2. Omar Alonso, Daniel E. Rose, and Benjamin Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42, 2008.
3. K. Bellare, S. Iyengar, A. Parameswaran, and V. Rastogi. Active sampling for entity matching. In KDD, 2012.
4. George Casella and Roger Berger. Statistical Inference. Duxbury Resource Center, 2001.
5. Steve Cooper and Mehran Sahami. Reflections on stanford's moocs. Communications of the ACM, 56(2):28-30, 2013.
6. Image Comparison Dataset. http://www.stanford.edu/-manasrj/ic-data.tar.gz.
7. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20-28, 1979.
8. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B, 39(1):1-38, 1977.
9. J. Whitehill et al. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS. 2009.
10. M. Motoyama et al. Recaptchas : Understanding captcha-solving services in an economic context. In USENIX Security Symposium '10.
11. P. Donmez et al. Efficiently learning the accuracy of labeling sources for selective sampling. In KDD, 2009.
12. R. Snow et al. Cheap and fast-but is it good?: Evaluating non-expert annotations for natural language tasks. In EMNLP, 2008.
13. V. Raykar et al. Supervised learning from multiple experts: Whom to trust when everyone lies a bit. In ICML, 2009.
14. S. Guo, A. Parameswaran, and H. Garcia-Molina. So Who Won? Dynamic Max Discovery with the Crowd. In SIGMOD, 2012.
15. Maya R. Gupta and Yihua Chen. Theory and use of the em algorithm. Found. Trends Signal Process., 4(3):223-296, March 2011.
16. Manas Joglekar, Hector Garcia-Molina, and Aditya Parameswaran. Infolab technical report, 2012. http://ilpubs.stanford.edu:8090/1051/.
17. David R. Karger, Sewoong Oh, and Devavrat Shah. Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs/1110.3564, 2011.
18. Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, and Meihui Zhang. Cdas: A crowdsourcing data analytics system. Proc. VLDB Endow., 5(10):1040-1051, June 2012.
19. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Human-powered sorts and joins. In VLDB, 2012.
20. Geoffrey J. McLachlan and Thriyambakam Krishnan. The EM Algorithm and Extensions (Wiley Series in Probability and Statistics). Wiley-Interscience, 2 edition, March 2008.
21. A. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: Algorithms for filtering data with humans. In SIGMOD, 2012.
22. A. Parameswaran, A. Das Sarma, H. Garcia-Molina, N. Polyzotis, and J. Widom. Human-Assisted graph search: it's okay to ask questions. In VLDB, 2011.
23. R. Gomes et al. Crowdclustering. In NIPS, 2011.
24. A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis. Identifying reliable workers swiftly. Infolab technical report, Stanford University, 2012.
25. Vikas C. Raykar and Shipeng Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research, 13:491-518, 2012.
26. Flavio P. Ribeiro, Dinei A. F. Florêncio, and Vitor H. Nascimento. Crowdsourcing subjective image quality evaluation. In ICIP, 2011.
27. D. B. Rubin. Multiple Imputation for Nonresponse in Surveys. Wiley, 1987.
28. V. S. Sheng, F. Provost, and P. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In SIGKDD, pages 614-622, 2008.
29. Yuandong Tian and Jun Zhu. Learning from crowds in the presence of schools of thought. In KDD, 2012.
30. B. Walczak and D.L. Massart. Dealing with missing data: Part ii. Chemometrics and Intelligent Laboratory Systems, 58(1):29 - 42, 2001.
31. J. Wang, T. Kraska, M. Franklin, and J. Feng. Crowder: Crowdsourcing entity resolution. In VLDB, 2012.
32. Larry Wasserman. All of Statistics. Springer, 2003.
33. P. Welinder and P. Perona. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In CVPR, 2010.
34. Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209-212, 1927.
35. O. Zaidan and C. Callison-Burch. Feasibility of human-in-The-loop minimum error rate training. In EMNLP, 2009.