Comprehensive and reliable crowd assessment algorithms

Proceedings - International Conference on Data Engineering

Volumn 2015-May, Issue , 2015, Pages 195-206

  Joglekar, Manas ^a   Garcia Molina, Hector ^a   Parameswaran, Aditya ^b  

^a Stanford University   (United States)

^b University of Illinois at Urbana Champaign ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONFIDENCE INTERVAL; ERROR RATE; REAL-WORLD DATASETS;

WELL TESTING;

EID: 84940851758     PISSN: 10844627     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDE.2015.7113284     Document Type: Conference Paper

References (31)

1. L. Wasserman, All of Statistics, 2003.
2. M. Joglekar, H. Garcia-Molina, and A. Parameswaran, "Evaluating the crowd with confidence," in SIGKDD, 2013.
3. "Mechanical Turk," http://mturk. com.
4. A. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom, "Crowdscreen: Algorithms for filtering data with humans," in SIGMOD, 2012.
5. S. Guo, A. Parameswaran, and H. Garcia-Molina, "So Who Won? Dynamic Max Discovery with the Crowd," in SIGMOD, 2012.
6. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller., "Humanpowered sorts and joins," in VLDB, 2012.
7. K. Bellare, S. Iyengar, A. Parameswaran, and V. Rastogi, "Active sampling for entity matching," in KDD, 2012.
8. R. Gomes et al., "Crowdclustering," in NIPS, 2011.
9. J. Wang, T. Kraska, M. Franklin, and J. Feng, "Crowder: Crowdsourcing entity resolution," in VLDB, 2012.
10. A. Parameswaran, A. D. Sarma, H. Garcia-Molina, N. Polyzotis, and J. Widom, "Human-assisted graph search: it's okay to ask questions," in VLDB, 2011.
11. F. P. Ribeiro, D. A. F. Florêncio, and V. H. Nascimento, "Crowdsourcing subjective image quality evaluation," in ICIP, 2011.
12. G. Casella and R. Berger, Statistical Inference. Duxbury Resource Center, 2001.
13. A. P. Dawid and A. M. Skene, "Maximum likelihood estimation of observer error-rates using the em algorithm," Applied Statistics, vol. 28, 1979.
14. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the em algorithm," JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B, 1977.
15. M. R. Gupta and Y. Chen, "Theory and use of the em algorithm," Found. Trends Signal Process., vol. 4, 2011.
16. G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions (Wiley Series in Probability and Statistics), 2008.
17. B. Walczak and D. Massart, "Dealing with missing data: Part ii," Chemometrics and Intelligent Laboratory Systems, vol. 58, 2001.
18. J. Whitehill et al., "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," in NIPS, 2009.
19. V. Raykar et al., "Supervised learning from multiple experts: whom to trust when everyone lies a bit," in ICML, 2009.
20. V. C. Raykar and S. Yu, "Eliminating spammers and ranking annotators for crowdsourced labeling tasks," Journal of Machine Learning Research, vol. 13, 2012.
21. P. Welinder and P. Perona, "Online crowdsourcing: rating annotators and obtaining cost-effective labels," in CVPR, 2010.
22. A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis, "Identifying reliable workers swiftly," Stanford University, Infolab Technical Report, 2012.
23. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang, "Cdas: a crowdsourcing data analytics system," Proc. VLDB Endow., vol. 5.
24. Q. Liu, J. Peng, and A. Ihler, "Variational inference for crowdsourcing," in NIPS, 2012.
25. V. S. Sheng, F. Provost, and P. Ipeirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers," in SIGKDD, 2008.
26. P. Donmez et al., "Efficiently learning the accuracy of labeling sources for selective sampling," in KDD, 2009.
27. N. Dalvi, A. Dasgupta, R. Kumar, and V. Rastogi, "Aggregating crowdsourced binary ratings," in WWW, 2013.
28. A. Ghosh, S. Kale, and P. McAfee, "Who moderates the moderators? crowdsourcing abuse detection in user-generated content," in EC, 2011.
29. D. Karger, S. Oh, and D. Shah, "Effcient crowdsourcing for multi-class labeling," in SIGMETRICS, 2013.
30. R. Snow et al., "Cheap and fast-but is it good: evaluating non-expert annotations for natural language tasks," in EMNLP, 2008.
31. "(data provided by scott klemmer and chinmay kulkarni from stanford university). "