ICrowd: An adaptive crowdsourcing framework

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn 2015-May, Issue , 2015, Pages 1015-1030

  Fan, Ju ^a   Li, Guoliang ^b   Ooi, Beng Chin ^a   Tan, Kian Lee ^a   Feng, Jianhua ^b  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^b TSINGHUA UNIVERSITY   (China)

Author keywords

Adaptive task assignment; Crowdsourcing; Quality control

Indexed keywords

CROWDSOURCING;

AMAZON MECHANICAL TURKS; LOW QUALITIES; ON THE FLIES; QUALIFICATION TEST; REAL DATA SETS; TASK ASSIGNMENT;

QUALITY CONTROL;

EID: 84957558382     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2723372.2750550     Document Type: Conference Paper

References (35)

1. http://crowdflower.com/.
2. Amazon Mechanical Turk. https://www.mturk.com.
3. G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng., 17(6):734-749, 2005.
4. M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: enabling realtime crowd-powered interfaces. In UIST, pages 33-42, 2011.
5. M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In UIST, pages 313-322, 2010.
6. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. In NIPS, pages 601-608, 2001.
7. C. C. Cao, J. She, Y. Tong, and L. Chen. Whom to ask? jury selection for decision making tasks on micro-blog services. PVLDB, 5(11):1495-1506, 2012.
8. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20-28, 1979.
9. G. Demartini, D. E. Difallah, and P. Cudré-Mauroux. ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW, pages 469-478, 2012.
10. J. Fan, M. Lu, B. C. Ooi, W. Tan, and M. Zhang. A hybrid machine-crowdsourcing system for matching web tables. In IEEE 30th International Conference on Data Engineering, Chicago, ICDE 2014, IL, USA, March 31 - April 4, 2014, pages 976-987, 2014.
11. A. Feng, M. J. Franklin, D. Kossmann, T. Kraska, S. Madden, S. Ramesh, A. Wang, and R. Xin. Crowddb: Query processing with the vldb crowd. PVLDB, 4(12):1387-1390, 2011.
12. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: answering queries with crowdsourcing. In SIGMOD, pages 61-72, 2011.
13. S. Guo, A. G. Parameswaran, and H. Garcia-Molina. So who won?: dynamic max discovery with the crowd. In SIGMOD, pages 385-396, 2012.
14. T. H. Haveliwala. Topic-sensitive pagerank. In WWW, pages 517-526, 2002.
15. C. Ho, S. Jabbari, and J. W. Vaughan. Adaptive task assignment for crowdsourced classification. In ICML, pages 534-542, 2013.
16. C. Ho and J. W. Vaughan. Online task assignment in crowdsourcing markets. In AAAI, 2012.
17. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In SIGKDD Workshop on Human Computation, pages 64-67, 2010.
18. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.
19. D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1-24, 2014.
20. H. W. Kuhn. The hungarian method for the assignment problem. In 50 Years of Integer Programming 1958-2008 - From the Early Years to the State-of-the-Art, pages 29-47. 2010.
21. Q. Liu, J. Peng, and A. T. Ihler. Variational inference for crowdsourcing. In NIPS, pages 701-709, 2012.
22. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: A crowdsourcing data analytics system. PVLDB, 5(10):1040-1051, 2012.
23. A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. Human-powered sorts and joins. PVLDB, 5(1):13-24, 2011.
24. A. Marcus, E. Wu, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, pages 211-214, 2011.
25. A. G. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom. Optimal crowd-powered rating and filtering algorithms. PVLDB, 7(9):685-696, 2014.
26. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: algorithms for filtering data with humans. In SIGMOD, pages 361-372, 2012.
27. A. G. Parameswaran and N. Polyzotis. Answering queries using humans, algorithms and databases. In CIDR, pages 160-166, 2011.
28. A. G. Parameswaran, A. D. Sarma, H. Garcia-Molina, N. Polyzotis, and J. Widom. Human-assisted graph search: it's okay to ask questions. PVLDB, 4(5):267-278, 2011.
29. H. Park and J. Widom. Crowdfill: collecting structured data from the crowd. In SIGMOD, pages 577-588, 2014.
30. V. C. Raykar and S. Yu. Ranking annotators for crowdsourced labeling tasks. In NIPS, pages 1809-1817, 2011.
31. V. S. Sheng, F. J. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In SIGKDD, pages 614-622, 2008.
32. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. Crowder: Crowdsoucing entity resolution. PVLDB, 5(10), 2012.
33. J. Wang, G. Li, J. X. Yu, and J. Feng. Entity matching: How similar is similar. PVLDB, 4(10):622-633, 2011.
34. S. E. Whang, P. Lofgren, and H. Garcia-Molina. Question selection for crowd entity resolution. Technical report, Stanford University.
35. D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf. Learning with local and global consistency. In NIPS, 2003.