On optimality of jury selection in crowdsourcing

EDBT 2015 - 18th International Conference on Extending Database Technology, Proceedings

Volumn , Issue , 2015, Pages 193-204

  Zheng, Yudian ^a   Cheng, Reynold ^a   Maniu, Silviu ^a   Mo, Luyi ^a  

^a UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; BUDGET CONTROL; CROWDSOURCING; DECISION MAKING;

APPROXIMATE ALGORITHMS; COMPUTATION ALGORITHM; COMPUTATIONALLY EFFICIENT; ENTITY RESOLUTIONS; HIGHLY ACCURATE; RANDOMIZED STRATEGY; SENTIMENT ANALYSIS; VOTING STRATEGIES;

OPTIMIZATION;

EID: 84976285854     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.5441/002/edbt.2015.18     Document Type: Conference Paper

References (45)

1. A.P. Dawid and A.M. Skene. Maximum likelihood estimation of observer error-rates using em algorithm. Appl. Statist., 28(1): 20-28, 1979.
2. Ashish Goel and David Lee, Triadic Consensus: A Randomized Algorithm for Voting in a Crowd. http://arxiv.org/pdf/1210.0664v1.pdf.
3. Bayes' Theorem. http://en.wikipedia.org/wiki/Bayes'-theorem.
4. R. Boim, O. Greenshpan, T. Milo, S. Novgorodov, N. Polyzotis, and W. C. Tan. Asking the right questions in crowd data sourcing. In ICDE, 2012.
5. D. Bookstaber. Simulated annealing for traveling salesman problem.
6. D. Braham. Crowdsourcing as a model for problem solving: An introduction and cases. Convergence, 14(1): 75-90, 2008.
7. C. C. Cao, J. She, Y. Tong, and L. Chen. Whom to ask? jury selection for decision making tasks on micro-blog services. PVLDB, 5(11): 1495-1506, 2012.
8. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. J.R. Statist. Soc. B, 30(1): 1-38, 1977.
9. X. L. Dong, B. Saha, and D. Srivastava. Less is more: Selecting sources wisely for integration. PVLDB, 6(2): 37-48, 2012.
10. A. Drexl. A simulated annealing approach to the multiconstraint zero-one knapsack problem. Computing, 40: 1-8, 1988.
11. J. Duggan and C. Martinelli. Abayesian model of voting injuries. Games and Economic Behavior, 37(2): 259-294, 2001.
12. J. Fan, M. Lu, B. C. Ooi, W.-C. Tan, and M. Zhang. A hybrid machine-crowdsourcing system for matching web tables. In ICDE, 2014.
13. F. Farhadi, E. Hoseini, S. Hashemi, and A. Hamzeh. Teamfinder: A co-clustering based framework for finding an effective team of experts in social networks. In ICDM Workshops, pages 107-114, 2012.
14. A. Feng, M. J. Franklin, D. Kossmann, T. Kraska, S. Madden, S. Ramesh, A. Wang, and R. Xin. Crowddb: Query processing with the vldb crowd. PVLDB, 4(12): 1387-1390, 2011.
15. Full version. http://i.cs.hku.hk/~ydzheng2/Jury/Jury.pdf.
16. J. Gao, X. Liu, B. C. Ooi, H. Wang, and G. Chen. An online cost sensitive decision-making method in crowdsourcing systems. In SIGMOD, 2013.
17. N. Q. V. Hung, N. T. Tam, Z. Miklós, and K. Aberer. On leveraging crowdsourcing techniques for schema matching networks. In Database Systems for Advanced Applications, pages 139-154. Springer, 2013.
18. P. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In SIGKDD workshop, pages 64-67, 2010.
19. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science, New Series, 220(4598): 671-680, 1983.
20. A. Lacasse, F. Laviolette, M. Marchand, and F. Turgeon-Boutin. Learning with randomized majority votes. pages 162-177, 2010.
21. L. Landau and E. Lifshitz. Statistical Physics. Course of Theoretical Physics 5 (3 ed.). Oxford: Pergamon Press, 1980.
22. T. Lappas, K. Liu, and E. Terzi. Finding a team of experts in social networks. In KDD, pages 467-476, 2009.
23. N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Inf. Comput., 108(2): 212-261, Feb. 1994.
24. X. Liu, X. L. Dong, B. C. Ooi, and D. Srivastava. Online data fusion. PVLDB, 4(11): 932-943, 2011.
25. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: A crowdsourcing data analytics system. PVLDB, 5(10): 1040-1051, 2012.
26. T. Lu, P. Tang, A. D. Procaccia, and C. Boutilier. Bayesian vote manipulation: Optimal strategies and impact on welfare. In UAI, pages 543-553, 2012.
27. A. Marcus, E. Wu, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, pages 211-214, 2011.
28. L. Mo, R. Cheng, B. Kao, X. S. Yang, C. Ren, S. Lei, D. W. Cheung, and E. Lo. Optimizing plurality for human intelligence tasks. In CIKM, 2013.
29. S. Nitzan and J. Paroush. Collective decision making and jury theorems.
30. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: algorithms for filtering data with humans. In SIGMOD, pages 361-372, 2012.
31. H. Park, R. Pang, A. G. Parameswaran, H. Garcia-Molina, N. Polyzotis, and J. Widom. Deco: A system for declarative crowdsourcing. PVLDB, 2012.
32. Partition Problem. http://en.wikipedia.org/wiki/Partition-problem.
33. Random Ballot Voting. http://en.wikipedia.org/wiki/Random-ballot.
34. V. C. Raykar and S. Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research, 2012.
35. T. Rekatsinas, X. L. Dong, and D. Srivastava. Characterizing and selecting fresh data sources. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data, pages 919-930. ACM, 2014.
36. M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web, pages 155-164. International World Wide Web Conferences Steering Committee, 2014.
37. P. Venetis and H. Garcia-Molina. Quality control for comparison microtasks. In CrowdKDD, KDD 2012 Workshop, 2012.
38. L. von Ahn, B. Maurer, C. McMilen, D. Abraham, and M. Blum. Recaptcha: Human-based character recognition via web security measures. Science, 321(5895): 1465-1468, 2008.
39. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. Crowder: Crowdsourcing entity resolution. PVLDB, 5(11): 1483-1494, 2012.
40. J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T. Kraska, and T. Milo. A sample-and-clean framework for fast and accurate query processing on dirty data. In International Conference on Management of Data, SIGMOD 2014, Snowbird, UT, USA, June 22-27, 2014, pages 469-480, 2014.
41. J. Wang, G. Li, T. Kraska, M. J. Franklin, and J. Feng. Leveraging transitive relations for crowdsourced joins. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, New York, NY, USA, June 22-27, 2013, pages 229-240, 2013.
42. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043, 2009.
43. X. S. Yang, R. Cheng, L. Mo, B. Kao, and D. W. Cheung. On incentive-based tagging. In ICDE, pages 685-696, 2013.
44. C. J. Zhang, L. Chen, H. V. Jagadish, and C. C. Cao. Reducing uncertainty of schema matching via crowdsourcing. PVLDB, 6(9): 757-768, 2013.
45. B. Zhao, B. I. Rubinstein, J. Gemmell, and J. Han. A bayesian approach to discovering truth from conflicting sources for data integration. Proceedings of the VLDB Endowment, 5(6): 550-561, 2012.