Truth inference in crowdsourcing: Is the problem solved?

Proceedings of the VLDB Endowment

Volumn 10, Issue 5, 2016, Pages 541-552

  Zheng, Yudian ^a   Li, Guoliang ^b   Li, Yuanbing ^b   Shan, Caihua ^a   Cheng, Reynold ^a  

^a UNIVERSITY OF HONG KONG   (Hong Kong)

^b TSINGHUA UNIVERSITY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; DATA MINING; SENTIMENT ANALYSIS;

COMPREHENSIVE EVALUATION; DATA MINING COMMUNITY; DATABASE COMMUNITY; ENTITY RESOLUTIONS; INFERENCE PROBLEM; LOW QUALITIES; REAL DATA SETS; TASK-BASED;

INFERENCE ENGINES;

EID: 85020468498     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/3055540.3055547     Document Type: Conference Paper

References (63)

1. https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/amt-ui.pdf.
2. Amazon mechanical turk. https://www.mturk.com/.
3. Chi-squared. https://en.wikipedia.org/wiki/Chi-squared_distribution.
4. Adult Datset. https://github.com/ipeirotis/Get-Another-Label/tree/master/data.
5. B. I. Aydin, Y. S. Yilmaz, Y. Li, Q. Li, J. Gao, and M. Demirbas. Crowdsourcing for multiple-choice question answering. In AAAI, pages 2946-2953, 2014.
6. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. JMLR, 3(Jan):993-1022, 2003.
7. R. Boim, O. Greenshpan, T. Milo, S. Novgorodov, N. Polyzotis, and W.-C. Tan. Asking the right questions in crowd data sourcing. In ICDE, pages 1261-1264, 2012.
8. D. G. Brizan and A. U. Tansel. A survey of entity resolution and record linkage methodologies. Communications of the IIMA, 6(3):5, 2015.
9. C. Buckley, M. Lease, and M. D. Smucker. Overview of the trec 2010 relevance feedback track (notebook). In The Nineteenth TREC Notebook, 2010.
10. C. Callison-Burch. Fast, cheap, and creative: evaluating translation quality using amazon's mechanical turk. In EMNLP, pages 286-295, 2009.
11. C. Chai, G. Li, J. Li, D. Deng, and J. Feng. Cost-effective crowdsourced entity resolution: A partial-order approach. In SIGMOD, pages 969-984, 2016.
12. CrowdFlower. http://crowdflower.com/.
13. Crowdsourcing Datasets. http://dbgroup.cs.tsinghua.edu.cn/ligl/crowddata/.
14. S. B. Davidson, S. Khanna, T. Milo, and S. Roy. Using the crowd for top-k and group-by queries. In ICDT, pages 225-236, 2013.
15. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20-28, 1979.
16. G. Demartini, D. E. Difallah, and P. Cudre-Mauroux. Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW, pages 469-478, 2012.
17. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society. Series B (methodological), pages 1-38, 1977.
18. B. Efron and R. J. Tibshirani. An introduction to the bootstrap. 1994.
19. J. Fan, G. Li, B. C. Ooi, K.-I. Tan, and J. Feng. icrowd: An adaptive crowdsourcing framework. In SIGMOD, pages 1015-1030, 2015.
20. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: answering queries with crowdsourcing. In S1GMOD, pages 61-72, 2011.
21. Y. Gao and A. Parameswaran. Finish them!: Pricing algorithms for human computation. PVLDB, 7(14):1965-1976, 2014.
22. S. Guo, A. G. Parameswaran, and H. Garcia-Molina. So who won?: dynamic max discovery with the crowd. In SIGMOD, pages 385-396, 2012.
23. D. Haas, J. Wang, E. Wu, and M. J. Franklin. Clamshell: Speeding up crowds for low-latency data labeling. PVLDB, 9(4):372-383, 2015.
24. H. Hu, Y. Zheng, Z. Bao, G. Li, J. Feng, and R. Cheng. Crowdsourced poi labelling: Location-aware result inference and task assignment. In ICDE, pages 61-72, 2016.
25. M. Joglekar, H. Garcia-Molina, and A. Parameswaran. Evaluating the crowd with confidence. In SIGKDD, pages 686-694, 2013.
26. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.
27. H.-C. Kim and Z. Ghahramani. Bayesian classifier combination. In AISTATS, pages 619-627, 2012.
28. D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT Press, 2009.
29. G. Li, J. Wang, Y. Zheng, and M. J. Franklin. Crowdsourced data management: Asurvey. TKDE, 28(9):2296-2319, 2016.
30. Q. Li, Y. Li, J. Gao, L. Su, B. Zhao, M. Demirbas, W. Fan, and J. Han. A confidence-aware approach for truth discovery on long-tail data. PVLDB, 8(4):425-436, 2014.
31. Q. Li, Y. Li, J. Gao, B. Zhao, W. Fan, and J. Han. Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In SIGMOD, pages 1187-1198, 2014.
32. B. Liu. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167, 2012.
33. Q. Liu, J. Peng, and A. T. Ihler. Variational inference for crowdsourcing. In NIPS, pages 692-700, 2012.
34. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: a crowdsourcing data analytics system. PVLDB, 5(10):1040-1051, 2012.
35. F. Ma, Y. Li, Q. Li, M. Qiu, J. Gao, S. Zhi, L. Su, B. Zhao, H. Ji, and J. Han. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In KDD, pages 745-754, 2015.
36. A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. Human-powered sorts and joins. PVLDB, 5(1):13-24, 2011.
37. A. Marcus, E. Wu, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, pages 211-214, 2011.
38. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: Algorithms for filtering data with humans. In SIGMOD, pages 361-372, 2012.
39. A. G. Parameswaran, H. Park, H. Garcia-Molina, N. Polyzotis, and J. Widom. Deco: declarative crowdsourcing. In CIKM, pages 1203-1212, 2012.
40. Project Page. http://dbgroup.cs.tsinghua.edu.cn/ligl/crowd_truth_inference/.
41. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. JMLR, 11(Apr):1297-1322, 2010.
42. C. E. Shannon. A mathematical theory of communication. SIGMOBILE Mob. Comput. Commun. Rev., 5(1):3-55, 2001.
43. E. D. Simpson, M. Venanzi, S. Reece, P. Kohli, J. Guiver, S. J. Roberts, and N. R. Jennings. Language understanding in the wild: Combining crowdsourcing and machine learning. In WWW, pages 992-1002, 2015.
44. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP, pages 254-263, 2008.
45. Twitter Sentiment. http://www.sananalytics.com/lab/twitter-sentiment/.
46. M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In WWW, pages 155-164, 2014.
47. P. Venetis, H. Garcia-Molina, K. Huang, and N. Polyzotis. Max algorithms in crowdsourcing environments. In WWW, pages 989-998, 2012.
48. L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. recaptcha: Human-based character recognition via web security measures. Science, 321(5895):1465-1468, 2008.
49. M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.
50. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. Crowder: Crowdsourcing entity resolution. PVLDB, 5(11):1483-1494, 2012.
51. P. Welinder, S. Branson, P. Perona, and S. J. Belongie. The multidimensional wisdom of crowds. In NIPS, pages 2424-2432, 2010.
52. S. E. Whang, P. Lofgren, and H. Garcia-Molina. Question selection for crowd entity resolution. PVLDB, 6(6):349-360, 2013.
53. J. Whitehill, T.-F. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043, 2009.
54. Y. Yan, R. Rosales, G. Fung, M. W. Schmidt, G. H. Valadez, L. Bogoni, L. Moy, and J. G. Dy. Modeling annotator expertise: Learning when everybody knows a bit of something. In AISTATS, pages 932-939, 2010.
55. X. Zhang, G. Li, and J. Feng. Crowdsourced top-k algorithms: An experimental evaluation. PVLDB, 9(8):612-623, 2016.
56. W. X. Zhao, J. Jiang, J. Weng, J. He, E.-P. Lim, H. Yan, and X. Li. Comparing twitter and traditional media using topic models. In ECIR, pages 338-349. 2011.
57. Z. Zhao, F. Wei, M. Zhou, W. Chen, and W. Ng. Crowd-selection query processing in crowdsourcing databases: A task-driven approach. EDBT, pages 397-408, 2015.
58. Y. Zheng, R. Cheng, S. Maniu, and L. Mo. On optimality of jury selection in crowdsourcing. In EDBT, pages 193-204, 2015.
59. Y. Zheng, G. Li, and R. Cheng. Docs: a domain-aware crowdsourcing system using knowledge bases. PVLDB, 10(4):361-372, 2016.
60. Y. Zheng, J. Wang, G. Li, R. Cheng, and J. Feng. Qasca: A quality-aware task assignment system for crowdsourcing applications. In SIGMOD, pages 1031-1046, 2015.
61. D. Zhou, S. Basu, Y. Mao, and J. C. Platt. Learning from the wisdom of crowds by minimax entropy. In NIPS, pages 2195-2203, 2012.
62. D. Zhou, Q. Liu, J. Platt, and C. Meek. Aggregating ordinal labels from crowds by minimax conditional entropy. In ICML, pages 262-270, 2014.
63. S. Zhu, Y. Wu, and D. Mumford. Minimax entropy principle and its application to texture modeling. Neural computation, 9(8):1627-1660, 1997.