Minimizing efforts in validating crowd answers

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn 2015-May, Issue , 2015, Pages 999-1014

  Hung, Nguyen Quoc Viet ^a   Thang, Duong Chi ^a   Weidlich, Matthias ^b   Aberer, Karl ^a  

^a EPFL   (Switzerland)

^b IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Crowdsourcing; Expectation maximization; Guiding user feedback; Validation

Indexed keywords

CROWDSOURCING; MAXIMUM PRINCIPLE;

BASELINE METHODS; COMPREHENSIVE EVALUATION; EXPECTATION - MAXIMIZATIONS; POST PROCESSING; PROBABILISTIC MODELING; SYNTHETIC DATASETS; USER FEEDBACK; VALIDATION;

QUALITY CONTROL;

EID: 84957602716     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2723372.2723731     Document Type: Conference Paper

References (52)

1. https://code.google.com/p/crowdvalidator/.
2. http://www.crowdflower.com/.
3. http://www.mturk.com/.
4. Y. Amsterdamer, Y. Grossman, T. Milo, and P. Senellart. Crowd mining. In SIGMOD, pages 241-252, 2013.
5. A. Arasu, M. Götz, and R. Kaushik. On active learning of record matching packages. In SIGMOD, pages 783-794, 2010.
6. A. Artikis, M. Weidlich, F. Schnitzler, I. Boutsis, T. Liebig, N. Piatkowski, C. Bockermann, K. Morik, V. Kalogeraki, J. Marecek, et al. Heterogeneous stream processing and crowdsourcing for urban traffic management. In EDBT, pages 712-723, 2014.
7. J. A. Blakeley, P.-A. Larson, and F. W. Tompa. Efficiently updating materialized views. In SIGMOD, pages 61-71, 1986.
8. C. C. Cao, J. She, Y. Tong, and L. Chen. Whom to ask?: jury selection for decision making tasks on micro-blog services. In VLDB, pages 1495-1506, 2012.
9. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. J. R. Stat. Soc., pages 20-28, 1979.
10. G. Demartini, D. E. Difallah, and P. Cudré-Mauroux. Zencrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW, pages 469-478, 2012.
11. X. L. Dong, L. Berti-Equille, Y. Hu, and D. Srivastava. Solomon: Seeking the truth via copying detection. In VLDB, pages 1617-1620, 2010.
12. X. L. Dong, L. Berti-Equille, and D. Srivastava. Truth discovery and copying detection in a dynamic world. In VLDB, pages 562-573, 2009.
13. X. L. Dong and F. Naumann. Data fusion: resolving data conflicts for integration. In VLDB, pages 1654-1655, 2009.
14. C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika, pages 211-218, 1936.
15. A. Galland, S. Abiteboul, A. Marian, and P. Senellart. Corroborating information from disagreeing views. In WSDM, pages 131-140, 2010.
16. F. Garcin, B. Faltings, R. Jurca, and N. Joswig. Rating aggregation in collaborative filtering systems. In RecSys, pages 349-352, 2009.
17. C. Gokhale, S. Das, A. Doan, J. F. Naughton, N. Rampalli, J. Shavlik, and X. Zhu. Corleone: Hands-off crowdsourcing for entity matching. In SIGMOD, pages 601-612, 2014.
18. D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley Longman, 1989.
19. R. G. Gomes, P. Welinder, A. Krause, and P. Perona. Crowdclustering. In NIPS, pages 558-566, 2011.
20. Q. Hu, Q. He, H. Huang, K. Chiew, and Z. Liu. Learning from crowds under experts supervision. In PAKDD, pages 200-211. 2014.
21. Z. Huang, A. Olteanu, and K. Aberer. Credibleweb: a platform for web credibility evaluation. In CHI, pages 1887-1892, 2013.
22. N. Q. V. Hung, N. T. Tam, L. N. Tran, and K. Aberer. An evaluation of aggregation techniques in crowdsourcing. In WISE, pages 1-15, 2013.
23. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. HCOMP, pages 64-67, 2010.
24. S. R. Jeffery, M. J. Franklin, and A. Y. Halevy. Pay-as-you-go user feedback for dataspace systems. In SIGMOD, pages 847-860, 2008.
25. H. J. Jung and M. Lease. Improving quality of crowdsourced labels via probabilistic matrix factorization. In HCOMP, pages 101-106, 2012.
26. H. Kajino, Y. Tsuboi, I. Sato, and H. Kashima. Learning from crowds and experts. In HCOMP, pages 107-113, 2012.
27. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.
28. G. Karypis and V. Kumar. Metis-unstructured graph partitioning and sparse matrix ordering system, version 2.0. 1995.
29. G. Kazai, J. Kamps, and N. Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In CIKM, pages 1941-1944, 2011.
30. C.-W. Ko, J. Lee, and M. Queyranne. An exact algorithm for maximum entropy sampling. Operations Research, pages 684-691, 1995.
31. S. K. Lam and J. Riedl. Shilling recommender systems for fun and profit. In WWW, pages 393-402, 2004.
32. K. Lee, J. Caverlee, and S. Webb. The social honeypot project: protecting online communities from spammers. In WWW, pages 1139-1140, 2010.
33. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: a crowdsourcing data analytics system. In VLDB, pages 1040-1051, 2012.
34. Q. V. H. Nguyen, T. T. Nguyen, Z. Miklós, K. Aberer, A. Gal, and M. Weidlich. Pay-as-you-go reconciliation in schema matching networks. In ICDE, pages 220-231, 2014.
35. M. O'Mahony, N. Hurley, N. Kushmerick, and G. Silvestre. Collaborative recommendation: A robustness analysis. TOIT, pages 344-377, 2004.
36. J. Pasternack and D. Roth. Latent credibility analysis. In WWW, pages 1009-1020, 2013.
37. A. J. Quinn and B. B. Bederson. Human computation: a survey and taxonomy of a growing field. In CHI, pages 1403-1412, 2011.
38. V. C. Raykar and S. Yu. Ranking annotators for crowdsourced labeling tasks. In NIPS, pages 1809-1817, 2011.
39. J. Reason. Human error. Cambridge university press, 1990.
40. J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI, pages 2863-2872, 2010.
41. N. Rubens, D. Kaplan, and M. Sugiyama. Active learning in recommender systems. In Recommender Systems Handbook, pages 735-767. Springer, 2011.
42. S. J. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Pearson Education, 2003.
43. C. E. Shannon. A mathematical theory of communication. SIGMOBILE, pages 3-55, 2001.
44. C. Sun, N. Rampalli, F. Yang, and A. Doan. Chimera: Large-scale classification using machine learning, rules, and crowdsourcing. In VLDB, pages 1529-1540, 2014.
45. J. Surowiecki. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies. Economies, Societies and Nations, 2004.
46. TRAVAIL. Global wage report 2012-13. International Labour Organization (ILO), 2012.
47. D. Wang, L. Kaplan, H. Le, and T. Abdelzaher. On truth discovery in social sensing: A maximum likelihood estimation approach. In IPSN, pages 233-244, 2012.
48. P. Welinder and P. Perona. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In CVPRW, pages 25-32, 2010.
49. M. Yakout, A. K. Elmagarmid, J. Neville, M. Ouzzani, and I. F. Ilyas. Guided data repair. In VLDB, pages 279-289, 2011.
50. T. Yan, V. Kumar, and D. Ganesan. Crowdsearch: Exploiting crowds for accurate real-time image search on mobile phones. In MobiSys, pages 77-90, 2010.
51. C. J. Zhang, L. Chen, H. V. Jagadish, and C. C. Cao. Reducing uncertainty of schema matching via crowdsourcing. In VLDB, pages 757-768, 2013.
52. B. Zhao, B. I. Rubinstein, J. Gemmell, and J. Han. A bayesian approach to discovering truth from conflicting sources for data integration. In VLDB, pages 550-561, 2012.