Reliable Aggregation of Boolean Crowdsourced Tasks

Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015

Volumn , Issue , 2015, Pages 42-51

  de Alfaro, Luca ^a   Polychronopoulos, Vassilis ^a   Shavlovsky, Michael ^a  

^a University of California   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE;

BAYESIAN PARAMETER ESTIMATION; BINARY LABELS; CROWDSOURCING PLATFORMS; LABELINGS; NOVEL ALGORITHM; REPUTATION MODELING; SIMPLE++; UNSUPERVISED ALGORITHMS; USER REPUTATIONS; WEB CONTENT;

CROWDSOURCING;

EID: 85039860908     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1609/hcomp.v3i1.13240     Document Type: Conference Paper

References (20)

1. Adler, B.; de Alfaro, L.; and Pye, I. 2010. Detecting wikipedia vandalism using wikitrust. Notebook papers of CLEF 1:22–23.
2. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics 20–28.
3. de Alfaro, L.; Polychronopoulos, V.; and Shavlovsky, M. 2015. Reliable aggregation of boolean crowdsourced tasks. Technical Report UCSC-SOE-15-16, University of California, Santa Cruz.
4. Glickman, M. E. 1999. Parameter estimation in large dynamic paired comparison experiments. Applied Statistics 48:377–394.
5. Hong, L.; Page, S. E.; and Riolo, M. 2012. Incentives, information, and emergent collective accuracy. Managerial and Decision Economics 33(5-6):323–334.
6. Hung, N. Q. V.; Tam, N. T.; Tran, L. N.; and Aberer, K. 2013. An evaluation of aggregation techniques in crowdsourcing. In Web Information Systems Engineering–WISE 2013. Springer. 1–15.
7. Karger, D. R.; Oh, S.; and Shah, D. 2011. Iterative learning for reliable crowdsourcing systems. In NIPS, 1953–1961.
8. Koller, D., and Friedman, N. 2009. Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning. The MIT Press.
9. Liu, Q.; Peng, J.; and Ihler, A. T. 2012. Variational inference for crowdsourcing. In NIPS’12, 701–709.
10. Mola-Velasco, S. M. 2012. Wikipedia vandalism detection through machine learning: Feature review and new proposals: Lab report for pan at clef 2010. arXiv preprint arXiv:1210.5560.
11. Piech, C.; Huang, J.; Chen, Z.; Do, C.; Ng, A.; and Koller, D. 2013. Tuned models of peer assessment in moocs. arXiv preprint arXiv:1307.2579.
12. Potthast, M. 2010. Crowdsourcing a Wikipedia Vandalism Corpus. In Crestani, F.; Marchand-Maillet, S.; Chen, H.-H.; Efthimiadis, E.; and Savoy, J., eds., 33rd International ACM Conference on Research and Development in Information Retrieval (SIGIR 10), 789–790. ACM.
13. Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010. Learning from crowds. J. Mach. Learn. Res. 11:1297–1322.
14. Sheshadri, A., and Lease, M. 2013. Square: A benchmark for research on computing crowd consensus. In First AAAI Conference on Human Computation and Crowdsourcing.
15. Smyth, P.; Fayyad, U. M.; Burl, M. C.; Perona, P.; and Baldi, P. 1994. Inferring ground truth from subjective labelling of venus images. In Tesauro, G.; Touretzky, D. S.; and Leen, T. K., eds., NIPS, 1085–1092. MIT Press.
16. Snow, R.; O’Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP’08, 254–263. Stroudsburg, PA, USA: Association for Computational Linguistics.
17. Sprinthall, R. C. 2011. Basic statistical analysis.
18. Welinder, P.; Branson, S.; Belongie, S.; and Perona, P. 2010. The Multidimensional Wisdom of Crowds. In Lafferty, J.; Williams, C. K. I.; Shawe-Taylor, J.; Zemel, R. S.; and Culotta, A., eds., NIPS. 2424–2432.
19. Whitehill, J.; Wu, T.-f.; Bergsma, J.; Movellan, J. R.; and Ruvolo, P. L. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, 2035–2043.
20. Yedidia, J. S.; Freeman, W. T.; and Weiss, Y. 2003. Exploring artificial intelligence in the new millennium. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. chapter Understanding Belief Propagation and Its Generalizations, 239–269.