Identifying and Accounting for Task-Dependent Bias in Crowdsourcing

Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015

Volumn , Issue , 2015, Pages 92-101

  Kamar, Ece ^a   Kapoor, Ashish ^a   Horvitz, Eric ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE;

ACTIVE LEARNING; CITIZEN SCIENCE; GROUND TRUTH; LARGE DATASETS; MODELING TASK; PROBABILISTIC GRAPHICAL MODELS; PROBABILISTIC MODELS; SCIENCE PROJECTS; TRANSFER LEARNING; WORKERS';

CROWDSOURCING;

EID: 85167565368     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1609/hcomp.v3i1.13238     Document Type: Conference Paper

References (30)

1. Bachrach, Y.; Graepel, T.; Minka, T.; and Guiver, J. 2012. How to grade a test without knowing the answers - A bayesian graphical model for adaptive crowdsourcing and aptitude testing. In International Conference on Machine Learning, ICML.
2. Bishop, C. M. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc.
3. Condorcet, M. d. 1785. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. L’imprimerie royale.
4. Dai, P.; Lin, C. H.; Mausam; and Weld, D. S. 2013. POMDP-based control of workflows for crowdsourcing. Artificial Intelligence 202:52–85.
5. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics 20–28.
6. Galaxy Zoo 2 data release. 2013. Galaxy zoo 2 data sets. http://data.galaxyzoo.org/.
7. Galaxy Zoo 2 Team. 2009. Galaxy Zoo 2. http://zoo2.galaxyzoo.org/.
8. Ipeirotis, P., and others. 2014. Dawid-Skene Algorithm Implementation. https://github.com/ipeirotis/get-another-label.
9. Ipeirotis, P. G.; Provost, F.; and Wang, J. 2010. Quality management on amazon mechanical turk. In ACM SIGKDD workshop on human computation, 64–67. ACM.
10. Ipeirotis, P. G. 2010. Analyzing the Amazon Mechanical Turk marketplace. ACM Crossroads 17(2):16–21.
11. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In International Conference on Autonomous Agents and Multiagent Systems, AAMAS.
12. Kim, H., and Ghahramani, Z. 2012. Bayesian classifier combination. In International Conference on Artificial Intelligence and Statistics, AISTATS, 619–627.
13. Koller, D., and Friedman, N. 2009. Probabilistic Graphical Models - Principles and Techniques. MIT Press.
14. Lintott, C. J.; Schawinski, K.; Slosar, A.; Land, K.; Bamford, S.; Thomas, D.; Raddick, M. J.; Nichol, R. C.; Szalay, A.; Andreescu, D.; Murray, P.; and Vandenberg, J. 2008. Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389(3):1179–1189.
15. Liu, C., and Wang, Y. 2012. Truelabel + confusions: A spectrum of probabilistic models in analyzing multiple ratings. In International Conference on Machine Learning, ICML.
16. MacKay, D. J. C. 1992. Information-based objective functions for active data selection. Neural Computation 4(4):590–604.
17. Minka, T., and Winn, J. 2008. Gates: A graphical notation for mixture models. In Advances in Neural Information Processing Systems, 1073–1080.
18. Minka, T.; Winn, J.; Guiver, J.; Webster, S.; Zaykov, Y.; Yangel, B.; Spengler, A.; and Bronskill, J. 2014. Infer.NET 2.6. Microsoft Research Cambridge. http://research.microsoft.com/infernet.
19. Pearl, J. 1989. Probabilistic reasoning in intelligent systems - networks of plausible inference. Morgan Kaufmann.
20. Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010. Learning from crowds. Journal of Machine Learning Research 11:1297–1322.
21. Sheshadri, A., and Lease, M. 2013. SQUARE: A benchmark for research on computing crowd consensus. In AAAI Conference on Human Computation and Crowdsourcing, HCOMP.
22. Simpson, E.; Roberts, S. J.; Psorakis, I.; and Smith, A. 2013. Dynamic bayesian combination of multiple imperfect classifiers. In Decision Making and Imperfection. 1–35.
23. Venanzi, M.; Guiver, J.; Kazai, G.; Kohli, P.; and Shokouhi, M. 2014. Community-based bayesian aggregation models for crowdsourcing. In International World Wide Web Conference, WWW, 155–164.
24. von Ahn, L., and Dabbish, L. 2008. Designing games with a purpose. Commun. ACM 51(8):58–67.
25. Wauthier, F. L., and Jordan, M. I. 2011. Bayesian bias mitigation for crowdsourcing. In Conference on Neural Information Processing Systems, 1800–1808.
26. Welinder, P.; Branson, S.; Belongie, S.; and Perona, P. 2010. The multidimensional wisdom of crowds. In Conference on Neural Information Processing Systems, 2424–2432.
27. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. R. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Conference on Neural Information Processing Systems 2009, 2035–2043.
28. Willett, K. W.; Lintott, C. J.; Bamford, S. P.; Masters, K. L.; Simmons, B. D.; Casteels, K. R. V.; Edmondson, E. M.; Fortson, L. F.; Kaviraj, S.; Keel, W. C.; Melvin, T.; Nichol, R. C.; Raddick, M. J.; Schawinski, K.; Simpson, R. J.; Skibba, R. A.; Smith, A. M.; and Thomas, D. 2013. Galaxy Zoo 2: detailed morphological classifications for 304 122 galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society stt1458.
29. Yan, Y.; Rosales, R.; Fung, G.; and Dy, J. G. 2011. Active learning from crowds. In International Conference on Machine Learning, ICML, 1161–1168.
30. Zhou, D.; Platt, J. C.; Basu, S.; and Mao, Y. 2012. Learning from the wisdom of crowds by minimax entropy. In Conference on Neural Information Processing Systems, 2204–2212.