MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy

Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2016

Volumn , Issue , 2016, Pages 32-41

  Drapeau, Ryan ^a   Chilton, Lydia B ^a   Bragg, Jonathan ^a   Weld, Daniel S ^a  

^a University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BUDGET CONTROL; MAXIMUM PRINCIPLE;

AGGREGATION MECHANISM; AMAZON'S MECHANICAL TURKS; EXPECTATION MAXIMIZATION; HIGH QUALITY; HIGH-ACCURACY; LOCAL MINIMUMS; MECHANISM-BASED; SIMPLE++; WORK-FLOWS; WORKERS';

CROWDSOURCING;

EID: 85167627193     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. André, P.; Kraut, R. E.; and Kittur, A. 2014. Effects of Simultaneous and Sequential Work Structures on Distributed Collaborative Interdependent Tasks. In CHI, 139-148.
2. Angeli, G.; Tibshirani, J.; Wu, J.; and Manning, C. D. 2014. Combining distant and partial supervision for relation extraction. In EMNLP 2014, 1556-1567.
3. Bernstein, M. S.; Brandt, J.; Miller, R. C.; and Karger, D. R. 2011. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In UIST.
4. Blumenstock, J. E. 2008. Size matters: word count as a measure of quality on wikipedia. In WWW 2008, 1095-1096.
5. Dai, P.; Lin, C. H.; Mausam; and Weld, D. S. 2013. Pomdpbased control of workflows for crowdsourcing. Artificial Intelligence 202:52-85.
6. Dawid, A., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics 28(1):20-28.
7. Dekel, O., and Shamir, O. 2009a. Good learners for evil tecahers. In ICML.
8. Dekel, O., and Shamir, O. 2009b. Vox populi: Collecting highquality labels from a crowd. In COLT.
9. Demartini, G.; Difallah, D. E.; and Cudré-Mauroux, P. 2012. Zencrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW 2012.
10. Donahue, J., and Grauman, K. 2011. Annotator rationales for visual recognition. In ICCV 2011.
11. Doroudi, S.; Kamar, E.; Brunskill, E.; and Horvitz, E. 2016. Toward a Learning Science for Complex Crowdsourcing Tasks. In CHI.
12. Dow, S.; Kulkarni, A.; Klemmer, S.; and Hartmann, B. 2012. Shepherding the crowd yields better work. In CSCW 2012.
13. Giles, J. 2005. Internet encyclopaedias go head to head. Nature.
14. Grudin, J. 1994. Groupware and social dynamics: Eight challenges for developers. Commun. ACM 37(1):92-105.
15. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS.
16. Karger, D. R.; Oh, S.; and Shah, D. 2011. Budget-optimal crowdsourcing using low-rank matrix approximations. In Conference on Communication, Control, and Computing.
17. Kincaid, J. P.; Fishburne Jr, R. P.; Rogers, R. L.; and Chissom, B. S. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, DTIC Document.
18. Kittur, A., and Kraut, R. E. 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In CSCW 2008.
19. Kriplean, T.; Bonnar, C.; Borning, A.; Kinney, B.; and Gill, B. 2014. Integrating on-demand fact-checking with public dialogue. In CSCW, 1188-1199.
20. Lasecki, W. S., and Bigham, J. P. 2013. Interactive crowds: Realtime crowdsourcing and crowd agents. In Handbook of Human Computation. 509-521.
21. Lasecki, W. S.; Wesley, R.; Nichols, J.; Kulkarni, A.; Allen, J. F.; and Bigham, J. P. 2013. Chorus: A Crowd-Powered Conversational Assistant. In UIST, 151-162.
22. Little, G.; Chilton, L. B.; Goldman, M.; and Miller, R. C. 2009. TurKit: Tools for Iterative Tasks on Mechanical Turk. In HCOMP.
23. Liu, A.; Soderland, S.; Bragg, J.; Lin, C. H.; Ling, X.; and Weld, D. S. 2016. Effective crowd annotation for relation extraction. In Proceedings of NAACL and HLT 2016.
24. Mamykina, L.; Dimond, J.; Smyth, T.; and Gajos, K. Z. 2016. Learning From the Crowd: Observational Learning in Crowdsourcing Communities. In CHI.
25. Oleson, D.; Sorokin, A.; Laughlin, G. P.; Hester, V.; Le, J.; and Biewald, L. 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In Human Computation Workshop.
26. Prelec, D., and Seung, S. 2006. An algorithm that finds truth even if most people are wrong. Working paper.
27. Shapiro, S. S., and Wilk, M. B. 1965. An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591-611.
28. Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, 254-263.
29. Sun, Y.-A.; Dance, C. R.; Roy, S.; and Little, G. 2011. How to assure the quality of human computation tasks when majority voting fails. In Workshop on Computational Social Science and the Wisdom of Crowds, NIPS.
30. Surdeanu, M. 2013. Overview of the TAC2013 knowledge base population evaluation: English slot filling and temporal slot filling. In TAC 2013.
31. Suzuki, R.; Salehi, N.; Lam, M. S.; Marroquin, J. C.; and Bernstein, M. S. 2016. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships. In CHI 2016.
32. Wauthier, F. L., and Jordan, M. I. 2011. Bayesian bias mitigation for crowdsourcing. In NIPS.
33. Welinder, P.; Branson, S.; Belongie, S.; and Perona, P. 2010. The multidimensional wisdom of crowds. In NIPS.
34. Whitehill, J.; Ruvolo, P.; Bergsma, J.; Wu, T.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS.
35. Zaidan, O. F.; Eisner, J.; and Piatko, C. D. 2007. Using "annotator rationales" to improve machine learning for text categorization. In Proceedings of NAACL and HLT 2007.
36. Zhang, C.; Niu, F.; Ré, C.; and Shavlik, J. 2012a. Big data versus the crowd: Looking for relationships in all the right places. In ACL.
37. Zhang, H.; Law, E.; Miller, R.; Gajos, K.; Parkes, D. C.; and Horvitz, E. 2012b. Human computation tasks with global constraints. In CHI 2012.