Ability Grouping of Crowd Workers via Reward Discrimination

Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013

Volumn , Issue , 2013, Pages 147-155

  Sakurai, Yuko ^a   Okimoto, Tenda ^b   Oka, Masaaki ^a   Shinoda, Masato ^c   Yokoo, Makoto ^a  

^a KYUSHU UNIVERSITY   (Japan)

^b Transdisciplinary Research Integration Center   (Japan)

^c NARA WOMEN'S UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

AMAZON'S MECHANICAL TURKS; CONFIDENCE INTERVAL; EXPECTED UTILITY; PROBABILITY OF CORRECTNESS; WORKERS';

ARTIFICIAL INTELLIGENCE;

EID: 84992197431     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. Bacon, D. F.; Chen, Y.; Kash, I.; Parkes, D. C.; Rao, M.; and Sridharan, M. 2012. Predicting your own effort. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 695–702.
2. Boutilier, C. 2012. Eliciting forecasts from self-interested experts: Scoring rules for decision makers. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 737–744.
3. Cavallo, R., and Jain, S. 2012. Efficient crowdsourcing contests. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 677–686.
4. Dai, P.; Mausam; and Weld, D. S. 2010. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2010).
5. Gneiting, T., and Raftery, A. E. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477):359–378.
6. Ho, C.-J., and Vaughan, J. W. 2012. Online task assignment in crowdsourcing markets. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012).
7. Kato, K., and Zhang, Y. 2010. An item response model for probability testing. In International Meeting of the Psychometric Society.
8. Law, E., and Ahn, L. V. 2011. Human Computation. Morgan & Claypool Publishers.
9. Lin, C. H.; Mausam; and Weld, D. S. 2012. Crowdsourcing control: Moving beyond multiple choice. In Proceedings of the 4th Human Computation Workshop, 25–26.
10. Matheson, J. E., and Winkler, R. L. 1976. Scoring rules for continuous probability distributions. Management Science 22(10):1087–1096.
11. Nisan, N.; Roughgarden, T.; Tardos, E.; and Vazirani, V. V. 2007. Algorithmic Game Theory. Cambridge University Press.
12. Robu, V.; Kota, R.; Chalkiadakis, G.; Rogers, A.; and Jennings, N. R. 2012. Cooperative virtual power plant formation using scoring rules. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012), 370–376.
13. Rose, H.; Rogers, A.; and Gerding, E. H. 2012. A scoring rule-based mechanism for aggregate demand prediction in the smart grid. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 661–668.
14. Savage, L. J. 1971. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66(336):783–801.
15. Shaw, A. D.; Horton, J. J.; and Chen, D. L. 2011. Designing incentives for inexpert human raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (CSCW 2011), 275–284.
16. Snow, R.; O’Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), 254–263.
17. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems 22:2035–2043.
18. Witkowski, J., and Parkes, D. C. 2012a. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce (EC 2012), 964–981.
19. Witkowski, J., and Parkes, D. C. 2012b. A robust bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012), 1492–1498.