Incentives for effort in crowdsourcing using the peer truth serum

ACM Transactions on Intelligent Systems and Technology

Volumn 7, Issue 4, 2016, Pages

  Radanovic, Goran ^a   Faltings, Boi ^a   Jurca, Radu ^b  

^a EPFL   (Switzerland)

^b GOOGLE   (Switzerland)

Author keywords

Crowdsourcing; Mechanism design; Peer prediction

Indexed keywords

CURRICULA; GRADING; MACHINE DESIGN; WEBSITES;

MECHANISM DESIGN; ONLINE COURSE; RANDOM DATA; REAL-WORLD; WEB SITE CONTENTS;

CROWDSOURCING;

EID: 84964653355     PISSN: 21576904     EISSN: 21576912     Source Type: Journal    
DOI: 10.1145/2856102     Document Type: Article

References (33)

1. Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
2. R. N. Colvile, N. K. Woodfield, D. J. Carruthers, B. E. A. Fisher, A. Rickard, S. Neville, and A. Hughes. 2002. Uncertainty in dispersion modeling and urban air quality mapping. Environmental Science and Policy 5, 207-220.
3. Anirban Dasgupta and ArpitaGhosh. 2013. Crowd sourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd ACM International World Wide Web Conference (WWW13).
4. Boi Faltings, Jason J. Li, and Radu Jurca. 2014a. Incentive mechanisms for community sensing. IEEE Transactions on Computers 63, 115-128.
5. Boi Faltings, Pearl Pu, Bao Duy Tran, and Radu Jurca. 2014b. Incentives to counter bias in human computation. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowd sourcing (HCOMP'14).
6. Xi Alice Gao, Andrew Mao, and Yiling Chen. 2013. Trick or treat: Putting peer prediction to the test. In Workshop on Crowdsourcing and Online Behavioral Experiments, in Conjunction with ACM EC13.
7. Florent Garcin and Boi Faltings. 2014. Swiss noise: Online polls with game-theoretic incentives. Proceedings of the 26th Conference on Innovative Applications of AI, 2972-2977.
8. Sharad Goel, Daniel M. Reeves, and David M. Pennock. 2009. Collective revelation: A mechanism for self verified, weighted, and truthful predictions. In Proceedings of the 10th ACM Conference on Electronic Commerce (EC'09).
9. Christopher Harris. 2011. You're hired! An examination of crowd sourcing incentive models in human resource tasks. In Proceedings of the Workshop on Crowd sourcing for Search and Data Mining (CSDM) at the 4th ACM International Conference on Web Search and Data Mining (WSDM'11).
10. Shih-Wen Huang and Wai-Tat Fu. 2013a. Don't hide in the crowd!: Increasing social transparency between peer workers improves crowd sourcing outcomes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
11. Shih-Wen Huang and Wai-Tat Fu. 2013b. Enhancing reliability using peer consistency evaluation in human computation. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work.
12. Radu Jurca and Boi Faltings. 2009. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research (JAIR) 34, 209-253.
13. Radu Jurca and Boi Faltings. 2011. Incentives for answering hypothetical questions. In Workshop on Social Computing and User Generated Content (EC'11).
14. Ece Kamar and Eric Horvitz. 2012. Incentives for truthful reporting in crowd sourcing. In Proceedings of the 2012 International Conference on Autonomous Agents and Multiagent Systems (AAMAS'12).
15. Nicolas Lambert and Yoav Shoham. 2008. Truthful surveys. In Proceedings of the 3rd International Workshop on Internet and Network Economics (WINE'08).
16. Nolan Miller, Paul Resnick, and Richard Zeckhauser. 2005. Eliciting informative feedback: The peer prediction method. Management Science 51, 1359-1373.
17. Athanasios Papakonstantinou, Alex Rogers, Enrico Gerding, and Nicholas Jennings. 2011. Mechanism design for the truthful elicitation of costly probabilistic estimates in distributed information systems. Artificial Intelligence 175, 648-672.
18. Drazen Prelec. 2004. A bayesian truth serum for subjective data. Science 34, 5695 (2004), 462-466.
19. Drazen Prelec and Sebastian Seung. 2006. An algorithm that finds truth even if most people are wrong. Working paper.
20. Goran Radanovic and Boi Faltings. 2013. A robust Bayesian truth serum for non-binary signals. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI'13).
21. Goran Radanovic and Boi Faltings. 2014. Incentives for truthful information elicitation of continuous signals. In Proceedings of the 28th AAAI Conference on Artificial Intelligence (AAAI'14).
22. Goran Radanovic and Boi Faltings. 2015a. Incentive schemes for participatory sensing. In Proceedings of the 2015 International Conference on Autonomous Agents and Multi agent Systems (AAMAS'15).
23. Goran Radanovic and Boi Faltings. 2015b. Incentives for subjective evaluations with private beliefs. In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI'15).
24. Aaron Shaw, Daniel L. Chen, and John Horton. 2011. Designing incentives for inexpert human raters. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW'11).
25. Adish Singla and Andreas Krause. 2013. Truthful incentives in crowd sourcing tasks using regret minimization mechanisms. In Proceedings of the 22nd International Conference on World Wide Web.
26. Hossein Azari Soufiani, David C. Parkes, and Lirong Xia. 2012. Random utility theory for social choice. In Proceedings of Neural Information Processing Systems.
27. Bo Waggoner and Yiling Chen. 2013. Information elicitation sans verification. In Information Elicitation Sans Verification. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC'13).
28. Bo Waggoner and Yiling Chen. 2014. Output agreement mechanisms and common knowledge. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowd sourcing (HCOMP'14).
29. Jens Witkowski, Yoram Bachrach, Peter Key, and David C. Parkes. 2013. Dwelling on the negative: Incentivizing effort in peer prediction. In Proceedings of the 1st AAAI Conference on Human Computation and Crowd sourcing.
30. Jens Witkowski and David C. Parkes. 2012a. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce (EC'12). 964-981.
31. Jens Witkowski and David C. Parkes. 2012b. A robust Bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12).
32. Jens Witkowski and David C. Parkes. 2013. Learning the prior in minimal peer prediction. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC'13).
33. Peter Zhang and Yiling Chen. 2014. Elicitability and knowledge-free elicitation with peer prediction. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi agent Systems (AAMAS'14).