Double or nothing: Multiplicative incentive mechanisms for crowdsourcing

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Shah, Nihar B ^a   Zhou, Dengyong ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Crowdsourcing; High quality labels; Mechanism design; Proper scoring rules; Supervised learning

Indexed keywords

LEARNING SYSTEMS; MACHINE DESIGN; SUPERVISED LEARNING;

HIGH QUALITY; INCENTIVE COMPATIBLE; INCENTIVE COMPATIBLE MECHANISMS; INCENTIVE MECHANISM; MACHINE LEARNING APPLICATIONS; MECHANISM DESIGN; PAYMENT MECHANISM; PROPER SCORING RULES;

CROWDSOURCING;

EID: 84995380129     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (66)

1. Nima Anari, Gagan Goel, and Afshin Nikzad. Mechanism design for crowdsourcing: An optimal 1-1/e competitive budget-feasible mechanism for large markets. In Foundations of Computer Science (FOCS), pages 266-275, 2014.
2. Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
3. Jason Baldridge and Alexis Palmer. How well does active learning actually work?: Time-based evaluation of cost-reduction strategies for language documentation. In Conference on Empirical Methods in Natural Language Processing, pages 296-305, 2009.
4. Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. Soylent: a word processor with a crowd inside. In ACM symposium on User interface software and technology (UIST), pages 313-322, 2010.
5. John Bohannon. Social science for pennies. Science, 334(6054):307-307, 2011.
6. GlennWBrier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
7. Peter Bühlmann and Torsten Hothorn. Boosting algorithms: Regularization, prediction and model fitting. Statistical Science, pages 477-505, 2007.
8. Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 2005.
9. Yang Cai, Constantinos Daskalakis, and Christos H Papadimitriou. Optimum statistical estimation with strategic data sources. In Conference on Learning Theory (COLT), 2015.
10. Izquierdo JM Cano, Yannis A Dimitriadis, Sánchez E Gómez, and Coronado J López. Learning from noisy information in fasart and fasback neuro-fuzzy systems. Neural networks: the official journal of the International Neural Network Society, 14(4-5):407-425, 2001.
11. Andrew Carlson, Justin Betteridge, Richard CWang, Estevam R Hruschka Jr, and Tom M Mitchell. Coupled semi-supervised learning for information extraction. In ACM international conference on Web search and data mining, pages 101-110, 2010.
12. Jenny J Chen, Natala J Menezes, Adam D Bradley, and TA North. Opportunities for crowdsourcing research on Amazon mechanical turk. Interfaces, 5(3), 2011.
13. Xi Chen, Paul N Bennett, Kevyn Collins-Thompson, and Eric Horvitz. Pairwise ranking aggregation in a crowdsourced setting. In ACM international conference on Web search and data mining, pages 193-202, 2013.
14. Xi Chen, Sivakanth Gopi, Jieming Mao, and Jon Schneider. Competitive analysis of the top-K ranking problem. arXiv preprint arXiv:1605.03933, 2016.
15. Fang Chu, Yizhou Wang, and Carlo Zaniolo. An adaptive learning approach for noisy data streams. In IEEE International Conference on Data Mining (ICDM), pages 351-354, 2004.
16. Vincent Conitzer. Prediction markets, mechanism design, and cooperative game theory. In Uncertainty in Artificial Intelligence (UAI), pages 101-108, 2009.
17. Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous proficiency. In International conference on World Wide Web (WWW), pages 319-330, 2013.
18. Alexander Philip Dawid and AllanMSkene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics, pages 20-28, 1979.
19. Ofer Dekel, Felix Fischer, and Ariel D Procaccia. Incentive compatible regression learning. In ACM-SIAM symposium on Discrete algorithms (SODA), pages 884-893, 2008.
20. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
21. Double or Nothing. http://wikipedia.org/wiki/Double-or-nothing, 2014. Last accessed: July 31, 2014.
22. Fang Fang, Maxwell Stinchcombe, and Andrew Whinston. Putting your money where your mouth is: A betting platform for better prediction. Review of Network Economics, 6(2), 2007.
23. Michael J Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. CrowdDB: answering queries with crowdsourcing. In ACM SIGMOD International Conference on Management of Data, pages 61-72, 2011.
24. Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378, 2007.
25. Steve Hanneke and Liu Yang. Negative results for active learning with convex losses. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 321-325, 2010.
26. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82-97, 2012.
27. Chien-Ju Ho, Shahin Jabbari, and JenniferWVaughan. Adaptive task assignment for crowdsourced classification. In International Conference on Machine Learning (ICML), pages 534-542, 2013.
28. Panagiotis G Ipeirotis, Foster Provost, Victor S Sheng, and Jing Wang. Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 28(2):402-441, 2014.
29. Srikanth Jagabathula, Lakshminarayanan Subramanian, and Ashwin Venkataraman. Reputationbased worker filtering in crowdsourcing. In Advances in Neural Information Processing Systems (NIPS), pages 2492-2500, 2014.
30. W Paul Jones and Scott A Loe. Optimal number of questionnaire response categories more may not be better. SAGE Open, 3(2):2158244013489691, 2013.
31. David R Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems (NIPS), 2011.
32. Gabriella Kazai, Jaap Kamps, Marijn Koolen, and Natasa Milic-Frayling. Crowdsourcing for book search evaluation: impact of HIT design on comparative system ranking. In ACM SIGIR conference on Research and development in Information Retrieval, pages 205-214, 2011.
33. Firas Khatib, Frank DiMaio, Seth Cooper, Maciej Kazmierczyk, Miroslaw Gilski, Szymon Krzywda, Helena Zabranska, Iva Pichova, James Thompson, Zoran Popović, Mariusz Jaskolski, and David Baker. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural&molecular biology, 18(10):1175-1177, 2011.
34. Ashish Khetan and Sewoong Oh. Reliable crowdsourcing under the generalized dawid-skene model. arXiv preprint arXiv:1602.03481, 2016.
35. Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-fei Li. L.: Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, CVPR, 2011.
36. Nicolas Lambert and Yoav Shoham. Eliciting truthful answers to multiple-choice questions. In ACM conference on Electronic commerce, pages 109-118, 2009.
37. ASID Lang and Joshua Rio-Ross. Using Amazon Mechanical Turk to transcribe historical handwritten documents. The Code4Lib Journal, 2011.
38. Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. A sparse texture representation using local affine regions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1265-1278, 2005.
39. John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, pages 21-26, 2010.
40. Eric WM Lee, Chee Peng Lim, Richard KK Yuen, and SM Lo. A hybrid neural network model for noisy data regression. IEEE Transactions on Systems, Man, and Cybernetics, 34(2):951-960, 2004.
41. Qiang Liu, Jian Peng, and Alexander T Ihler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems (NIPS), pages 701-709, 2012.
42. Philip M Long and Rocco A Servedio. Random classification noise defeats all convex potential boosters. Machine Learning, 78(3):287-304, 2010.
43. Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE Transactions on Cybernetics, 43(3):1146-1151, 2013.
44. David Mease, Abraham J Wyner, and Andreas Buja. Boosted classification trees and class probability/quantile estimation. The Journal of Machine Learning Research, 8:409-439, 2007.
45. George A Miller. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
46. Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peerprediction method. Management Science, 51(9):1359-1373, 2005.
47. Dražen Prelec. A Bayesian truth serum for subjective data. Science, 306(5695):462-466, 2004.
48. Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. The Journal of Machine Learning Research (JMLR), 11:1297-1322, 2010.
49. Leonard J Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783-801, 1971.
50. Nihar B Shah and Martin JWainwright. Simple, robust and optimal ranking from pairwise comparisons. arXiv preprint arXiv:1512.08949, 2015.
51. Nihar B Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J Wainwright. Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence. In Journal of Machine Learning Research (JMLR), 2016a.
52. Nihar B Shah, Sivaraman Balakrishnan, Adityanand Guntuboyina, and Martin JWainright. Stochastically transitive models for pairwise comparisons: Statistical and computational issues. International Conference on Machine Learning (ICML), 2016b.
53. Nihar B Shah, Sivaraman Balakrishnan, and Martin J Wainwright. A permutation-based model for crowd labeling: Optimal estimation and robustness. arXiv preprint arXiv:1606.09632, 2016c.
54. Richard M Shiffrin and Robert M Nosofsky. Seven plus or minus two: A commentary on capacity limitations. Psychological Review, 101(2):357-61, 1994.
55. Aditya Vempaty, Lav R Varshney, and Pramod K Varshney. Reliable crowdsourcing for multi-class labeling using coding theory. IEEE Journal of Selected Topics in Signal Processing, 8(4):667-679, 2014.
56. Luis Von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum. re-CAPTCHA: Human-based character recognition via web security measures. Science, 321(5895): 1465-1468, 2008.
57. Jeroen Vuurens, Arjen P de Vries, and Carsten Eickhoff. How much spam can you take? An analysis of crowdsourcing results to increase accuracy. In ACM SIGIR Workshop on Crowdsourcing for Information Retrieval, pages 21-26, 2011.
58. Paul Wais, Shivaram Lingamneni, Duncan Cook, Jason Fennell, Benjamin Goldenberg, Daniel
59. Lubarov, David Marin, and Hari Simons. Towards building a high-quality workforce with Mechanical Turk. NIPS workshop on computational social science and the wisdom of crowds, 2010.
60. Fabian L Wauthier and Michael Jordan. Bayesian bias mitigation for crowdsourcing. In Advances in Neural Information Processing Systems (NIPS), 2011.
61. Justin Wolfers and Eric Zitzewitz. Prediction markets. Technical report, National Bureau of Economic Research, 2004.
62. Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. Task matching in crowdsourcing. In IEEE International Conference on Cyber, Physical and Social Computing, pages 409-412, 2011.
63. Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I Jordan. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Advances in Neural Information Processing Systems (NIPS), 2014.
64. Dengyong Zhou, John Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems (NIPS), pages 2204-2212, 2012.
65. Dengyong Zhou, Qiang Liu, John C Platt, Christopher Meek, and Nihar B Shah. Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240, 2015.
66. Yuan Zhou, Xi Chen, and Jian Li. Optimal PAC multiple arm identification with applications to crowdsourcing. In International Conference on Machine Learning (ICML), pages 217-225, 2014.