Approval voting and incentives in crowdsourcing

32nd International Conference on Machine Learning, ICML 2015

Volumn 1, Issue , 2015, Pages 10-19

  Shah, Nihar B ^a   Zhou, Dengyong ^b   Peres, Yuval ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COUPLINGS; CROWDSOURCING; LEARNING ALGORITHMS;

AMAZON MECHANICAL TURKS; AXIOMATIC CHARACTERIZATION; COMPENSATION MECHANISM; EMPIRICAL STUDIES; INCENTIVE COMPATIBLE; LABELED TRAINING DATA; PARTIAL KNOWLEDGE; THEORETICAL GUARANTEES;

LEARNING SYSTEMS;

EID: 84969567943     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (43)

1. Brams, Steven J and Fishburn, Peter C. Approval voting. American Political Science Review, 72(03):831-847, 1978.
2. Brams, Steven J and Kilgour, D Marc. Satisfaction approval voting. In Voting Power and Procedures, pp. 323-346. Springer, 2014.
3. Brier, Glenn W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1): 1-3, 1950.
4. Caragiannis, Ioannis, Kalaitzis, Dimitris, and Markakis, Evangelos. Approximation algorithms and mechanism design for minimax approval voting. In AAA/, 2010.
5. Collet, Leverne S. Elimination scoring: An empirical evaluation. Journal of Educational Measurement, 8(3):209-214, 1971.
6. Coombs, Clyde H. On the use of objective examinations. Educational and Psychological Measurement, 13 (2):308-310, 1953.
7. Coombs, Clyde H, Milholland, John Edgar, and Womer, Frank Burton. The assessment of partial knowledge. Educational and Psychological Measurement, 16(1): 13-37, 1956.
8. Dasgupta, Anirban and Ghosh, Arpita. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd international conference on World Wide Web, pp. 319-330. International World Wide Web Conferences Steering Committee, 2013.
9. Dawid, Alexander Philip and Skene, Allan M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics, pp. 20-28, 1979.
10. Doignon, Jean-Paul, Pekec, Aleksandar, and Regenwet-Ter, Michel. The repeated insertion model for rankings: Missing link between two subset choice models. Psy-chometrika, 69(l):33-54, 2004.
11. Falmagne, J-CL and Regenwetter, Michael. A random utility model for approval voting. Journal of Mathematical Psychology, 40(2): 152-159, 1996.
12. Faltings, Boi, Jurca, Radu, Pu, Pearl, and Tran, Bao Duy. Incentives to counter bias in human computation. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.
13. Gibbons, Jean D, Olkin, Ingram, and Sobel, Milton. A subset selection technique for scoring items on a multiple choice test. Psychometrika, 44(3):259-270, 1979.
14. Gneiting, Tilmann and Raftery, Adrian E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378, 2007.
15. Horst, Paul. The chance element in the multiple choice test item. The Journal of General Psychology, 6( 1 ):209-211, 1932.
16. Jain, Shaili, Chen, Yiling, and Parkes, David C. Designing incentives for online question and answer forums. In Proceedings of the 10th ACM conference on Electronic commerce, pp. 129-138, 2009.
17. Jones, W Paul and Loe, Scott A. Optimal number of questionnaire response categories more may not be better. SAGE Open, 3(2):2158244013489691, 2013.
18. Karger, David R, Oh, Sewoong, and Shah, Devavrat. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, pp. 1953-1961,2011.
19. Kazai, Gabriella, Kamps, Jaap, Koolen, Marijn, and Milic-Frayling, Natasa. Crowdsourcing for book search evaluation: impact of HIT design on comparative system ranking. In ACM SIGIR conference on Research and development in Information Retrieval, pp. 205-214, 2011.
20. Kellett, John and Mott, Kenneth. Presidential primaries: Measuring popular choice. Polity, pp. 528-537, 1977.
21. Lambert, Nicolas and Shoham, Yoav. Eliciting truthful answers to multiple-choice questions. In ACM conference on Electronic commerce, pp. 109-118, 2009.
22. Liu, Qiang, Peng, Jian, and Ihler, Alexander T. Variational inference for crowdsourcing. In NIPS, pp. 701-709, 2012.
23. Marley, AAJ. Aggregation theorems and the combination of probabilistic rank orders. In Probability models and statistical analyses for ranking data, pp. 216-240. Springer, 1993.
24. Masso, Jordi and Vorsatz, Marc. Weighted approval voting. Economic Theory, 36(1): 129-146, 2008.
25. Miller, George A. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
26. Miller, Nolan, Resnick, Paul, and Zeckhauser, Richard. Eliciting informative feedback: The peer-prediction method. Management Science, 51(9): 1359-1373, 2005.
27. Mullainathan, Sendhil, Schwartzstein, Joshua, and Shleifer, Andrei. Coarse thinking and persuasion. The Quarterly journal of economics, 123(2):577-619, 2008
28. Ottewell, Guy. The arthmetic of voting. In defence of variety, 1977.
29. Prelec, Drazen. A Bayesian truth serum for subjective data. Science, 306(5695):462-466, 2004.
30. Raykar, Vikas C, Yu, Shipeng, Zhao, Linda H, Valadez, Gerardo Hermosillo, Florin, Charles, Bogoni, Luca, and Moy, Linda. Learning from crowds. The Journal of Machine Learning Research, 11:1297-1322, 2010.
31. Regenwetter, Michel and Tsetlin, Ilia. Approval voting and positional voting methods: Inference, relationship, examples. Social Choice and Welfare, 22(3):539-566, 2004.
32. Saaty, Thomas L and Ozdemir, Mujgan S. Why the magic number seven plus or minus two. Mathematical and Computer Modelling, 38(3):233-244, 2003.
33. Savage, Leonard J. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783-801,1971.
34. Shah, Nihar B. and Zhou, Dengyong. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. arXiv:1408.1387, 2014.
35. Shah, Nihar B, Balakrishnan, Sivaraman, Bradley, Joseph K, Parekh, Abhay, Ramchandran, Kannan, and Wainwright, Martin. Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence. In AlStats, 2015.
36. Shiffrin, RM and Nosofsky, RM. Seven plus or minus two: a commentary on capacity limitations. Psychological review, 101(2):357, 1994.
37. Siddiqi, Hammad. Does coarse thinking matter for option pricing? evidence from an experiment. IUP Journal of Behavioral Finance, 8(2), 2011.
38. Vuurens, Jeroen, de Vries, Arjen P, and Eickhoff, Carsten. How much spam can you take? An analysis of crowdsourcing results to increase accuracy. In ACM SIGIR Workshop on Crowdsourcing for Information Retrieval, pp. 21-26, 2011.
39. Wais, Paul, Lingamneni, Shivaram, Cook, Duncan, Fen-Nell, Jason, Goldenberg, Benjamin, Lubarov, Daniel, Marin, David, and Simons, Hari. Towards building a high-quality workforce with Mechanical Turk. NIPS workshop on computational social science and the wisdom of crowds, 2010.
40. Weber, Robert J. Comparison of voting systems. New-Haven: Cowles Foundation Discussion paper A, 498, 1977.
41. Whitehill, Jacob, Ruvolo, Paul, Wu, Ting-Fan, Bergsma, Jacob, and Movellan, Javier. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, pp. 2035-2043, 2009.
42. Zhou, Dengyong, Piatt, John, Basu, Sumit, and Mao, Yi. Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems 25, pp. 2204-2212, 2012.
43. Zou, James, Meir, Reshef, and Parkes, David. Approval voting behavior in doodle polls. In The 5th Workshop on Computational Social Choice, June 2014.