Eliciting categorical data for optimal aggregation

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 2450-2458

  Ho, Chien Ju ^a   Frongillo, Rafael ^b   Chen, Yiling ^c  

^a Cornell University ^*  (United States)

^b UNIVERSITY OF COLORADO   (United States)

^c HARVARD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BAYESIAN MODEL; CATEGORICAL DATA; CROWDSOURCING PLATFORMS; ERROR RATE; INTERFACE DESIGNS; MULTIPLE CHOICE; QUALITY OF INFORMATION;

BAYESIAN NETWORKS;

EID: 85018921016     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. R. M. Frongillo, Y. Chen, and I. Kash. Elicitation for aggregation. In The Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
2. N. B. Shah and D. Zhou. Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing. In Neural Information Processing Systems, NIPS '15, 2015.
3. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
4. L. J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783-801, 1971.
5. T. Gneiting and A. E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359-378, 2007.
6. N.S. Lambert, D.M. Pennock, and Y. Shoham. Eliciting properties of probability distributions. In Proceedings of the 9th ACM Conference on Electronic Commerce, EC '08, pages 129-138. ACM, 2008.
7. R. Frongillo and I. Kash. Vector-Valued Property Elicitation. In Proceedings of the 28th Conference on Learning Theory, pages 1-18, 2015.
8. N. Miller, P. Resnick, and R. Zeckhauser. Eliciting informative feedback: The peer-prediction method. Management Science, 51(9):1359-1373, 2005.
9. D. Prelec. A bayesian truth serum for subjective data. Science, 306(5695):462-466, 2004.
10. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39:1-38, 1977.
11. V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
12. S. R. Cholleti, S. A. Goldman, A. Blum, D. G. Politte, and S. Don. Veritas: Combining expert opinions without labeled data. In Proceedings 20th IEEE international Conference on Tools with Artificial intelligence, 2008.
13. R. Jin and Z. Ghahramani. Learning with multiple labels. In Advances in Neural Information Processing Systems, Volume 15, pages 897-904, 2003.
14. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, Volume 22, pages 2035-2043, 2009.
15. A. P. Dawid and A. M. Skene. Maximum likeihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28:20-28, 1979.
16. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In The 25th Annual Conference on Neural Information Processing Systems (NIPS), 2011.
17. D. R. Karger, S. Oh, and D. Shah. Budget-optimal crowdsourcing using low-rank matrix approximations. In Proc. 49th Annual Conference on Communication, Control, and Computing (Allerton), 2011.
18. J. Zou and D. C. Parkes. Get another worker? Active crowdlearning with sequential arrivals. In Proceedings of the Workshop on Machine Learning in Human Computation and Crowdsourcing, 2012.
19. C. Ho, S. Jabbari, and J. W. Vaughan. Adaptive task assignment for crowdsourced classification. In The 30th International Conference on Machine Learning (ICML), 2013.
20. N. Lambert and Y. Shoham. Eliciting truthful answers to multiple-choice questions. In Proceedings of the Tenth ACM Conference on Electronic Commerce, EC '09, pages 109-118, 2009.
21. R. Frongillo and I. Kash. General truthfulness characterizations via convex analysis. In Web and Internet Economics, pages 354-370. Springer, 2014.
22. F. Aurenhammer. Power diagrams: properties, algorithms and applications. SIAM Journal on Computing, 16(1):78-96, 1987.
23. J. Witkowski and D. Parkes. A robust Bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence, AAAI '12, 2012.
24. V. Sheng, F. Provost, and P. Ipeirotis. Get another label? Improving data quality using multiple, noisy labelers. In ACM SIGKDD Conferences on Knowledge Discovery and Data Mining (KDD), 2008.
25. P. Ipeirotis, F. Provost, V. Sheng, and J. Wang. Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 2014.