Scoring workers in crowdsourcing: How many control questions are enough?

Advances in Neural Information Processing Systems

Volumn , Issue , 2013, Pages

  Liu, Qiang ^a   Steyvers, Mark ^b   Ihler, Alexander ^a  

^a Irvine   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONSENSUS METHODS; CONTINUOUS QUANTITIES; CROWDSOURCING; RULE OF THUMB; WORKERS';

EID: 84898947098     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (15)

1. James Surowiecki. The wisdom of crowds. Anchor, 2005.
2. Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proc. SIGKDD Int'l Conf. on Knowledge Discovery and Data Mining, pages 614-622. ACM, 2008.
3. A.P. Dawid and A.M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, pages 20-28, 1979.
4. Jacob Whitehill, Paul Ruvolo, TingfanWu, Jacob Bergsma, and Javier Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems (NIPS), pages 2035-2043, 2009.
5. D.R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems (NIPS), pages 1953-1961, 2011.
6. Qiang Liu, Jian Peng, and Alexander Ihler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems (NIPS), pages 701-709, 2012.
7. Dengyong Zhou, John Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems (NIPS), pages 2204- 2212, 2012.
8. A Kimball Romney, William H Batchelder, and Susan C Weller. Recent applications of cultural consensus theory. American Behavioral Scientist, 31(2):163-177, 1987.
9. Michael D Lee, Mark Steyvers, Mindy de Young, and Brent Miller. Inferring expertise in knowledge and prediction ranking tasks. Topics in cognitive science, 4(1):151-163, 2012.
10. Gary Chamberlain. Multivariate regression models for panel data. Journal of Econometrics, 18(1): 5-46, 1982.
11. Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
12. Shlomo Hoory, Nathan Linial, and AviWigderson. Expander graphs and their applications. Bulletin of the American Mathematical Society, 43(4):439-561, 2006.
13. Joel Friedman, Jeff Kahn, and Endre Szemeredi. On the second eigenvalue of random regular graphs. In Proc. ACM Symp. on Theory of Computing, pages 587-598. ACM, 1989.
14. Doron Puder. Expansion of random graphs: New proofs, new results. arXiv preprint arXiv:1212.5216, 2012.
15. Cade Massey, Joseph P Simmons, and David A Armor. Hope over experience: Desirability and the persistence of optimism. Psychological Science, 22(2):274-281, 2011.