Learning from crowds in the presence of schools of thought

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn , Issue , 2012, Pages 226-234

  Tian, Yuandong ^a   Zhu, Jun ^b  

^a Carnegie Mellon University   (United States)

^b TSINGHUA UNIVERSITY   (China)

Author keywords

crowdsourcing; pattern analysis; schools of thought

Indexed keywords

CROWDSOURCING; EMPIRICAL STUDIES; EXPERIMENTAL DATA; FORM SCHOOL; GOLD STANDARDS; HYPER-PARAMETER; MECHANICAL TURKS; PATTERN ANALYSIS; SCHOOLS OF THOUGHT; SOCIAL SCIENTISTS; STATISTICAL MODELS;

DATA MINING; ESTIMATION; GOLD; INFERENCE ENGINES; RELIABILITY;

SOCIETIES AND INSTITUTIONS;

EID: 84866010071     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2339530.2339571     Document Type: Conference Paper

References (24)

1. J. Abernethy and R.M. Frongillo. A collaborative mechanism for crowdsourcing prediction problems. In NIPS, 2011.
2. E. Adar. Why I hate Mechanical Turk research (and workshops). In CHI, 2011.
3. C.E. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Annals of statistics, 2(6):1152-1174, 1974.
4. D. Blackwell and J.B. MacQueen. Ferguson distributions via Pólya urn schemes. Annals of statistics, 1(2):353-355, 1973.
5. AP Dawid and AM Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28, 1979.
6. S.J. Gershman and D. Blei. A tutorial on bayesian nonparametric models. Journal of Mathematical Psychology (in press), 2011.
7. T.L. Griffiths and M. Steyvers. Finding scientific topics. National Academy of Sciences of the United States of America, 101(Suppl 1):5228, 2004.
8. P.D. Hoff, A.E. Raftery, and M.S. Handcock. Latent space approaches to social network analysis. Journal of the American Statistical Association, 97(460):1090-1098, 2002.
9. P.G. Ipeirotis, F. Provost, and J. Wang. Quality management on Amazon Mechanical Turk. In ACM SIGKDD workshop on human computation, 2010.
10. D.R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, 2011.
11. A. Kittur, E. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. In CHI, 2008.
12. R.M. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of computational and graphical statistics, 9(2):249-265, 2000.
13. A. Ng, M. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, 2001.
14. D. Pinar, J. Carbonell, and J. Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In SIGKDD, 2009.
15. V.C. Raykar, S. Yu, L.H. Zhao, G.H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 99:1297-1322, 2010.
16. J. Ross, L. Irani, M. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers? Shifting demographics in Mechanical Turk. In CHI, 2010.
17. D.M. Russell, M.J. Stefik, P. Pirolli, and S.K. Card. The cost structure of sensemaking. In CHI, 1993.
18. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi. Inferring ground truth from subjective labelling of venus images. In NIPS, 1995.
19. R. Snow, B. O'Connor, D. Jurafsky, and A. Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, 2008.
20. Y.W. Teh, D. Newman, and M. Welling. A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In NIPS, 2006.
21. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In NIPS, 2010.
22. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, 2009.
23. Y. Yan, R. Rosales, G. Fung, M. Schmidt, G. Hermosillo, L. Bogoni, L. Moy, J.G. Dy, and PA Malvern. Modeling annotator expertise: Learning when everybody knows a bit of something. In AISTATS, 2010.
24. J. Zhu, N. Chen, and E. Xing. Infinite Latent SVM for Classification and Multi-task Learning. In NIPS, 2011.