Improving quality of crowdsourced labels via probabilistic matrix factorization

AAAI Workshop - Technical Report

Volumn WS-12-08, Issue , 2012, Pages 101-106

  Jung, Hyun Joon ^a   Lease, Matthew ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MEASURE-ACCURACY; MISSING DATA PROBLEM; PROBABILISTIC MATRIX FACTORIZATIONS; RELEVANCE FEEDBACK; RELEVANCE JUDGMENT; SPARSE MATRICES; SYNTHETIC DATASETS; WEIGHTED VOTING;

QUALITY ASSURANCE;

MATRIX ALGEBRA;

EID: 84875728960     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (14)

1. Alonso, O.; Rose, D.; and Stewart, B. 2008. Crowdsourcing for relevance evaluation. SIGIR Forum 42(2):9-15.
2. Buckley, C.; Lease, M.; and Smucker, M. D. 2010. Overview of the TREC 2010 Relevance Feedback Track (Notebook). In Proc. of the 19th Text Retrieval Conference.
3. Chen, S.; Zhang, J.; Chen, G.; and Zhang, C. 2010. What if the irresponsible teachers are dominating? a method of training on samples and clustering on teachers.
4. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. In Applied Statistics, Vol. 28, No. 1.
5. Dekel, O., and Shamir, O. 2009. Vox populi: Collecting high-quality labels from a crowd. In COLT.
6. Ipeirotis, P.; Provost, F.; and Wang, J. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, 64-67. ACM.
7. Jung, H. J., and Lease, M. 2011. Improving Consensus Accuracy via Z-score and Weighted Voting. In Proceedings of the 3rd Human Computation Workshop (HCOMP) at AAAI.
8. Raykar, V.; Yu, S.; Zhao, L.; Valadez, G.; Florin, C.; Bogoni, L.; and Moy, L. 2010. Learning from crowds. The Journal of Machine Learning Research 99:1297-1322.
9. Salakhutdinov, R., and Mnih, A. 2008. Probabilistic matrix factorization. In Advances in Neural Information Processing Systems (NIPS), volume 20.
10. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining.
11. Smucker, M. 2012. Crowdsourcing with a Crowd of One and Other TREC 2011 Crowdsourcing and Web Track Experiments. In Proceedings of TREC.
12. Snow, R.; O'Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast - but is it good? In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
13. Welinder, P., and Perona, P. 2010. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In Computer Vision and Pattern Recognition Workshops (CVPRW), 25-32.
14. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems (NIPS).