Quality assurance in crowdsourcing via matrix factorization based task routing

WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web

Volumn , Issue , 2014, Pages 3-7

  Jung, Hyun Joon ^a   Lease, Matthew ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Collaborative ltering; Crowdsourcing; Matrix factorization; Quality assurance; Recommendation; Task routing

Indexed keywords

CROWDSOURCING; FACTORIZATION; QUALITY ASSURANCE; WORLD WIDE WEB;

BASELINE METHODS; COLLABORATIVE LTERING; MATRIX FACTORIZATIONS; MISSING VALUES; PRELIMINARY ANALYSIS; RANDOM ASSIGNMENT; RECOMMENDATION; TASK ROUTING;

MATRIX ALGEBRA;

EID: 84991004954     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2567948.2567951     Document Type: Conference Paper

References (20)

1. Alonso, O., and Baeza-Yates, R. Design and implementation of relevance assessments using crowdsourcing. In Proceedings of the ECIR'11 (Berlin, Heidelberg, 2011), pp. 153-164.
2. Bernstein, M. S., Karger, D. R., Miller, R. C., and Brandt, J. Analytic methods for optimizing realtime crowdsourcing. In Collective Intelligence (2012).
3. Cohen, J. Statistical power analysis for the behavioral sciences, 2nd ed. Lawrence Erlbaum Assoc., 1988.
4. Cosley, D., Frankowski, D., Terveen, L., and Riedl, J. Suggestbot: using intelligent task routing to help people find work in wikipedia. In In Proceedings of IUI'07 (2007), pp. 32-41.
5. Ho, C., Jabbari, S., and Vaughan, J. W. Adaptive task assignment for crowdsourced classification. In Proceedings of the ICML'13 (2013), pp. 534-542.
6. Jung, H. J., and Lease, M. Improving Quality of Crowdsourced Labels via Probabilistic Matrix Factorization. In Proceedings of the AAAI HCOMP workshop'12 (2012), pp. 101-106.
7. Kamar, E., Hacker, S., and Horovitz, E. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the AAMAS'12 (2012), pp. 467-474.
8. Karger, D. R., Oh, S., and Shah, D. Iterative learning for reliable crowdsourcing systems. In Proceedings of the NIPS'11 (2011), pp. 1953-1961.
9. Kolobov, A., Mausam and Weld, D. S. Joint Crowdsourcing of Multiple Tasks. In Proceedings of the HCOMP'13 (2013).
10. Koren, Y., Bell, R., and Volinsky, C. Matrix factorization techniques for recommender systems. IEEE Computer 42, 8 (2009), 30-37.
11. Lee, J., Sun, M., and Lebanon, G. A comparative study of collaborative filtering algorithms. CoRR abs/1205.3193 (2012).
12. Paritosh, P. Human computation must be reproducible. In CrowdSearch: WWW Workshop on Crowdsourcing Web Search (2012), pp. 20-25.
13. Ramesh, A., Parameswaran, A., Garcia-Molina, H., and Polyzotis, N. Identifying reliable workers swiftly. Tech. rep., Stanford InfoLab, 2012.
14. Salakhutdinov, R., and Mnih, A. Probabilistic matrix factorization. In Proceedings of NIPS'08 (2008).
15. Shapiro, S., and Wilk, M. An analysis of variance test for normality (complete samples). Biometrika 3, 52 (1965).
16. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the EMNLP'08 (2008), pp. 254-263.
17. Vijayanarasimhan, S., and Grauman, K. Large-scale live active learning: Training object detectors with crawled data and crowds. In Proceedings of IEEE CVPR'11 (2011).
18. Yi, J., Jin, R., Jain, A. K., and Jain, S. Crowdclustering with sparse pairwise labels: A matrix completion approach. In Proceedings of the AAAI HComp workshop (2012), pp. 47-53.
19. Yuen, M., King, I., and Leung, K.-S. Task recommendation in crowdsourcing systems. In Proceedings of the CrowdKDD workshop (2012), pp. 22-26.
20. Zhang, H., Horvitz, E., Chen, Y., and Parkes, D. C. Task routing for prediction tasks. In Proceedings of the AAMAS'12 (2012), pp. 889-896.