Getting more for less: Optimized Crowdsourcing with dynamic tasks and goals

WWW 2015 - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 592-602

  Kobren, Ari ^a   Tan, Chun How ^b   Ipeirotis, Panagiotis ^c   Gabrilovich, Evgeniy ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b GOOGLE INC   (United States)

^c NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MOTIVATION; WORLD WIDE WEB;

AMOUNT OF INFORMATION; DYNAMIC TASKS; EMPIRICAL EVALUATIONS; INSTANT GRATIFICATION; SHORT TERM; STAKE HOLDERS; SURVIVAL MODEL; USER PARTICIPATION;

CROWDSOURCING;

EID: 84968817591     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2736277.2741681     Document Type: Conference Paper

References (28)

1. S. S. Anand and B. Mobasher. Intelligent techniques for web personalization. In Proceedings of the 2003 international conference on Intelligent Techniques for Web Personalization, pages 1-36. Springer-Verlag, 2003.
2. A. Anderson, D. Huttenlocher, J. Kleinberg, and J. Leskovec. Steering user behavior with badges. In Proceedings of the 22nd international conference on World Wide Web, pages 95-106. International World Wide Web Conferences Steering Committee, 2013.
3. B. I. Aydin, Y. S. Yilmaz, Y. Li, Q. Li, J. Gao, and M. Demirbas. Crowdsourcing for multiple-choice question answering. In Twenty-Sixth IAAI Conference, 2014.
4. Y. Bachrach, T. Graepel, T. Minka, and J. Guiver. How to grade a test without knowing the answers|a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In Proceedings of the 29th International Conference on Machine Learning (ICML-12), pages 1183-1190, 2012.
5. D. Cosley, D. Frankowski, L. Terveen, and J. Riedl. Using intelligent task routing and contribution review to help communities build artifacts of lasting value. In Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 1037-1046. ACM, 2006.
6. P. Dai, C. H. Lin, D. S. Weld, et al. Pomdp-based control of workows for crowdsourcing. Artificial Intelligence, 202:52-85, 2013.
7. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20-28, 1979.
8. D. Easley and A. Ghosh. Incentives, gamification, and game theory: An economic approach to badge design. In Proceedings of the fourteenth ACM conference on Electronic commerce, pages 359-376. ACM, 2013.
9. M. Eirinaki and M. Vazirgiannis. Web mining for web personalization. ACM Transactions on Internet Technology (TOIT), 3(1):1-27, 2003.
10. P. G. Ipeirotis and E. Gabrilovich. Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the 23rd international conference on World wide web, pages 143-154. International World Wide Web Conferences Steering Committee, 2014.
11. M. Krieger, E. M. Stark, and S. R. Klemmer. Coordinating tasks on the commons: designing for personal goals, expertise and serendipity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1485-1494. ACM, 2009.
12. P. Langley. Machine learning for adaptive user interfaces. In KI-97: Advances in artificial intelligence, pages 53-62. Springer, 1997.
13. J. Lehmann, M. Lalmas, E. Yom-Tov, and G. Dupret. Models of user engagement. In User Modeling, Adaptation, and Personalization, pages 164-175. Springer, 2012.
14. J. M. Linacre et al.computer-Adaptive testing: A methodology whose time has come. Chae, S.-Kang, U.-Jeon, E.-Linacre, JM (eds.): Development of Computerised Middle School Achievement Tests, MESA Research Memorandum, (69), 2000.
15. A. Mao, E. Kamar, and E. Horvitz. Why stop now? predicting worker engagement in online crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing, 2013.
16. W. Mason and D. J. Watts. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2):100-108, 2010.
17. S. McBurney, E. Papadopoulou, N. Taylor, and H. Williams. Adapting pervasive environments through machine learning and dynamic personalization. In Parallel and Distributed Processing with Applications, 2008. ISPA'08. International Symposium on, pages 395-402. IEEE, 2008.
18. G. Neumann and J. R. Peters. Fitted q-iteration by advantage weighted regression. In Advances in neural information processing systems, pages 1177-1184, 2009.
19. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
20. M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming, volume 414. John Wiley & Sons, 2009.
21. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. The Journal of Machine Learning Research, 11:1297-1322, 2010.
22. S. Robertson, M. Vojnovic, and I. Weber. Rethinking the esp game. In CHI'09 Extended Abstracts on Human Factors in Computing Systems, pages 3937-3942. ACM, 2009.
23. J. M. Rzeszotarski, E. Chi, P. Paritosh, and P. Dai. Inserting micro-breaks into crowdsourcing workows. In First AAAI Conference on Human Computation and Crowdsourcing, 2013.
24. L. von Ahn. Duolingo: learn a language for free while helping to translate the web. In Proceedings of the 2013 international conference on Intelligent user interfaces, pages 1-2. ACM, 2013.
25. L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. recaptcha: Human-based character recognition via web security measures. Science, 321(5895):1465-1468, 2008.
26. T. P. Waterhouse. Pay by the bit: An information-Theoretic metric for collective human judgment. In Proceedings of the 2013 conference on Computer supported cooperative work, pages 623-638. ACM, 2013.
27. J. Whitehill, T.-f. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, pages 2035-2043, 2009.
28. M. B. Wilk and R. Gnanadesikan. Probability plotting methods for the analysis for the analysis of data. Biometrika, 55(1):1-17, 1968.