Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing

Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013

Volumn , Issue , 2013, Pages 103-111

  Mao, Andrew ^a   Kamar, Ece ^b   Horvitz, Eric ^b  

^a HARVARD UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL RESEARCH; CROWDSOURCING; SENSITIVITY ANALYSIS; SOCIAL NETWORKING (ONLINE);

CITIZEN SCIENCE; GUIDING SYSTEMS; ONLINE CITIZENS; REAL-TIME SETTINGS; SCIENCE PROJECTS; SETS OF FEATURES; STATISTIC MODELING; USER BEHAVIORS; USER STUDY; WORKERS';

FORECASTING;

EID: 84923517612     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Adar, E.; Teevan, J.; and Dumais, S. T. 2008. Large scale analysis of web revisitation patterns. In Proceedings of the 26th ACM Conference on Human Factors in Computing Systems (CHI).
2. Anderson, A.; Huttenlocher, D.; Kleinberg, J.; and Leskovec, J. 2013. Steering user behavior with badges. In Proceedings of the 22nd International World Wide Web Conference (WWW).
3. Cosley, D.; Frankowski, D.; Terveen, L.; and Riedl, J. 2006. Using intelligent task routing and contribution review to help communities build artifacts of lasting value. In Proceedings of the 2006 ACM Conference on Human Factors in Computing Systems (CHI), CHI’06, 1037–1046. New York, NY, USA: ACM.
4. Dai, P.; Mausam; and Weld, D. S. 2010. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI).
5. Dai, P.; Mausam; and Weld, D. S. 2011. Artificial intelligence for artificial artificial intelligence. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI).
6. Huberman, B. A.; Romero, D. M.; and Wu, F. 2009. Crowdsourcing, attention and productivity. J. Inf. Sci. 35(6):758–765.
7. Ipeirotis, P. G. 2010. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17(2):16–21.
8. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS).
9. Kaufmann, N.; Schulze, T.; and Veit, D. 2011. More than fun and money. worker motivation in crowdsourcing–a study on mechanical turk. In Proceedings of the Seventeenth Americas Conference on Information Systems, 1–11.
10. Khatib, F.; Cooper, S.; Tyka, M. D.; Xu, K.; Makedon, I.; Popović, Z.; Baker, D.; and Players, F. 2011. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108(47):18949–18953.
11. Kittur, A., and Kraut, R. E. 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW), CSCW’08, 37–46. New York, NY, USA: ACM.
12. Kittur, A.; Nickerson, J. V.; Bernstein, M.; Gerber, E.; Shaw, A.; Zimmerman, J.; Lease, M.; and Horton, J. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, 1301–1318. ACM.
13. Kulkarni, A.; Can, M.; and Hartmann, B. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the 15th ACM Conference on Computer Supported Cooperative Work (CSCW).
14. Lin, C. H.; Mausam; and Weld, D. S. 2012. Dynamically switching between synergistic workflows for crowdsourcing. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI).
15. Lintott, C. J.; Schawinski, K.; Slosar, A.; Land, K.; Bamford, S.; Thomas, D.; Raddick, M. J.; Nichol, R. C.; Szalay, A.; Andreescu, D.; Murray, P.; and Vandenberg, J. 2008. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389:1179–1189.
16. Little, G.; Chilton, L. B.; Goldman, M.; and Miller, R. C. 2010. Exploring iterative and parallel human computation processes. In Proceedings of the 2nd Human Computation Workshop (HCOMP).
17. Mao, A.; Procaccia, A. D.; and Chen, Y. 2013. Better human computation through principled voting. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI).
18. McCaffrey, R. E. 2005. Using citizen science in urban bird studies. Urban Habitats 3(1):70–86.
19. Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010. Learning from crowds. JournalofMachineLearningResearch11(Apr):1297–1322.
20. Rogstadius, J.; Kostakos, V.; Kittur, A.; Smus, B.; Laredo, J.; and Vukovic, M. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media: Barcelona, Spain.
21. Sculley, D.; Malkin, R.; Basu, S.; and Bayardo, R. J. 2009. Predicting bounce rates in sponsored search advertisements. In Proceedings of the 15th International Conference on Knowledge Discovery and Data Mining (KDD).
22. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems 22(2035-2043):7–13.