Scheduling human intelligence tasks in multi-Tenant crowd-powered systems

25th International World Wide Web Conference, WWW 2016

Volumn , Issue , 2016, Pages 855-865

  Difallah, Djellel Eddine ^a   Demartini, Gianluca ^b   Cudré Mauroux, Philippe ^a  

^a UNIVERSITY OF FRIBOURG   (Switzerland)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Crowd Powered System; Crowdsourcing; Scheduling

Indexed keywords

CROWDSOURCING; EFFICIENCY; HUMAN COMPUTER INTERACTION; INFORMATION MANAGEMENT; SCHEDULING; WORLD WIDE WEB;

CONCURRENT USERS; CROWD-POWERED SYSTEM; CROWDSOURCING PLATFORMS; HUMAN INTELLIGENCE; SCHEDULING TECHNIQUES; SHARED RESOURCES; SYSTEM ARCHITECTURES; TASK EXECUTIONS;

SCHEDULING ALGORITHMS;

EID: 85011386598     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2872427.2883030     Document Type: Conference Paper

References (40)

1. O. Alonso and R. A. Baeza-Yates. Design and Implementation of Relevance Assessments Using Crowdsourcing. In ECIR, pages 153-164, 2011.
2. A. Anagnostopoulos, L. Becchetti, C. Castillo, A. Gionis, and S. Leonardi. Power in unity: forming teams in large-scale community systems. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 599-608. ACM, 2010.
3. A. Anagnostopoulos, L. Becchetti, C. Castillo, A. Gionis, and S. Leonardi. Online team formation in social networks. In WWW, pages 839-848. ACM, 2012.
4. J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, et al. Vizwiz: nearly real-Time answers to visual questions. In UIST, pages 333-342. ACM, 2010.
5. A. Bozzon, M. Brambilla, S. Ceri, M. Silvestri, and G. Vesci. Choosing the right crowd: expert finding in social networks. In EDBT 13, pages 637-648. ACM, 2013.
6. M. J. Crump, J. V. McDonnell, and T. M. Gureckis. Evaluating amazons mechanical turk as a tool for experimental behavioral research. PloS one, 8(3):e57410, 2013.
7. G. Demartini, D. E. Difallah, and P. Cudŕe-Mauroux. ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW, pages 469-478, 2012.
8. G. Demartini, D. E. Difallah, and P. Cudŕe-Mauroux. Large-scale linked data integration using probabilistic reasoning and crowdsourcing. The VLDB Journal, 22(5):665-687, 2013.
9. D. Deng, C. Shahabi, and U. Demiryurek. Maximizing the number of workers self-selected tasks in spatial crowdsourcing. In Proc. SIGSPATIAL/GIS, pages 324-333. ACM, 2013.
10. E. Diaz-Aviles and R. Kawase. Exploiting twitter as a social channel for human computation. In CrowdSearch, pages 15-19, 2012.
11. D. E. Difallah, M. Catasta, G. Demartini, and P. Cudŕe-Mauroux. Scaling-up the crowd: Micro-Task pricing schemes for worker retention and latency improvement. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.
12. D. E. Difallah, M. Catasta, G. Demartini, P. G. Ipeirotis, and P. Cudŕe-Mauroux. The dynamics of micro-Task crowdsourcing: The case of amazon mturk. In WWW, pages 238-247. ACM, 2015.
13. D. E. Difallah, G. Demartini, and P. Cudŕe-Mauroux. Pick-A-crowd: Tell me what you like, and ill tell you what to do. In WWW, pages 367-374, 2013.
14. S. Faradani, B. Hartmann, and P. G. Ipeirotis. Whats the right price? pricing tasks for finishing on time. In Human Computation, 2011.
15. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. CrowdDB: Answering queries with crowdsourcing. In SIGMOD 11, pages 61-72. ACM, 2011.
16. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica. Dominant resource fairness: fair allocation of multiple resource types. In NSDI11, pages 24-24. USENIX Association, 2011.
17. P. G. Ipeirotis. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2):16-21, 2010.
18. P. G. Ipeirotis and E. Gabrilovich. Quizz: Targeted crowdsourcing with a billion (potential) users. In WWW, pages 143-154. ACM, 2014.
19. R. Jurca and B. Faltings. Mechanisms for making crowds truthful. J. Artif. Intell. Res. (JAIR), 34:209-253, 2009.
20. R. Khazankin, H. Psaier, D. Schall, and S. Dustdar. Qos-based task scheduling in crowdsourcing environments. In Service-Oriented Computing, pages 297-311. Springer, 2011.
21. W. S. Lasecki, A. Marcus, J. M. Rzeszotarski, and J. P. Bigham. Using Microtask Continuity to Improve Crowdsourcing. Technical Report, 2014.
22. E. Law and L. v. Ahn. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(3):1-121, 2011.
23. S. Lazebnik, C. Schmid, J. Ponce, et al. Semi-local affine parts for object recognition. In British Machine Vision Conference (BMVC04), pages 779-788, 2004.
24. V. Nunia, B. Kakadiya, C. Hota, and M. Rajarajan. Adaptive Task Scheduling in Service Oriented Crowd Using SLURM. In ICDCIT, pages 373-385, 2013.
25. V. Rajan, S. Bhattacharya, L. E. Celis, D. Chander, K. Dasgupta, and S. Karanam. Crowdcontrol: An online learning approach for optimal task scheduling in a dynamic crowd platform. In ICML Workshop on Machine Learning meets Crowdsourcing, 2013.
26. S. B. Roy, I. Lykourentzou, S. Thirumuruganathan, S. Amer-Yahia, and G. Das. Optimization in knowledge-intensive crowdsourcing. CoRR, abs/1401.1302, 2014.
27. J. M. Rzeszotarski, E. Chi, P. Paritosh, and P. Dai. Inserting micro-breaks into crowdsourcing workflows. In HCOMP (Works in Progress /Demos), volume WS-13-18 of AAAI Workshops. AAAI, 2013.
28. N. Seemakurty, J. Chu, L. von Ahn, and A. Tomasic. Word sense disambiguation via human computation. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP 10, pages 60-63. ACM, 2010.
29. A. Singla and A. Krause. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In WWW 13, pages 1167-1178, 2013.
30. M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin. Mapreduce and parallel dbmss: friends or foes? Communications of the ACM, 53(1):64-71, 2010.
31. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. OMalley, S. Radia, B. Reed, and E. Baldeschwieler. Apache hadoop yarn: Yet another resource negotiator. In SOCC 13, pages 5:1-5:16. ACM, 2013.
32. K. Vertanen and P. O. Kristensson. A versatile dataset for text entry evaluations based on genuine mobile emails. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, pages 295-298. ACM, 2011.
33. L. von Ahn and L. Dabbish. Labeling images with a computer game. In CHI 04, pages 319-326. ACM, 2004.
34. J. Wang, S. Faridani, and P. Ipeirotis. Estimating the completion time of crowdsourced tasks using survival analysis models. Crowdsourcing for search and data mining (CSDM 2011), 31, 2011.
35. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. CrowdER: Crowdsourcing Entity Resolution. Proc. VLDB Endow., 5(11):1483-1494, July 2012.
36. F. L. Wauthier and M. I. Jordan. Bayesian bias mitigation for crowdsourcing. In NIPS, pages 1800-1808, 2011.
37. S. E. Whang, P. Lofgren, and H. Garcia-Molina. Question Selection for Crowd Entity Resolution. Proc. VLDB Endow., 6(6):349-360, Apr. 2013.
38. M. Yin, Y. Chen, and Y.-A. Sun. Monetary Interventions in Crowdsourcing Task Switching. In Proceedings of the 2nd AAAI Conference on Human Computation (HCOMP), 2014.
39. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In EuroSys 10, pages 265-278. ACM, 2010.
40. C. J. Zhang, L. Chen, H. V. Jagadish, and C. C. Cao. Reducing uncertainty of schema matching via crowdsourcing. Proc. VLDB Endow., 6(9):757-768, July 2013.