Modeling Task Complexity in Crowdsourcing

Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2016

Volumn , Issue , 2016, Pages 249-258

  Yang, Jie ^a   Redi, Judith ^a   Demartini, Gianluca ^b   Bozzon, Alessandro ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; FORECASTING; REGRESSION ANALYSIS;

EXECUTION INTERFACE; HIGH-DIMENSIONAL REGRESSIONS; MODELING TASK; MONETARY REWARDS; REAL-WORLD; TASK COMPLEXITY; TASK PERFORMANCE; TASK TYPE; THEORY AND PRACTICE; WORKERS';

CROWDSOURCING;

EID: 85016497602     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Alonso, O.; Marshall, C.; and Najork, M. 2014. Crowdsourcing a subjective labeling task: a human-centered framework to ensure reliable results. Technical report, MSR-TR-2014-91.
2. Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent dirichlet allocation. the Journal of Machine Learning Research 3:993-1022.
3. Campbell, D. J. 1988. Task complexity: A review and analysis. Academy of Management Review 13(1):40-52.
4. Cheng, J.; Teevan, J.; Iqbal, S. T.; and Bernstein, M. S. 2015b. Break it down: A comparison of macro- and microtasks. In CHI, 4061-4064. ACM.
5. Cheng, J.; Teevan, J.; and Bernstein, M. S. 2015a. Measuring crowdsourcing effort with error-time curves. In CHI, 1365-1374. ACM.
6. Coursaris, C. K.; Swierenga, S. J.; and Watrall, E. 2008. An empirical investigation of color temperature and gender effects on web aesthetics. Journal of Usability Studies 3(3):103-117.
7. Difallah, D. E.; Catasta, M.; Demartini, G.; Ipeirotis, P. G.; and Cudré-Mauroux, P. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In WWW, 238-247. ACM.
8. Eickhoff, C., and de Vries, A. P. 2013. Increasing cheat robustness of crowdsourcing tasks. Information Retrieval 16(2):121-137.
9. Gadiraju, U.; Kawase, R.; and Dietze, S. 2014. A taxonomy of microtasks on the web. In HyperText, 218-223. ACM.
10. Halvey, M., and Villa, R. 2014. Evaluating the effort involved in relevance assessments for images. In SIGIR, 887-890. ACM.
11. Harper, S.; Michailidou, E.; and Stevens, R. 2009. Toward a definition of visual complexity as an implicit measure of cognitive load. ACM Transactions on Applied Perception (TAP) 6(2):10.
12. Hart, S. G., and Staveland, L. E. 1988. Development of nasa-tlx (task load index): Results of empirical and theoretical research. Advances in Psychology 52:139-183.
13. Hart, S. G. 2006. Nasa-task load index (nasa-tlx); 20 years later. In HFES, volume 50, 904-908. Sage Publications.
14. Hasler, D., and Suesstrunk, S. E. 2003. Measuring colorfulness in natural images. In Electronic Imaging, 87-95. International Society for Optics and Photonics.
15. Hoßfeld, T.; Keimel, C.; Hirth, M.; Gardlo, B.; Habigt, J.; Diepold, K.; and Tran-Gia, P. 2014. Best practices for qoe crowdtesting: Qoe assessment with crowdsourcing. IEEE Transactions on Multimedia 16(2):541-558.
16. Hoßfeld, T.; Schatz, R.; and Egger, S. 2011. Sos: The mos is not enough! In QoMEX, 131-136. IEEE.
17. Ivory, M. Y.; Sinha, R. R.; and Hearst, M. A. 2001. Empirically validated web page design metrics. In CHI, 53-60. ACM.
18. Jung, H. J., and Lease, M. 2015a. A discriminative approach to predicting assessor accuracy. In ECIR, 159-171. Springer.
19. Jung, H. J., and Lease, M. 2015b. Modeling temporal crowd work quality with limited supervision. HCOMP.
20. Kittur, A.; Nickerson, J. V.; Bernstein, M.; Gerber, E.; Shaw, A.; Zimmerman, J.; Lease, M.; and Horton, J. 2013. The future of crowd work. In CSCW, 1301-1318. ACM.
21. Law, E., and von Ahn, L. 2011. Human Computation. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers.
22. Lee, D. D., and Seung, H. S. 1999. Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788-791.
23. Lischke, L.; Mayer, S.; Wolf, K.; Henze, N.; Schmidt, A.; Leifert, S.; and Reiterer, H. 2015. Using space: Effect of display size on users' search performance. In CHI, 1845-1850. ACM.
24. Mitra, T.; Hutto, C.; and Gilbert, E. 2015. Comparing personand process-centric strategies for obtaining quality data on amazon mechanical turk. In CHI, 1345-1354. ACM.
25. Redi, J., and Povoa, I. 2014. Crowdsourcing for rating image aesthetic appeal: Better a paid or a volunteer crowd? In CrowdMM, 25-30. ACM.
26. Reinecke, K.; Yeh, T.; Miratrix, L.; Mardiko, R.; Zhao, Y.; Liu, J.; and Gajos, K. Z. 2013. Predicting users' first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness. In CHI, 2049-2058. ACM.
27. Rogstadius, J.; Kostakos, V.; Kittur, A.; Smus, B.; Laredo, J.; and Vukovic, M. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM.
28. Sheshadri, A., and Lease, M. 2013. Square: A benchmark for research on computing crowd consensus. In HCOMP.
29. Tibshirani, R. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological) 267-288.
30. Venanzi, M.; Guiver, J.; Kazai, G.; Kohli, P.; and Shokouhi, M. 2014. Community-based bayesian aggregation models for crowdsourcing. In WWW, 155-164. ACM.
31. Vukovic, M.; Laredo, J.; and Rajagopal, S. 2010. Challenges and experiences in deploying enterprise crowdsourcing service. In ICWE, 460-467. Springer.
32. Wood, R. E. 1986. Task complexity: Definition of the construct. Organizational Behavior and Human Decision Processes 37(1):60-82.
33. Yang, J., and Bozzon, A. 2016. On the improvement of quality and reliability of trust cues in micro-task crowdsourcing. In Weaving Relations of Trust in Crowd Work: Transparency and Reputation across Platforms. Workshop at WebScience 2016.
34. Yang, J.; Hauff, C.; Bozzon, A.; and Houben, G.-J. 2014. Asking the right question in collaborative q&a systems. In HyperText, 179-189. ACM.