Mobileworks: Designing for quality in a managed crowdsourcing architecture

IEEE Internet Computing

Volumn 16, Issue 5, 2012, Pages 28-35

  Kulkarni, Anand ^a   Gutheim, Philipp ^b   Narula, Prayag ^b   Rolnitzky, David ^b   Parikh, Tapan ^b   Hartmann, Björn ^b  

^a DEPARTMENT OF INDUSTRIAL ENGINEERING AND OPERATIONS RESEARCH   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

crowdsourcing; development; human computation; marketplace

Indexed keywords

CROWDSOURCING; DEVELOPMENT; HIGH-ACCURACY; HUMAN COMPUTATION; MARKETPLACE; SOCIAL INTERACTIONS;

INTERNET;

COMMERCE;

EID: 84867322288     PISSN: 10897801     EISSN: None     Source Type: Journal    
DOI: 10.1109/MIC.2012.72     Document Type: Article

References (19)

1. L. von Ahn and L. Dabbish, "Labeling Images with a Computer Game," Proc. Conf. Human-Computer Interaction (CHI), ACM, 2004, pp. 319-326.
2. G. Little et al., "Exploring Iterative and Parallel Human Computation," Proc. ACM SIGKDD Workshop Human Computation (HCOMP 10), ACM, 2010, pp. 68-76.
3. L. von Ahn, "Games with a Purpose," Computer, vol. 39, no. 6, 2006, pp. 92-94.
4. S. Cooper et al., "Predicting Protein Structures with an Online Game," Nature, vol. 466, 5 Aug. 2010, pp. 756-760.
5. L. von Ahn et al., "reCAPTCHA: Human-Based Character Recognition via Web Security Measures," Science, vol. 321, no. 5895, 2008, pp. 1465-1468.
6. D. Oleson et al., "Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing," Proc. ACM SIGKDD Workshop on Human Computation (HCOMP 11), ACM, 2011, pp. 43-48.
7. P. Ipeirotis, "Mechanical Turk, Low Wages, and the Market for Lemons," blog, 27 July 2010; www.behind-the-enemy-lines.com/2010/07/ mechanical-turklow-wages-and-market.html.
8. M.S. Bernstein et al., "Soylent: A Word Processor with a Crowd Inside," Proc. ACM Symp. User Interface Software and Technology (UIST 10), ACM, 2010, pp. 313-322.
9. A. Kulkarni, M. Can, and B. Hartmann, "Collaboratively Crowdsourcing Workflows with Turkomatic," Proc. Conf. Computer Supported Cooperative Work (CSCW 12), 2012, pp. 1003-1012.
10. S. Dow et al., "Shepherding the Crowd Yields Better Work," Proc. Conf. Computer Supported Cooperative Work (CSCW 12), 2012, pp. 1013-1022.
11. J.P. Bigham et al., "VizWiz: Nearly Real-Time Answers to Visual Questions," Proc. ACM Symp. User Interface Software and Technology (UIST 10), ACM, 2010, pp. 333-342.
12. M. Bernstein et al., "Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces," Proc. ACM Symp. User Interface Software and Technology (UIST 11), ACM, 2011, pp. 33-42.
13. N. Eagle, "txteagle: Mobile Crowdsourcing," Internationalization, Design and Global Development. LNCS 5623, Springer, 2009.
14. S. Khanna et al., "Evaluating and Improving the Usability of Mechanical Turk for Low-Income workers in India," Proc. 1st ACM Symp. Computing for Development (ACM DEV 10), ACM, 2010, pp. 12:1-12:10.
15. B.B. Bederson and A.J. Quinn, "Web Workers Unite! Addressing Challenges of Online Laborers," Proc. Ann. Conf. Extended Abstracts on Human Factors in Computing Systems (CHI EA 11), ACM, 2011, pp. 97-106.
16. M.S. Silberman, L. Irani, and R. Ross, "Ethics and Tactics of Professional Crowdwork," XRDS: Cossroads, vol. 17, no. 2, 2010, pp. 39-43.
17. V. Lehdonvirta and M. Enrkvist, "Knowledge Map of the Virtual Economy," InfoDev, 2011; www.infodev.org/en/Document.1056.pdf.
18. P. Ipeirotis, "Analyzing the Mechanical Turk Marketplace," XRDS: Crossroads, vol. 17, no. 2, 2010, pp. 16-21.
19. J. Ross et al., "Who Are the Crowdworkers? Shifting Demographics in Mechanical Turk," Proc. Int'l Conf. Extended Abstracts on Human Factors in Computing Systems (CHI 10), ACM, 2010, pp. 2863-2872.