CrowdForge: Crowdsourcing complex work

UIST'11 - Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology

Volumn , Issue , 2011, Pages 43-52

  Kittur, Aniket ^a   Smus, Boris ^a   Khamkar, Susheel ^a   Kraut, Robert E ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Coordination; Crowdsourcing; Distributed processing; Human computation; MapReduce; Mechanical Turk

Indexed keywords

CROWDSOURCING; DISTRIBUTED PROCESSING; HUMAN COMPUTATION; MAP-REDUCE; MECHANICAL TURKS;

COMMERCE; DECISION MAKING; USER INTERFACES;

COORDINATION REACTIONS;

EID: 80755168388     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2047196.2047202     Document Type: Conference Paper

References (29)

1. Bal, H.E., Steiner, J.G. and Tanenbaum, A.S. Programming languages for distributed computing systems. ACM Computing Surveys (CSUR) (1989) vol. 21 (3).
2. Becker, G.S. and Murphy, K.M. The division of labor, coordination costs, and knowledge. The Quarterly Journal of Economics (1992) vol. 107 (4) pp. 1137-1160
3. Boto, http://code.google.com/p/boto/
4. Callison-Burch, C. Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 (2009) pp. 286-295
5. Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., and Singh, M. How well do line drawings depict shape? In ACM SIGGRAPH (2009), 1-9.
6. Dean, J. and Ghemawat, S. Map Reduce: Simplified data processing on large clusters. Communications of the ACM, 51, 1 (2008), 107-114.
7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., FeiFei, L. "ImageNet: A large-scale hierarchical image database," cvpr, pp.248-255, IEEE Conference on Computer Vision and Pattern Recognition (2009).
8. Diehl, M. and Stroebe, W. Productivity loss in brainstorming groups: Toward the solution of a riddle. Journal of personality and social psychology 53, 3 (1987), 497-509.
9. Django, http://www.djangoproject.com
10. Ipeirotis, P. (2010, Mar 3). The New Demographics of Mechanical Turk. http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of- mechanical-turk.html. Retrieved 9-21-20
11. Isard, M., Budiu, M., Yu, Y., Birrell, A., and Fetterly, D. Dryad: distributed data-parallel programs from sequential building blocks. Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, (2007), 59-72. (Pubitemid 47281575)
12. Heer, J. and Bostock, M. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In Proceedings of CHI (2010). ACM, New York, 203-212.
13. Horton, J. J., Rand, D.G. and Zeckhauser, R.J., The online laboratory: Conducting experiments in a real labor market (2010). NBER Working Paper Series, Vol. w15961.
14. Kittur, A., Chi, E., Suh, B. Crowdsourcing User Studies With Mechanical Turk. In Proceedings of CHI (2008).
15. Kittur, A., Lee, B., Kraut, R. E. Coordination in Collective Intelligence: The Role of Team Structure and Task Interdependence. In Proceedings of CHI (2009). ACM, New York.
16. Kittur, A. and Kraut, R. E. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proceedings of CSCW (2008). ACM, New York.
17. Little, G., Chilton, L.B., Goldman, M., and Miller, R.C. Turkit: human computation algorithms on mechanical turk. In Proceedings of UIST (2010), ACM, New York, 57-66.
18. Malone, T. and Crowston, K. The interdisciplinary study of coordination. ACM Computing Surveys, 26, (1994), 87-119.
19. Malone, T., Yates, J. and Benjamin, R. Electronic markets and electronic hierarchies. Communications of the ACM, 30 (6). 484-497.
20. Mintzberg, H. 1979. The Structuring of Organizations. Prentice-Hall, Englewood Cliffs, N.J.
21. Olston, C., Reed, B., Srivastava, U., Kumar, R., and Tomkins, A. Pig latin: a not-so-foreign language for data processing. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, (2008), 1099-1110.
22. Salganik, M.J., Dodds, P.S., & Watts, D.J. Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311 (2006), 854.
23. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A.Y. Cheap and fast - but is it good?: evaluating nonexpert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics (2008), 254-263.
24. Skillicorn, D.B., Talia, D. Models and languages for parallel computation, ACM Computing Surveys 30(2) (1998) 123-169.
25. Sorokin, A., Forsyth, D. Utility data annotation with Amazon Mechanical Turk, First IEEE Workshop on Internet Vision at CVPR (2008).
26. Van de Ven, A., Delbecq, A. and Koenig, R. Determinants of coordination modes within organizations. American Sociological Review, 41 (1976), 322-338.
27. Williamson, O. Transaction-Cost Economics: The Governance of Contractual Relations. The Journal of Law and Economics, 22 (2). 233.
28. Wikipedia Simple entry on New York City, http://simple.wikipedia.org/ wiki/New-York-City
29. Woods, D. & Cook, R. 1999. Perspectives on human error: Hindsight biases and local rationality. In F. Durso (Ed.), Handbook of applied cognition. NY: John Wiley.