Shepherding the crowd yields better work

Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW

Volumn , Issue , 2012, Pages 1013-1022

  Dow, Steven ^a   Kulkarni, Anand ^b   Klemmer, Scott ^c   Hartmann, Björn ^d  

^a Carnegie Mellon University   (United States)

^b DEPARTMENT OF INDUSTRIAL ENGINEERING AND OPERATIONS RESEARCH   (United States)

^c Stanford University   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

crowdsourcing; feedback system; human computation; review system

Indexed keywords

CROWDSOURCING; EXPERT FEEDBACK; FEEDBACK SYSTEM; FEEDBACK SYSTEMS; HIGH QUALITY; HUMAN COMPUTATION; ONLINE WORK; REAL-TIME ASSESSMENT; SELF-ASSESSMENT;

COMPUTER SUPPORTED COOPERATIVE WORK; INTERACTIVE COMPUTER SYSTEMS;

RATING;

EID: 84858213232     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2145204.2145355     Document Type: Conference Paper

References (37)

1. Annett, J. Feedback and human behaviour: the effects of knowledge of results, incentives, and reinforcement on learning and performance. Penguin Books, 1969.
2. Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. Proc of ACM Symposium on User Interface Software and Technology (2010), 313-322.
3. Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., and Yeh, T. VizWiz: nearly real-time answers to visual questions. Proc of ACM Symp. on User Interface Software and Technology (2010), 333-342.
4. Boud, D. Sustainable Assessment: Rethinking assessment for the learning society. Studies in Continuing Education 22, 2 (2000), 151.
5. Chandler, D. and Kapelner, A. Breaking monotony with meaning: Motivation in crowdsourcing markets. University of Chicago mimeo, (2010).
6. Cheshire, C. and Antin, J. The Social Psychological Effects of Feedback on the Production of Internet Information Pools. Journal of Computer-Mediated Communication 13, 3 (2008), 705-727.
7. David, B. Enhancing Learning Through Self-Assessment. Routledge, 1995.
8. Ericsson, K.A., Charness, N., Feltovich, P.J., and Hoffman, R.R. The Cambridge Handbook of Expertise and Expert Performance. Cambr. Univer. Press, 2006.
9. Gawande, A. The Checklist Manifesto: How to Get Things Right. Metropolitan Books, 2009.
10. Hanrahan, S.J. and Isaacs, G. Assessing Self- and Peer-assessment: The students' views. Higher Education Research & Development 20, 1 (2001), 53.
11. Havighurst, R.J. Human Development and Education. Longmans, Green and Co, 1955.
12. Heer, J. and Bostock, M. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. Proc of ACM conf on Human factors in computing systems (2010), 203-212.
13. Hinds, P. The Curse of Expertise: The Effects of Expertise and Debiasing Methods on Predictions of Novice Performance. Journal of Experimental Applied Psychology 5, (1999), 205-221.
14. Horton, J.J. Employer Expectations, Peer Effects and Productivity: Evidence from a Series of Field Experiments. SSRN eLibrary, (2010).
15. Hullman, J., Adar, E., and Shah, P. The impact of social information on visual judgments. Proc of conf on Human factors in computing systems, ACM (2011), 1461-1470.
16. Ipeirotis, P.G., Provost, F., and Wang, J. Quality management on Amazon Mechanical Turk. Proc of ACM SIGKDD Workshop on Human Computation, (2010), 64-67.
17. Ipeirotis, P.G. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17, 2010, 16-21.
18. Karger, D., Oh, S., and Shah, D. Budget-optimal Crowdsourcing using Low-rank Matrix Approximations. Proc. of the Allerton Conf. on Communication, Control, and Computing, (2011).
19. Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc of SIGCHI conference on Human factors in computing systems, ACM (2008), 453-456.
20. Lampe, C. and Resnick, P. Slash(dot) and burn: distributed moderation in a large online conversation space. Proc. of the SIGCHI conference on Human factors in computing systems, ACM (2004), 543-550.
21. Lave, J. and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Camb. University Press, 1991.
22. Levenshtein, V.I. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10, 8, 707-710.
23. Little, G., Chilton, L.B., Goldman, M., and Miller, R.C. Exploring iterative and parallel human computation processes. Proc. of the ACM SIGKDD Workshop on Human Computation, ACM (2010), 68-76.
24. Masten, A.S., Morison, P., and Pellegrini, D.S. A revised class play method of peer assessment. Developmental Psychology 21, 3 (1985), 523-533.
25. Mattheos, N., Nattestad, A., Falk-Nilsson, E., and Attstrom, R. The interactive examination: assessing students' selfassessment ability. Medical Education 38, 4 (2004), 378-389. (Pubitemid 38482464)
26. Musico, C. There's No Place Like Home. destinationCRM.com, 2008.
27. Orsmond, P., Merry, S., and Reiling, K. A Study in Selfassessment: tutor and students' perceptions of performance criteria. Assessment & Evaluation in Higher Education 22, 4 (1997), 357.
28. Sadler, D.R. Formative assessment and the design of instructional systems. Instructional Science 18, 2 (1989), 119-144.
29. Shute, V.J. Focus on Formative Feedback. Review of Educational Research 78, 1 (2008), 153 -189.
30. Silberman, M.S., Ross, J., Irani, L., and Tomlinson, B. Sellers' problems in human computation markets. Proc ACM SIGKDD Workshop on Human Computation, (2010), 18-21.
31. Taras, M. Using Assessment for Learning and Learning from Assessment. Assessment & Evaluation in Higher Education 27, 6 (2002), 501.
32. Taras, M. To Feedback or Not to Feedback in Student Self-assessment. Assessment & Evaluation in Higher Education 28, 5 (2003), 549.
33. Viégas, F., Wattenberg, M., and Mckeon, M. The Hidden Order of Wikipedia. In Online Communities and Social Computing. 2007, 445-454.
34. Zimmerman, B.J. Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology 11, 4 (1986), 307-313.
35. http://en.wikipedia.org/.
36. http://www.thejohnnycashproject.com/.
37. Crowdflower. Crowdflower.com/.