Reviewing versus doing: Learning and performance in crowd assessment

Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW

Volumn , Issue , 2014, Pages 1445-1455

  Zhu, Haiyi ^a   Dow, Steven P ^a   Kraut, Robert E ^a   Kittur, Aniket ^a  

^a Carnegie Mellon University   (United States)

Author keywords

Assessment; Crowdsourcing; Learning; Review

Indexed keywords

COMMERCE; INTERACTIVE COMPUTER SYSTEMS; REVIEWS;

ASSESSMENT; CROWDSOURCING; LEARNING; PEER REVIEWING; QUALITY ASSESSMENT; QUALITY MEASURES; TASK DOMAIN;

COMPUTER SUPPORTED COOPERATIVE WORK;

EID: 84898939077     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2531602.2531718     Document Type: Conference Paper

References (39)

1. Ariely D, Au WT, Bender RH, Budescu DV, Dietz CB, et al. 2000. The effects of averaging subjective probability estimates between and within judges. J. Exp. Psychol.: Applied 6: 130-47.
2. Barry, D.(1991) Managing the bossless team: Lessons in distributed leadership. Organizational Dynamics, 20, 31-47.
3. Brown, A., Bransford, J., Ferrara, R.,&Campione, J.(1983). Learning, remembering, and understanding. In J.H. Flavell &E.M. Markman (Eds.), Handbook of child psychology: Vol.3. Cognitive development (4th ed., pp.77-166). New York: Wiley.
4. Chase C.C., Chin, D.B., Oppezzo, M.A., and Schwartz, D.L. (2009) Teachable Agents and the Protégé Effect: Increasing the Effort Towards Learning. Journal of Science Education and Technology. Volume 18, Number 4 (2009), 334-352.
5. Dow, S.P., Kulkarni, A., Klemmer, S.R., and Hartmann, B.(2012) Shepherding the Crowd Yields Better Work. ACM Conference on Computer Supported Cooperative Work, 2012.
6. Felps, W., Mitchell, T., and Byington, E. How, When, and Why Bad Apples Spoil the Barrel: Negative Group Members and Dysfunctional Groups. Research in Organizational Behavior 27, (2006), 175-222.
7. Guilford, J. P. (1967). The nature of human intelligence. New York, NY: McGraw Hill.
8. Horton, J. J. (2010). Employer expectations, peer effects and productivity: Evidence from a series of field experiments. arXiv preprint arXiv: 1008.2437.
9. Isenberg, D. J. (1986). Group polarization: A critical review and meta-analysis. Journal of Personality and Social Psychology, 50(6), 1141.
10. Janis, I.L. Groupthink: Psychological Studies of Policy Decisions and Fiascoes. Wadsworth Publishing, 1982.
11. Kerr, N. L., Tindale, R.S. (2004) Group performance and decision making. Annu. Rev. Psychol. 2004. 55: 623-55.
12. Kittur, A., Chi, E.H., and Suh, B. (2008) Crowdsourcing user studies with Mechanical Turk. In Proceedings of the twentysixth annual SIGCHI conference on Human factors in computing systems (CHI '08). ACM, New York, NY, USA, 453-456.
13. Kittur, A. and Kraut, R.E. 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (CSCW '08)
14. Kittur, A., Smus, B., Khamkar, S., and Kraut, R.E. (2011) CrowdForge: crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 43-52.
15. Kraut, R. E. &Resnick, P. (2012). Building successful online communities: Evidence-based social design. Cambridge, MA: MIT Press.
16. Kulkarni, A., Gutheim, P., Narula, P., Rolnitzky, D., Parikh, T., &Hartmann, B. (2012). MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture. IEEE Internet Computing, 16(5), 28-35.
17. Laughlin PR, McGlynn RP. (1986). Collective induction: mutual group and individual influence by exchange of hypotheses and evidence. J. Exp. Soc. Psychol. 22: 567-589.
18. Levine, J.M., Moreland, R.L. (1990) Progress in small group research. Annual Review of Psychology, Vol 41, 1990, 585-634.
19. Mannix, E., and Neale, M.A. (2005) What Differences Make a Difference? The Promise and Reality of Diverse Teams in Organizations. Psychological Science in the Public Interest , Vol. 6, No. 2 (Oct., 2005), pp. 31-55.
20. Matsuda, N., Cohen, W. W., Koedinger, K. R., Keiser, V., Raizada, R., Yarzebinski, E., et al. (2012). Studying the Effect of Tutor Learning using a Teachable Agent that asks the Student Tutor for Explanations. In Digital Game and Intelligent Toy Enhanced Learning (DIGITEL), 2012 IEEE Fourth International Conference on (pp. 25-32). IEEE.
21. Manz, C.C., &Sims, H.P., Jr(1993) Business without bosses. New York: Wiley.
22. McGrath, J.E. (1984) Groups: interaction and performance. Prentice Hall, Inc., Englewood Cliffs. N.J.
23. Mullen, B., Johnson C., and Salas E.(1991) "Productivity loss in brainstorming groups: A meta-analytic integration." Basic and Applied Psychology, 12: 2-23.
24. Paolacci, G., Chandler, J., Ipeirotis, P.G. (2010) Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, Vol 5(5), Aug 2010, 411-419.
25. Paulus, P.B., Dzindolet, M.T., Poletes, G., and Camacho, L.M. (1993) "Perceptions of performance in group brainstorming: The illusion of group productivity." Personality and Social Psychology Bulletin, 19: 78-89.
26. Pearce, C.L., and Conger,J. A. (2003) Shared leadership: Reframing the hows and whys of leadership, Sage, Thousand Oaks.
27. Ryan, R. M.; Deci, E L. (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, Vol 55(1), Jan 2000, 68-78.
28. Schkade, D., Sunstein, C. R., &Kahneman, D. (2000). Deliberating about dollars: The severity shift. Columbia Law Review, 1139-1175.
29. Srivastava A., Bartol, K.M., and Locke, E.A.(2006) Empowering Leadership in Management Teams: Effects on Knowledge Sharing, Efficacy, and Performance. The Academy of Management Journal , Vol. 49, No. 6 (Dec., 2006), pp. 1239-1251.
30. Stroebe, W. and Diehl, M. Why Groups are less Effective than their Members: On Productivity Losses in Idea-generating Groups. European Review of Social Psychology 5, (1994), 271.
31. Sutton, R. and Hargadon, A. Brainstorming groups in context: effectiveness in a product design firm. Administrative Science Quarterly, (1996).
32. Van De Ven, A. H., and Delbecq A.L. (1971) Nominal versus Interacting Group Processes for Committee Decision-Making Effectiveness. The Academy of Management Journal, Vol. 14, No. 2 (Jun., 1971), pp. 203-212.
33. Van De Ven, A. H., and Delbecq A.L. (1974) The Effectiveness of Nominal, Delphi, and Interacting Group Decision Making Processes. The Academy of Management Journal, Vol. 17, No. 4 (Dec., 1974), pp. 605-621.
34. White, B.Y. and Frederiksen, J.R. (1998) Inquiry, Modeling, and Metacognition: Making Science Accessible to All Students. Cognition and Instruction, Vol. 16, No. 1 (1998), pp. 3-118.
35. Woolley, A.W., Chabris, C.F., Pentland, A., Hashmi, N., Malone, T.W. (2010) Evidence for a Collective Intelligence Factor in the Performance of Human Groups. Science 29 October 2010: Vol. 330 no. 6004 pp. 686-688
36. Yukl, G. A. (1998). Leadership in organizations (4th ed.). Upper Saddle River, N.J.: Prentice Hall.
37. Zhu, H., Kraut, R.E., &Kittur, A., (2012) Effectiveness of Shared Leadership in Online Communities. CSCW'2012.
38. Zhu, H., Kraut, R., &Kittur, A. (2012, February). Organizing without formal organization: group identification, goal setting and social modeling in directing online production. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (pp. 935-944). ACM.
39. Zhu, H., Kraut, R.E., Wang, Y.C., &Kittur, A. (2011) Identifying Shared Leadership in Wikipedia. In CHI'2011: Proceedings of the 2011 annual conference on Human factors in computing systems. New York: ACM Press