Quality control mechanisms for crowdsourcing: Peer review, arbitration, & expertiseat familysearch indexing

Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW

Volumn , Issue , 2013, Pages 649-660

  Hansen, Derek L ^a   Schone, Patrick ^b   Corey, Douglas ^a   Reid, Matthew ^a   Gehring, Jake ^b  

^a BRIGHAM YOUNG UNIVERSITY   (United States)

^b FamilySearch   (United States)

Author keywords

Crowdsourcing; Familysearch indexing; Genealogy; Historical documents; Peer review; Quality control; Transcription

Indexed keywords

CONTROL MECHANISM; CROWDSOURCING; DESIGN SUGGESTIONS; FIELD EXPERIMENT; GENEALOGY; HISTORICAL DOCUMENTS; PEER REVIEW; PRIOR EXPERIENCE;

COMPUTER SUPPORTED COOPERATIVE WORK; HISTORY; INTERACTIVE COMPUTER SYSTEMS; QUALITY CONTROL; TRANSCRIPTION;

INDEXING (OF INFORMATION);

EID: 84874863913     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2441776.2441848     Document Type: Conference Paper

References (27)

1. Anderson, J. Cognitive psychology and its implications, 7th edition. Worth Publishers, 2009.
2. Barret, B., Brown, M.S., Manmatha, R., and Gehring, J. Proceedings of the 2011 Workshop on Historical Document Imaging and Processing, ACM Press (2011).
3. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., and Panovich, K. Soylent: a word processor with crowd inside. Proc. UIST '10, ACM Press (2010), 313-322.
4. Chrons, O. and Sundell, S. Digitalkoot: Making old archives accessible using crowdsourcing. Human Computation: Papers from the 2011 AAAI Workshop, (2011).
5. Cosley, D., Frankowski, D., Kiesler, S., Terveen, L., and Riedl, J. 2005. How oversight improves member-maintained communities. In Proc. CHI 2005, ACM Press (2005), 11-20.
6. Demidenko, E. Mixed Models: Theory and Applications. Wiley-Interscience, 2004.
7. Dligach, D., and Palmer, M. 2011. Reducing the need for double annotation. In Proc. of the 5th Linguistic Annotation Workshop (LAW V '11), (2011), 65-73.
8. Doan, A., Ramakrishnan, R., and Halevy, A. Y. Crowdsourcing systems on the World-Wide Web. Commun. ACM 54, 4 (April 2011), 86-96.
9. Duguid, P. 2006. Limits of self-organization: Peer production and "laws of quality". First Monday, 11 (10).
10. Estellés-Arolas, E., and González-Ladrón-de-Guevara, F. Towards an integrated crowdsourcing definition. J. of Information Science, 20, 10 (2012), 1-14.
11. Hansen, D., Jacobs, D., Lewis, D., Biswas, A., Preece, J., Rotman, D., and Stevens, E. Odd Leaf Out: Improving visual recognition with games. In Proc. SocialCom '11, IEEE (2011), 87-94.
12. Haythornthwaite, C. Crowds and Communities: Light and Heavyweight Models of Peer Production. In Proc. HICSS '09, (2009), 1-10.
13. Howe, J. Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business http://www.crowdsourcing.com/.
14. Hsueh, P., Melville, P., and Sindhwani, V. Data quality from crowdsourcing: a study of annotation selection criteria. In Proc. NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing (HLT '09), (2009), 27-35.
15. Kittur, A., and Kraut, R. E. 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proc. CSCW '08, ACM Press (2008), 37-46.
16. Kraut, R.E. and Resnick, P. Building Successful Online Communities: Evidence-Based Social Design. The MIT Press, 2012.
17. Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., Cosley, D., Frakowski, D., Terveen, L., Rashid, A.M., Resnick, P., and Kraut, R. Using social psychology to motivate contributions to online communities. JCMC, 10, 4 (2005).
18. Little, G. and Sun, Y. Human OCR: insights from a complex human computation process. Workshop on Crowdsourcing and Human Computation, Services, Studies and Platforms, ACM CHI, (2011).
19. McCann, R., Doan, A, Varadajan, V., and Kramnik, A. Building Data Integration Systems via Mass Collaboration. In International Workshop on the Web and Data-base, (2003).
20. Nowak, S., and Rüger, S. How reliable are annotations via crowdsourcing: a study about interannotator agreement for multi-label image annotation. In Proc. MIR '10. ACM Press (2010), 557-566.
21. Quinn, A. J. and Bederson, B. B. Human computation: a survey and taxonomy of a growing field. In Proc. CHI 2011, ACM Press (2011), 1403-1412.
22. Stvilia, B., Twidale, M., Smith, L. C., Gasser, L. Information quality work organization in Wikipedia. JASIST, 59(6), (2008), 983-1001.
23. Sun, Y., Roy, S., and Little, G. D. Beyond independent agreement: a tournament selection approach for quality assurance of human computation tasks. Papers from the 2011 AAAI Workshop on Human Computation, (2011).
24. Toselli, A.H., Romero, V., and Vidal, E., Rodriguez, L. Computer Assisted Transcription of Handwritten Text Images. Proc. ICDAR '07. IEEE (2007), 944-948.
25. von Ahn, L. Games with a purpose. Computer, 39(6), (2006), 92-94.
26. von Ahn, L., Maurer, B., McMillen, C., Abraham, D., and Blum, M. reCAPTCHA: Human-based character recognition via web security measures. Science (2008), 1465-1468.
27. Wilkinson, D.M. Strong regularities in online peer production. In Proc. EC '08. ACM Press (2008), 302-309.