Worker perception of quality assurance mechanisms in crowdsourcing and human computation markets

19th Americas Conference on Information Systems, AMCIS 2013 - Hyperconnected World: Anything, Anywhere, Anytime

Volumn 5, Issue , 2013, Pages 4046-4056

  Schulze, Thimo ^a   Nordheimer, Dennis ^a   Schader, Martin ^a  

^a UNIVERSITY OF MANNHEIM   (Germany)

Author keywords

Crowdsourcing; Human computation; Labor psychology; Majority vote; Mechanical turk; Quality assurance; Survey

Indexed keywords

CONCEPTUAL MODEL; CROWDSOURCING; GOLD STANDARDS; HIGH QUALITY; HUMAN COMPUTATION; MAJORITY VOTE; MECHANICAL TURKS; QUALIFICATION TEST;

COMMERCE; INFORMATION SYSTEMS; SURVEYING; SURVEYS;

QUALITY ASSURANCE;

EID: 84893325426     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (43)

1. Aquino, K., Tripp, T. M., and Bies, R. J. 2006. "Getting even or moving on? Power, procedural justice, and types of offense as predictors of revenge, forgiveness, reconciliation, and avoidance in organizations," Journal of Applied Psychology (91:3), pp. 653-668.
2. Balkin, D. B., and Griffeth, R. W. 1993. "The determinants of employee benefits satisfaction," Journal of Business and Psychology (7:3), pp. 323-339.
3. Boone, H. N., Jr., and Boone, D. A. 2012. "Analyzing Likert Data," The Journal of Extension (JOE) (50:2).
4. Calhoun, C. 2004. "Subjectivity and emotion," In Thinking about FeelingU: Contemporary Philosophers on Emotions, Oxford University Press, pp. 107-121.
5. Chesbrough, H. 2003. Open Innovation: The New Imperative for Creating and Profiting from Technology, (1st ed), Mcgraw-Hill Professional.
6. Cornell, B., and Welch, I. 1996. "Culture, Information, and Screening Discrimination," Journal of Political Economy (104:3), pp. 542-71.
7. Downs, J. S., Holbrook, M. B., Sheng, S., and Cranor, L. F. 2010. "Are your participants gaming the system?," In Proceedings of the 28th international conference on Human factors in computing systems (CHI'10), Atlanta, Georgia, USA, pp. 2399.
8. Gilliland, S. W. 1993. "The perceived fairness of selection systems: An organizational justice perspective," The Academy of Management Review (18:4), pp. 694-734.
9. Harris, M. M., Dworkin, J. B., and Park, J. 1990. "Preemployment screening procedures: How human resource managers perceive them," Journal of Business and Psychology (4:3), pp. 279-292.
10. Holmstrom, B., and Milgrom, P. 1991. "Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design," Journal of Law, Economics & Organization (7), pp. 24.
11. Howe, J. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, Crown Publishing Group.
12. Iles, P., Mabey, C, and Robertson, I. 1990. "HRM Practices and Employee Commitment: Possibilities, Pitfalls and Paradoxes," British Journal of Management (1:3), pp. 147-157.
13. Ipeirotis, P. 2011. "Does lack of reputation help the crowdsourcing industry? I A Computer Scientist in a Business School," http://www.behind-the-enemy-lines.com/2011/11/does-lack-of-reputation-help.html
14. Ipeirotis, P. G., Provost, F., and Wang, J. 2010. "Quality management on Amazon Mechanical Turk," In Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, ACM, pp. 64-67.
15. Jung, H. J., and Lease, M. 2011. "Improving Consensus Accuracy via Z-Score and Weighted Voting," In The 3rd Human Computation Workshop (HCOMP 2011).
16. Kaufmann, N., Schulze, T., and Veit, D. 2011. "More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk," In AMCIS 2011 Proceedings.
17. Kazai, G. 2010. "An Exploration of the Influence that Task Parameters have on the Performance of Crowds," Presented at the CrowdConf 2010, October 4, 2010, San Francisco, CA.
18. Kern, R., Bauer, C, Thies, H., and Satzger, G. 2010. "Validating results of human-based electronic services leveraging multiple reviewers," AMCIS 2010 Proceedings.
19. Kern, R., Thies, H., and Satzger, G. 2011. "Efficient Quality Management of Human-Based Electronic Services Leveraging Group Decision Making," ECIS 2011 Proceedings.
20. Kern, R., Zirpins, C, and Agarwal, S. 2009. "Managing Quality of Human-Based eServices," In Service-Oriented Computing-ICSOC 2008 Workshops. Lecture Notes in Computer Science, G. Feuerlicht and W. Lamersdorf (eds.), (Vol. 5472), Springer Berlin/Heidelberg, pp. 304-309.
21. Leimeister, J. M., Huber, M., Bretschneider, U., and Krcmar, H. 2009. "Leveraging Crowdsourcing: Activation-Supporting Components for IT-Based Ideas Competition," Journal of Management Information Systems (26:1), pp. 197-224.
22. Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. 2010. "Exploring iterative and parallel human computation processes," In KDD-HCOMP'10, Washington, DC, USA.
23. Lomas, P. 1994. True & False Experience: The Human Element in Psychotherapy, Transaction Publishers.
24. Macan, T. H., Avedon, M. J., Paese, M., and Smith, D. E. 1994. "The effects of applicants' reactions to cognitive ability tests and an assessment center," Personnel Psychology (47:4), pp. 715-738.
25. Oleson, D., Sorokin, A., Laughlin, G., Hester, V., Le, J., and Biewald, L. 2011. "Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing," In The 3rd Human Computation Workshop (HCOMP 2011).
26. Quinn, A. J., and Bederson, B. B. 2011. "Human Computation: A Survey and Taxonomy of a Growing Field," In CHI 2011, May 7-12, 2011, Vancouver, BC, Canada.
27. Riedl, C, Blohm, I., Leimeister, J. M., and Krcmar, H. 2013. "The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities," International Journal of Electronic Commerce (IJEC) (17:3), pp. 7-36.
28. Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., and Tomlinson, B. 2010. "Who are the crowdworkers?: shifting demographics in mechanical turk," In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, Atlanta, Georgia, USA, ACM, pp. 2863-2872.
29. Ryan, A. M., and Ployhart, R. E. 2000. "Applicants' Perceptions of Selection Procedures and Decisions: A Critical Review and Agenda for the Future," Journal of Management (26:3), pp. 565-606.
30. Ryan, A. M., and Sackett, P. R. 1987. "Pre-employment honesty testing: Fakability, reactions of test takers, and company image," Journal of Business and Psychology (1:3), pp. 248-256.
31. Rzeszotarski, J. M., and Kittur, A. 2011. "Instrumenting the crowd: using implicit behavioral measures to predict task performance," In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11), New York, NY, USA:, ACM, pp. 13-22.
32. Sackett, P. R., and Wilk, S. L. 1994. "Within-group norming and other forms of score adjustment in preemployment testing," American Psychologist (49:11), pp. 929-954.
33. Schenk, E., and Guittard, C. 2011. "Towards a characterization of crowdsourcing practices," Journal of Innovation Economics (7:1), pp. 93-107.
34. Schulze, T., Indulska, M., Geiger, D., and Korthaus, A. 2012. "Idea assessment in open innovation: A state of practice," In: 20th European Conference on Information Systems (ECIS). Barcelona, Spain.
35. Schulze, T., Krug, S., and Schader, M. 2012. "Workers' Task Choice in Crowdsourcing and Human Computation Markets," In International Conference on Information Systems (ICIS), Orlando, FL, USA.
36. Schulze, T., Seedorf, S., Geiger, D., Kaufmann, N., and Schader, M. 2011. "Exploring Task Properties in Crowdsourcing-An Empirical Study on Mechanical Turk," In 19th European Conference on Information Systems (ECIS), Helsinki, Finland.
37. Shaw, A. D., Horton, J. J., and Chen, D. L. 2011. "Designing incentives for inexpert human raters," In Proceedings of the ACM 2011 conference on Computer supported cooperative work (CSCW11), New York, NY, USA, ACM, pp. 275284.
38. Sorokin, A., and Forsyth, D. 2008. "Utility data annotation with Amazon Mechanical Turk," In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW08), pp. 1-8.
39. Sun, Y.-A., Roy, S., and Little, G. 2011. "Beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks," In The 3rd Human Computation Workshop (HCOMP 2011).
40. Tabibnia, G., Satpute, A. B., and Lieberman, M. D. 2008. "The Sunny Side of Fairness Preference for Fairness Activates Reward Circuitry (and Disregarding Unfairness Activates Self-Control Circuitry)," Psychological Science (19:4), pp. 339-347.
41. Wais, P., Lingamneni, S., Cook, D., Fennell, J., Goldenberg, B., Lubarov, D., Marin, D., and Simons, H. 2010. "Towards Building a High-Quality Workforce with Mechanical Turk," In Proceedings of Computational Social Science and the Wisdom of Crowds (NIPS), Whister.
42. Weathington, B. L., and Tetrick, L. E. 2000. "Compensation or Right: An Analysis of Employee 'Fringe' Benefit Perception," Employee Responsibilities and Rights Journal (12:3), pp. 141-162.
43. Yampolskiy, R. V. 2012. "Al-Complete, AI-Hard, or AI-Easy: Classification of Problems in Artificial Intelligence," Proceedings of the Twenty-third Midwest Artificial Intelligence and Cognitive Science Conference (MAICS-2012), Cincinnati, Ohio, Omnipress, pp. 94-102.