Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places

Behavior Research Methods

Volumn 49, Issue 4, 2017, Pages 1333-1342

  Gleibs, Ilka H ^a  

^a LONDON SCHOOL OF ECONOMICS   (United Kingdom)

Author keywords

Crowdsourcing; Ethics; Methodology; MTurk; Sampling

Indexed keywords

ATTENTION; CROWDSOURCING; ETHICS; HUMAN; INFORMATION PROCESSING; INTERNET; MARKET; PREJUDICE; SCIENTIST; PERSONNEL; RESEARCH; RESEARCH ETHICS; SOCIAL MEDIA;

CROWDSOURCING; DATA COLLECTION; ETHICS, RESEARCH; HUMANS; INTERNET; RESEARCH; RESEARCH PERSONNEL; SOCIAL MEDIA;

EID: 84981541164     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-016-0789-y     Document Type: Article

References (88)

1. Aguinis, H., & Lawal, S. O. (2012). Conducting field experiments using eLancing’s natural environment. Journal of Business Venturing, 27, 493–505.
2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
3. Anderson, J. A., & Weijer, C. (2002). The research subject as wage earner. Theoretical Medicine and Bioethics, 23, 359–376.
4. Baldwin, M., Biernat, M., & Landau, M. J. (2015). Remembering the real me: Nostalgia offers a window to the intrinsic self. Journal of Personality and Social Psychology, 108, 128–147. doi:10.1037/a0038033
5. Barchard, K. A., & Williams, J. (2008). Practical advice for conducting ethical online experiments and questionnaires for United States psychologists. Behavior Research Methods, 40, 1111–1128. doi:10.3758/BRM.40.4.1111
6. Baumeister, R., & Bushman, B. (2011). Social psychology and human nature. Belmont: Wadsworth, Cengage Learning.
7. Bederson, B. B., & Quinn, A. J. (2011). Web workers unite! addressing challenges of online laborers. Paper presented at the CHI’11 Conference on Human Factors in Computing Systems, Vancouver, BC.
8. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813. doi:10.3758/s13428-011-0081-0
9. Belk, R. (2014a). Sharing versus pseudo-sharing in Web 2.0. The Anthropologist, 18, 7–23.
10. Belk, R. (2014b). You are what you can access: Sharing and collaborative consumption online. Journal of Business Research, 67, 1595–1600.
11. Benson, A., Sojourner, A. J., & Umyarov, A. (2015). The value of employer reputation in the absence of contract enforcement: A randomized experiment. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2557605
12. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368.
13. Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D., Marlow, C., Settle, J. E., & Fowler, J. H. (2012). A 61-million-person experiment in social influence and political mobilization. Nature, 489, 295–298.
14. Brawley, A. M., & Pury, C. L. (2016). Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54, 531–546.
15. Bruder, M., Göritz, A. S., Reips, U.-D., & Gebhard, R. K. (2015). Ein national gefördertes Onlinelabor als Infrastruktur für die psychologische Forschung. Psychologische Rundschau, 65, 75–85. doi:10.1026/0033-3042/a000198
16. Buchanan, E., & Zimmer, M. (2012). Internet research ethics. In Stanford Encyclopaedia of Philosophy, Retrieved from http://plato.stanford.edu/entries/ethics-internet-research
17. Buchanan, T., & Williams, J. E. (2010). Ethical issues in psychological research on the Internet. In S. D. Gosling & J. A. Johnson (Eds.), Advanced methods for conducting online behavioral research (pp. 255–271). Washington, DC: American Psychological Association.
18. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980
19. Busarovs, A. (2013). Ethical aspects of crowdsourcing, or is it a modern form of exploitation. International Journal of Economics & Business Administration, 1, 3–14.
20. Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? a comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160.
21. Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53–81. doi:10.1146/annurev-clinpsy-021815-093623
22. Corti, K., & Gillespie, A. (2015). Revisiting Milgram’s cyranoid method: Experimenting with hybrid human agents. Journal of Social Psychology, 155, 30–56. doi:10.1080/00224545.2014.959885
23. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8, e57410. doi:10.1371/journal.pone.0057410
24. Dandurand, F., Shultz, T. R., & Onishi, K. H. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior Research Methods, 40, 428–434. doi:10.3758/BRM.40.2.428
25. DeSoto, K. A. (2016). Under the hood of Mechanical Turk. Observer, 29(3).
26. Dickert, N., & Grady, C. (1999). What’s the price of a research subject? approaches to payment for research participation. New England Journal of Medicine, 341, 198–203.
27. Dholakia, U. (2015). My experiences as an Amazon Mechanical Turk (MTurk) worker. Blogpost retrieved from https://www.linkedin.com/pulse/my-experience-amazon-mechanical-turk-mturk-worker-utpal-dholakia
28. Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are your participants gaming the system? Screening Mechanical Turk workers. Paper presented at the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, GA.
29. Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 621. doi:10.3389/fpsyg.2015.00621
30. Eckhardt, G. M., & Bardhi, F. (2015). The sharing economy isn’t about sharing at all. Harvard Business Review. Retrieved from https://hbr.org/2015/01/the-sharing-economy-isnt-about-sharing-at-all
31. Felstiner, A. (2011). Working the crowd: Employment and labor law in the crowdsourcing industry. Berkeley Journal of Employment and Labor Law, 32, 143–203.
32. Fiske, S. T., & Hauser, R. M. (2014). Protecting human research participants in the age of big data. Proceedings of the National Academy of Sciences, 111, 13675–13676. doi:10.1073/pnas.1414626111
33. Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37, 413–420.
34. Gleibs, I. H. (2014). Turning virtual public spaces into laboratories: Thoughts on conducting online field studies using social network sites. Analyses of Social Issues and Public Policy, 14, 352–370.
35. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224.
36. Gosling, S. D., & Bonnenburg, A. V. (1998). An integrative approach to personality research in anthrozoology: Ratings of six species of pets and their owners. Anthrozoös, 11, 148–156.
37. Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902. doi:10.1146/annurev-psych-010814-015321
38. Gosling, S. D., Sandy, C. J., John, O. P., & Potter, J. (2010). Wired but not WEIRD: The promise of the Internet in reaching more diverse samples. Behavioral and Brain Sciences, 33, 94–95. doi:10.1017/S0140525X10000300
39. Grant, R. W., & Sugarman, J. (2004). Ethics in human subjects research: Do incentives matter? Journal of Medicine and Philosophy, 29, 717–738.
40. Greenaway, K. H., Haslam, S. A., Cruwys, T., Branscombe, N. R., Ysseldyk, R., & Heldreth, C. (2015). From “we” to “me”: Group identification enhances perceived personal control with consequences for health and well-being. Journal of Personality and Social Psychology, 109, 53–74. doi:10.1037/pspi0000019
41. Hauser, D. J., & Schwarz, N. (2015). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400–407. doi:10.3758/s13428-015-0578-z
42. Heintzelman, S. J., Trent, J., & King, L. A. (2013). Encounters with objective coherence and the experience of meaning in life. Psychological Science, 24, 991–998. doi:10.1177/0956797612465878
43. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world?. Behavioral and Brain Sciences, 33, 61–83, disc. 83–135. doi:10.1017/S0140525X0999152X
44. Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425.
45. Hui, B. P. H., Chen, S. X., Leung, C. M., & Berry, J. W. (2015). Facilitating adaptation and intercultural contact: The role of integration and multicultural ideology in dominant and non-dominant groups. International Journal of Intercultural Relations, 45, 70–84. doi:10.1016/j.ijintrel.2015.01.002
46. Ipeirotis, P. G. (2010). Analyzing the Amazon Mechanical Turk marketplace. XRDS, 17, 16–21. doi:10.1145/1869086.1869094
47. Irani, L. C., & Silberman, M. (2013). Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk. Paper presented at the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.
48. Jiang, L., Wagner, C., & Nardi, B. (2015). Not just in it for the money: A qualitative investigation of workers’ perceived benefits of micro-task crowdsourcing. Paper presented at the 48th Hawaii International Conference on System Sciences (HICSS), Koloa, Hawaii.
49. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
50. Kahn, J. P., Vayena, E., & Mastroianni, A. C. (2014). Learning as we go: Lessons from the publication of Facebook’s social-computing research. Proceedings of the National Academy of Sciences, 111, 13677–13679. doi:10.1073/pnas.1416405111
51. Kingsley, S. C., Gray, M. L., & Suri, S. (2015). Accounting for market frictions and power asymmetries in online labor markets. Policy and Internet, 7, 383–400.
52. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs’ Advisory Group on the Conduct of Research on the Internet. American Psychologist, 59, 105–117. doi:10.1037/0003-066X.59.2.105
53. Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8, 142–164.
54. Litman, L., Robinson, J., & Abberbock, T. (2016). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-016-0727-z
55. Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47, 519–528. doi:10.3758/s13428-014-0483-x
56. Mader J., & Fritz, M. (2015). The Internet's hidden science factory. PBS Newshour. Retrieved from http://www.pbs.org/newshour/updates/inside-amazons-hidden-science-factory/. Accessed 1 Aug 2016
57. Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014). Being a Turker. Paper presented at the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing, Baltimore, MD.
58. Mayer-Schönberger, V., & Cukier, K. (2013). Big data: A revolution that will transform how we live, work, and think. Boston: Houghton Mifflin Harcourt.
59. Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135, 151–165. doi:10.3200/GENP.135.2.151-166
60. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188. doi:10.1177/0963721414531598
61. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
62. Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y
63. Postmes, T., Spears, R., & Lea, M. (1998). Breaching or building social boundaries? SIDE-effects of computer-mediated communication. Communication Research, 25, 689–715.
64. Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. doi:10.1016/j.jtbi.2011.03.004
65. Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014). Social heuristics shape intuitive cooperation. Nature Communications, 5, 3677. doi:10.1038/ncomms4677
66. Ripley, E. B. D. (2006). A review of paying research participants: It’s time to move beyond the ethical debate. Journal of Empirical Research on Human Research Ethics, 1(4), 9–20. doi:10.1525/jer.2006.1.4.9
67. Rosnow, R. L. (1997). Hedgehogs, foxes, and the evolving social contract in psychological science: Ethical challenges and methodological opportunities. Psychological Methods, 2, 345–356. doi:10.1037/1082-989X.2.4.345
68. Ross, J., Irani, L., Silberman, M., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in Mechanical Turk. Paper presented at the CHI’10 Conference on Human Factors in Computing Systems, Atlanta, GA.
69. Roulin, N. (2015). Don’t throw the baby out with the bathwater: Comparing data quality of crowdsourcing, online panels, and student samples. Industrial and Organizational Psychology, 8, 190–196.
70. Sabou, M., Bontcheva, K., & Scharl, A. (2012). Crowdsourcing research opportunities: Lessons from natural language processing. Paper presented at the 12th International Conference on Knowledge Management and Knowledge Technologies, Graz, Austria.
71. Salehi, N., Irani, L. C., Bernstein, M. S., Alkhatib, A., Ogbe, E., Milland, K., & Clickhappier. (2015). We are dynamo: Overcoming stalling and friction in collective action for crowd workers. Paper presented at the Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems.
72. Sassenberg, K. (2002). Common bond and common identity groups on the Internet: Attachment and normative behavior in on-topic and off-topic chats. Group Dynamics: Theory, Research, and Practice, 6, 27–37. doi:10.1037/1089-2699.6.1.27
73. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51, 515–530. doi:10.1037/0022-3514.51.3.515
74. Selznick, P. (2000). On sustaining research agendas: Their moral and scientific basis. An address to the Western Academy of Management. Journal of Management Inquiry, 9, 277–282. doi:10.1108/S0733-558X20150000044002
75. Shank, D. B. (2016). Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. American Sociologist, 47, 47–55. doi:10.1007/s12108-015-9266-9
76. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220. doi:10.1177/2167702612469015
77. Shapiro, R. B., & Ossorio, P. N. (2013). Regulation of online social network studies. Science, 339, 144–145.
78. Silberman, M. S., & Irani, L. (2016). Operating an employer reputation system: Lessons from Turkopticon, 2008–2015. Comparative Labor Law and Policy Journal, 37(3). https://www.law.illinois.edu/publications/cllpj/
79. Silberman, M., Irani, L., & Ross, J. (2010). Ethics and tactics of professional crowdwork. XRDS: Crossroads, 17, 39–43.
80. Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479–491.
81. Suri, S., & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE, 6, e16836. doi:10.1371/journal.pone.0016836
82. Trinidad, S. B., Fullerton, S. M., Ludman, E. J., Jarvik, G. P., Larson, E. B., & Burke, W. (2011). Research practice and participant preferences: The growing gulf. Science, 331, 287–288. doi:10.1126/science.1199000
83. van Steenbergen, H., & Bocanegra, B. R. (2015). Promises and pitfalls of Web-based experimentation in the advance of replicable psychological science: A reply to Plant (2015). Behavior Research Methods. Advance online publication. doi:10.3758/s13428-015-0677-x
84. Vayena, E., Salathé, M., Madoff, L. C., & Brownstein, J. S. (2015). Ethical challenges of big data in public health. PLoS Computational Biology, 11, e1003904. doi:10.1371/journal.pcbi.1003904
85. Weinberg, J. D., Freese, J., & McElhattan, D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310.
86. Wolfson, S. M., & Lease, M. (2011). Look before you leap: Legal pitfalls of crowdsourcing. Proceedings of the American Society for Information Science and Technology, 48(1), 1–10.
87. Wright, T. A., & Wright, V. P. (1999). Ethical responsibility and the organizational researcher: A committed-to-participant research perspective. Journal of Organizational Behavior, 20, 1107–1112.
88. Wright, T. A., & Wright, V. P. (2002). Organizational researcher values, ethical responsibility, and the committed-to-participant research perspective. Journal of Management Inquiry, 11, 173–185.