Crowdsourcing Samples in Cognitive Science

Trends in Cognitive Sciences

Volumn 21, Issue 10, 2017, Pages 736-748

  Stewart, Neil ^a   Chandler, Jesse ^b,c   Paolacci, Gabriele ^d  

^a UNIVERSITY OF WARWICK   (United Kingdom)

^b MATHEMATICA POLICY RESEARCH   (United States)

^c UNIVERSITY OF MICHIGAN   (United States)

^d ERASMUS UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

STUDENTS;

AMAZON MECHANICAL TURKS; COGNITIVE SCIENCE; COLLEGE STUDENTS; METHODOLOGICAL INNOVATIONS; ON-LINE EXPERIMENTS; REPRODUCIBILITIES; TECHNICAL ASPECTS; TRAGEDY OF THE COMMONS;

CROWDSOURCING;

COLLEGE STUDENT; CROWDSOURCING; FEMALE; HUMAN; HUMAN EXPERIMENT; HUMAN TISSUE; INFORMATION PROCESSING; MALE; MARKET; PSYCHOLOGY; REACTION TIME; REPRODUCIBILITY; SAMPLE SIZE; SCIENTIST; VIDEORECORDING; PROCEDURES;

COGNITIVE SCIENCE; CROWDSOURCING; DATA COLLECTION; HUMANS; REPRODUCIBILITY OF RESULTS;

EID: 85028358998     PISSN: 13646613     EISSN: 1879307X     Source Type: Journal    
DOI: 10.1016/j.tics.2017.06.007     Document Type: Review

References (104)

1. Gosling, S.D., Mason, W., Internet research in psychology. Annu. Rev. Psychol. 66 (2015), 877–902, 10.1146/ annurev-psych-010814-015321.
2. Berinsky, A.J., et al. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Polit. Anal. 20 (2012), 351–368, 10.1093/pan/mpr057.
3. Mullinix, K.J., et al. The generalizability of survey experiments. J. Exp. Polit. Psychol. 2 (2016), 109–138, 10.1017/XPS.2015.19.
4. Kittur, A., et al. Crowdsourcing user studies with Mechanical Turk. Czerwinski, M., (eds.) Proceedings of the SIGCHI Conference on Human factors in Computing systems, 2008, ACM, 453–456.
5. Paolacci, G., et al. Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5 (2010), 411–419 http://journal.sjdm.org/10/10630a/jdm10630a.pdf.
6. Buhrmester, M., et al. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives On Psychol. Sci. 6 (2011), 3–5, 10.1177/1745691610393980.
7. Horton, J.J., et al. The online laboratory: Conducting experiments in a real labor market. Exp. Econ. 14 (2011), 399–425, 10.1007/s10683-011-9273-9.
8. Shank, D.B., Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. Am. Sociol. 47 (2016), 47–55, 10.1007/s12108-015-9266-9.
9. Shapiro, D.N., et al. Using Mechanical Turk to study clinical populations. Clinical Psychol. Sci. 1 (2013), 213–220, 10.1177/2167702612469015.
10. Goodman, J.K., Paolacci, G., Crowdsourcing consumer research. J. Consum. Res 44 (2017), 196–210, 10.1093/jcr/ucx047.
11. Bentley, J.W., Challenges with Amazon Mechanical Turk research in accounting. SSRN eLibrary., 2017 Published online February 28, 2017. https://ssrn.com/abstract=2924876.
12. Stritch, J.M., et al. The opportunities and limitations of using Mechanical Turk (Mturk) in public administration and management scholarship. Int. Public. Manag. J., 2017, 10.1080/10967494.2016.1276493 Published online January 19, 2017.
13. Lutz, J., The validity of crowdsourcing data in studying anger and aggressive behavior a comparison of online and laboratory data. Soc. Psychol. 47 (2016), 38–51, 10.1027/1864-9335/a000256.
14. Majima, Y., et al. Conducting online behavioral research using crowdsourcing services in Japan. Front. Psychol., 8, 2017, 378, 10.3389/fpsyg.2017.00378.
15. Peer, E., et al. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav. Res. Methods 46 (2014), 1023–1031, 10.3758/s13428-013-0434-y.
16. Crone, D.L., Williams, L.A., Crowdsourcing participants for psychological research in Australia: A test of microworkers. Aust. J. Psychol 69 (2016), 39–47.
17. Peer, E., et al. Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Soc. Psychol. 70 (2017), 153–163, 10.1016/j.jesp. 2017.01.006.
18. Estellés-Arolas, E., González-Ladrzón-De-Guevara, F., Towards an integrated crowdsourcing definition. J. Inf. Sci. 38 (2012), 189–200, 10.1177/0165551512437638.
19. Sulser, F., et al. Crowd-based semantic event detection and video annotation for sports videos. Redi, J., Lux, M., (eds.) Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia, 2014, ACM, New York, 63–68, 10.1145/ 2660114.2660119.
20. Casler, K., et al. Separate but equal?. A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Comput. Hum. Behav. 29 (2013), 2156–2160.
21. Casey, L., et al. Intertemporal differences among MTurk worker demographics. SAGE Open, 2017, 10.1177/2158244017712774 Published online June 14, 2017. https://osf.io/preprints/psyarxiv/8352x.
22. Levay, K.E., et al. The demographic and political composition of Mechanical Turk samples. Sage Open, 2016, 10.1177/2158244016636433 Published online March 15, 2016.
23. Behrend, T.S., et al. The viability of crowdsourcing for survey research. Behav. Res. Methods 43 (2011), 800–813, 10.3758/s13428-011-0081-0.
24. Arditte, K.A., et al. The importance of assessing clinical phenomena in Mechanical Turk research. Psychol. Assessment, 28, 2016, 684, 10.1037/pas0000217.
25. Goodman, J.K., et al. Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. J. Behav. Decis. Making. 26 (2013), 213–224, 10.1002/bdm.1753.
26. Kosara, R., Ziemkiewicz, C., et al. Do Mechanical Turks dream of square pie charts?. Sedlmair, M., (eds.) Proceedings of the 3rd BELIV’10 Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualisation, 2010, ACM, New York, 63–70, 10.1145/2110192.
27. Johnson, D.R., Borden, L.A., Participants at your fingertips: Using Amazon's Mechanical Turk to increase student-faculty collaborative research. Teach. Psychol. 39 (2012), 245–251, 10.1177/ 0098628312456615.
28. Veilleux, J.C., et al. Negative affect intensity influences drinking to cope through facets of emotion dysregulation. Pers. Indiv. Differ. 59 (2014), 96–101, 10.1016/j.paid.2013.11.012.
29. Chandler, J., Shapiro, D., Conducting clinical research using crowdsourced convenience samples. Annu. Rev. Clin. Psycho. 12 (2016), 53–81, 10.1146/annurev-clinpsy-021815-093623.
30. Arechar, A.A., et al. Turking overtime: How participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. J. Econ. Sci. Assoc. 3 (2016), 1–11, 10.2139/ssrn.2836946 https://ssrn.com/abstract=2836946.
31. Wang, X., et al. A community rather than a union: Understanding self-organization phenomenon on Mturk and how it impacts Turkers and requesters. Association for Computing Machinery CHI’17 Conference, 2017, ACM, New York, 2210–2216, 10.1145/ 3027063.3053150.
32. Stewart, N., et al. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgm. Decis. Mak. 10 (2015), 479–491 http://journal.sjdm.org/14/14725/jdm14725.pdf.
33. Chandler, J., et al. Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behav. Res. Methods 46 (2014), 112–130, 10.3758/ s13428-013-0365-7.
34. Henrich, J., et al. Most people are not WEIRD. Nature, 466, 2010, 10.1038/466029a 29–29.
35. de Leeuw, J.R., Motz, B.A., Psychophysics in a web browser? Comparing response times collected with javascript and psychophysics toolbox in a visual search task. Behav. Res. Methods 48 (2016), 1–12, 10.3758/s13428-015-0567-2.
36. Crump, M.J., et al. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLoS One, 8, 2013, e57410, 10.1371/journal.pone.0057410.
37. Hilbig, B.E., Reaction time effects in lab- versus web-based research: Experimental evidence. Behav. Res. Methods 48 (2016), 1718–1724, 10.3758/s13428-015-0678-9.
38. Simcox, T., Fiez, J.A., Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behav. Res. Methods 46 (2014), 95–111, 10.3758/s13428-013-0345-y.
39. Klein, R.A., et al. Investigating variation in replicability: A ‘many labs’ replication project. Soc. Psychol. 45 (2014), 142–152, 10.1027/1864-9335/a000178.
40. Zwaan, R.A., et al. Participant nonnaiveté and the reproducibility of cognitive psychology. Psychon. Bull. Rev., 2017 https://osf.io/preprints/psyarxiv/rbz29.
41. Clifford, S., et al. Are samples drawn from Mechanical Turk valid for research on political ideology?. Res. Polit., 2, 2015, 10.1177/2053168015622072 Published online December 15, 2015.
42. Munafo, M.R., et al. A manifesto for reproducible science. Nat. Hum. Behav., 1, 2017, 0021, 10.1038/s41562-016-0021.
43. Rosenthal, R., The file drawer problem and tolerance for null results. Psychol. Bull. 86 (1979), 638–641, 10.1037//0033-2909.86.3.638.
44. Simmons, J.P., et al. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22 (2011), 1359–1366, 10.1177/ 0956797611417632.
45. Frick, R.W., A better stopping rule for conventional statistical tests. Behav. Res. Methods, Instruments, & Computers 30 (1998), 690–697.
46. Kruschke, J.K., Doing Bayesian Data Analysis: A Tutorial with R and BUGS. 2011, Academic Press, Burlington, MA.
47. Simonsohn, U., Posterior-hacking: Selective reporting invalidates Bayesian results also. SSRN eLibrary., 2014, 10.2139/ssrn.2374040 Published online January 3, 2014. https://ssrn.com/abstract=2374040.
48. Cohen, J., Statistical Power Analysis for the Behavioral Sciences. 2nd edn, 1988, Erlbaum, Hillsdale, NJ.
49. Button, K.S., et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 (2013), 365–376, 10.1038/nrn3475.
50. Open Science Collaboration, Estimating the reproducibility of psychological science. Science, 349, 2015, aac4716, 10.1126/science.aac4716.
51. Cumming, G., The new statistics: Why and how. Psychol. Sci. 25 (2014), 7–29, 10.1177/0956797613504966.
52. Simonsohn, U., Small telescopes: Detectability and the evaluation of replication results. Psychol. Sci. 26 (2015), 559–569, 10.1177/0956797614567341.
53. Open Science Collaboration, An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. On Psychol. Sci. 7 (2012), 657–660, 10.1177/1745691612462588.
54. Schwarz, N., Strack, F., Does merely going through the same moves make for a ‘direct’ replication? Concepts, contexts, and operationalizations. Soc. Psychol. 45 (2014), 305–306.
55. Stroebe, W., Strack, F., The alleged crisis and the illusion of exact replication. Perspect. On Psychol. Sci. 9 (2014), 59–71, 10.1177/1745691613514450.
56. Mor, S., et al. Identifying and training adaptive cross-cultural management skills: The crucial role of cultural metacognition. Acad. Manag. Learn. Edu. 12 (2013), 139–161, 10.5465/amle.2012.0202.
57. Lease, M., et al. Mechanical Turk is not anonymous. SSRN eLibrary., 2013, 10.2139/ssrn.2228728 Published online March 9, 2013. https://ssrn.com/abstract=2836946.
58. Fort, K., et al. Amazon Mechanical Turk: Gold mine or coal mine?. Comput. Linguist. 37 (2011), 413–420, 10.1162/COLI\ _a\_00057.
59. Mason, W., Watts, D.J., Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter 11 (2010), 100–108.
60. Litman, L., et al. The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behav. Res. Methods 47 (2015), 519–528, 10.3758/s13428-014-0483-x.
61. Aker, A., et al. Assessing crowdsourcing quality through objective tasks. Calzolari, N., (eds.) Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC’12), 2012, European Language Resources Association, 1456–1461.
62. Ho, C-J. et al. (15) Incentivizing high quality crowdwork. In Proceedings of the 24th International Conference on World Wide Web, pp. 419–429, International World Wide Web Conferences Steering Committee. http://dx.doi.org/10.1145/2736277.2741102.
63. Kees, J., et al. An analysis of data quality: Professional panels, student subject pools, and Amazon's Mechanical Turk. J. Advertising 46 (2017), 141–155, 10.1080/00913367.2016. 1269304.
64. Berg, J., Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comp. Labor Law & Pol. J., 37, 2016.
65. Yin, M., et al. The communication network within the crowd. Proceedings of the 25th International Conference on World Wide Web, 2016, International World Wide Web Conferences Steering Committee, 1293–1303, 10.1145/2872427.2883036.
66. Frederick, S., Cognitive reflection and decision making. J. Econ. Perspect. 19 (2005), 25–42, 10.1257/089533005775196732.
67. Thompson, K.S., Oppenheimer, D.M., Investigating an alternate form of the cognitive reflection test. Judgm. Decis. Mak. 11 (2016), 99–113 http://journal.sjdm.org/15/151029/jdm151029.pdf.
68. Finucane, M.L., Gullion, C.M., Developing a tool for measuring the decision-making competence of older adults. Psychol. Aging 25 (2010), 271–288, 10.1037/a0019106.
69. Rand, D.G., et al. Social heuristics shape intuitive cooperation. Nat. Commun., 5, 2014, e3677, 10.1038/ ncomms4677.
70. Mason, W., et al. Long-run learning in games of cooperation. Proceedings of the Fifteenth ACM Conference on Economics and Computation, 2014, ACM, New York, 821–838, 10.1145/ 2600057.2602892.
71. Chandler, J., et al. Using non-naïve participants can reduce effect sizes. Psychol. Sci. 26 (2015), 1131–1139, 10.1177/0956797615585115.
72. DeVoe, S.E., House, J., Replications with MTurkers who are naïve versus experienced with academic studies: A comment on Connors, Khamitov, Moroz, Campbell, and Henderson (2015). J. Exp. Soc. Psychol. 67 (2016), 65–67, 10.1016/j.jesp.2015.11.004.
73. Hauser, D.J., Schwarz, N., Attentive turkers: Mturk participants perform better on online attention checks than subject pool participants. Behav. Res. Methods 48 (2016), 400–407, 10.3758/s13428-015-0578-z.
74. Chandler, J., Paolacci, G., Lie for a dime: When most prescreening responses are honest but most study participants are imposters. Soc. Psychol. Person. Sci., 2017, 10.1177/1948550617698203 Published online April 27, 2017.
75. Hertwig, R., Ortmann, A., Experimental practices in economics: A methodological challenge for psychologists?. Behav. Brain. Sci. 24 (2001), 383–451, 10.1037/e683322011-032.
76. Krupnikov, Y., Levin, A.S., Cross-sample comparisons and external validity. J. Exp. Polit. Psychol. 1 (2014), 59–80, 10.1017/xps.2014.7.
77. Litman, L., et al. TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav. Res. Methods 49 (2016), 433–442, 10.3758/s13428-016-0727-z.
78. Scott, K., Schulz, L., Lookit (part 1): A new online platform for developmental research. Open Mind 1 (2017), 4–14, 10.1162/opmi_ a_00002.
79. Tran, M., et al. Online recruitment and testing of infants with Mechanical Turk. J. Exp. Child Psychol. 156 (2017), 168–178, 10.1016/j.jecp.2016.12.003.
80. Arechar, A., et al. Conducting interactive experiments online. Exp. Econ., 2017, 10.1007/s10683-017-9527-2 Published online May 9, 2017.
81. Balietti, S., nodeGame: Real-time, synchronous, online experiments in the browser. Behav. Res. Methods., 2016, 10.3758/s13428-016-0824-z Published online November 18, 2016.
82. Yu, L., Nickerson, J.V., Cooks or cobblers? Crowd creativity through combination. Proceedings of the SIGCHI conference on human factors in computing systems, 2011, ACM, 1393–1402.
83. Kim, J. et al. (2016) Mechanical novel: Crowdsourcing complex work through reflection and revision. Comput. Res. Repository. http://dx.doi.org/10.1145/2998181.2998196.
84. Morris, R.R., Picard, R., Crowd-powered positive psychological interventions. J. Positive Psychol. 9 (2014), 509–516, 10.1080/ 17439760.2014.913671.
85. Bigham, J.P., et al. VizWiz: Nearly real-time answers to visual questions. Perlin, K., (eds.) Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, 2010, ACM, New York, 333–342.
86. Meier, A., et al. Usability of residential thermostats: Preliminary investigations. Build. Environ. 46 (2011), 1891–1898, 10.1016/j.buildenv.2011.03.009.
87. Boynton, M.H., Richman, L.S., An online diary study of alcohol use using Amazon's Mechanical Turk. Drug Alcohol Rev. 33 (2014), 456–461, 10.1111/dar.12163.
88. Dorrian, J., et al. Morningness/eveningness and the synchrony effect for spatial attention. Accident Anal. Prev. 99 (2017), 401–405, 10.1016/j.aap.2015.11.012.
89. Benoit, K., et al. Crowd-sourced text analysis: Reproducible and agile production of political data. Am. Polit. Sci. Rev. 110 (2016), 278–295, 10.1017/S0003055416000058.
90. Mueller, P., Chandler, J., Emailing workers using Python. SSRN eLibrary., 2012, 10.2139/ssrn.2100601 Published online July 5, 2012. https://ssrn.com/abstract=2100601.
91. Reimers, S., Stewart, N., Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behav. Res. Methods 47 (2015), 309–327, 10.3758/s13428-014-0471-1.
92. Reimers, S., Stewart, N., Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments. Behav. Res. Methods 48 (2016), 897–908, 10.3758/s13428-016-0758-5.
93. de Leeuw, J., Jspsych: A javascript library for creating behavioral experiments in a web browser. Behav. Res. Methods 47 (2015), 1–12, 10.3758/s13428-014-0458-y.
94. Gureckis, T.M., et al. Psiturk: An open-source framework for conducting replicable behavioral experiments online. Behav. Res. Methods 48 (2016), 829–842, 10.3758/ s13428-015-0642-8.
95. Stoet, G., PsyToolkit: A software package for programming psychological experiments using Linux. Behav. Res. Methods 42 (2010), 1096–1104.
96. Stoet, G., Psytoolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teach. Psychol. 44 (2017), 24–31, 10.1177/0098628316677643.
97. Schubert, T.W., et al. ScriptingRT: A software library for collecting response latencies in online studies of cognition. PLoS One, 2013, 8, 10.1371/journal.pone.0067769.
98. Neath, I., et al. Response time accuracy in Apple Macintosh computers. Behav. Res. Methods 43 (2011), 353–362, 10.3758/s13428-011-0069-9.
99. Ulrich, R., Giray, M., Time resolution of clocks: Effects on reaction time measurement-Good news for bad clocks. Brit. J. Math. Stat. Psy. 42 (1989), 1–12, 10.1111/j.2044-8317.1989.tb01111.x.
100. Brand, A., Bradley, M.T., Assessing the effects of technical variance on the statistical outcomes of web experiments measuring response times. Soc. Sci. Comput. Rev. 30 (2012), 350–357, 10.1177/0894439311415604.
101. Semmelmann, K., Weigelt, S., Online psychophysics: Reaction time effects in cognitive experiments. Behav. Res. Methods., 2016, 10.3758/s13428-016-0783-4 Published online August 5, 2016.
102. Slote, J., Strand, J.F., Conducting spoken word recognition research online: Validation and a new timing method. Behav. Res. Methods 48 (2016), 553–566, 10.3758/s13428-015-0599-7.
103. Zhou, H., Fishbach, A., The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. J. Pers. Soc. Psychol. 111 (2016), 493–504, 10.1037/pspa0000056.
104. Coppock, A. Generalizing from survey experiments conducted on mechanical Turk: A replication approach. Polit. Sci. Res. Methods, in press. https://alexandercoppock.files.wordpress.com/2016/02/ coppock_generalizability2.pdf.