Crowdsourcing a normative natural language dataset: A comparison of amazon mechanical turk and in-lab data collection

Journal of Medical Internet Research

Volumn 15, Issue 5, 2013, Pages

  Saunders, Daniel R ^a   Bex, Peter J ^a   Woods, Russell L ^a  

^a SCHEPENS EYE RESEARCH INSTITUTE   (United States)

Author keywords

Crowdsourcing; Free recall; Internet; Web

Indexed keywords

ADULT; AGED; ARTICLE; COMPUTER LANGUAGE; CROWDSOURCING; FEMALE; HUMAN; INFORMATION PROCESSING; INTERNET; MALE; METHODOLOGY; MIDDLE AGED; RECALL; UNITED STATES; WEB; YOUNG ADULT; PROCEDURES;

CROWDSOURCING; FREE RECALL; INTERNET; WEB;

ADULT; AGED; CROWDSOURCING; DATA COLLECTION; FEMALE; HUMANS; MALE; MIDDLE AGED; PROGRAMMING LANGUAGES; UNITED STATES; YOUNG ADULT;

EID: 84878732683     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/jmir.2620     Document Type: Article

References (31)

1. Swan M. Crowdsourced health research studies: An important emerging complement to clinical trials in the public health research ecosystem. J Med Internet Res 2012;14(2):e46 [FREE Full text] [doi: 10.2196/jmir.1988] [Medline: 22397809].
2. Tung JY, Do CB, Hinds DA, Kiefer AK, Macpherson JM, Chowdry AB, et al. Efficient replication of over 180 genetic associations with self-reported medical data. PLoS One 2011;6(8):e23473 [FREE Full text] [doi: 10.1371/journal.pone.0023473] [Medline: 21858135].
3. Duchaine BC, Nakayama K. Developmental prosopagnosia: A window to content-specific face processing. Curr Opin Neurobiol 2006 Apr;16(2):166-173. [doi: 10.1016/j.conb.2006.03.003] [Medline: 16563738].
4. Frost J, Okun S, Vaughan T, Heywood J, Wicks P. Patient-reported outcomes as a source of evidence in off-label prescribing: Analysis of data from PatientsLikeMe. J Med Internet Res 2011;13(1):e6 [FREE Full text] [doi: 10.2196/jmir.1643] [Medline: 21252034].
5. Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. J Med Internet Res 2012;14(3):e79 [FREE Full text] [doi: 10.2196/jmir.2063] [Medline: 22664384].
6. Saunders DR, Bex PJ, Rose DJ, Woods RL. Measuring information acquisition from sensory input using automated scoring of natural-language descriptions. Manuscript submitted for publication 2013 (forthcoming).
7. Howe J. The rise of crowdsourcing. Wired Magazine 2006 Jun;14.
8. Marge M, Banerjee S, Rudnicky AI. Using the Amazon Mechanical Turk for transcription of spoken language. 2010 Presented at: IEEE Conference on Acoustics, Speech and Signal Processing; March 25-30, 2010; Dallas, TX p. 14-19. [doi: 10.1109/icassp.2010.5494979].
9. Snow R, O'Connor B, Jurafsky D, Ng AY. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. 2008 Presented at: Conference on Empirical Methods in Natural Language Processing; Oct. 25-27, 2008; Honolulu, Hawaii.
10. Alonso O, Mizzaro S. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. 2009 Presented at: SIGIR 2009 Workshop on the Future of IR Evaluation; July 23, 2009; Boston, MA.
11. Paolacci G, Chandler J, Ipeirotis PG. Running experiments on Amazon Mechanical Turk. Judg Dec Mak 2010;5(5):411-419.
12. Behrend TS, Sharek DJ, Meade AW, Wiebe EN. The viability of crowdsourcing for survey research. Behav Res Methods 2011 Sep;43(3):800-813. [doi: 10.3758/s13428-011-0081-0] [Medline: 21437749].
13. Nosek BA, Banaji MR, Greenwald AG. E-Research: Ethics, security, design, and control in psychological research on the Internet. J Soc Issues 2002;58(1):161-176. [doi: 10.1111/1540-4560.00254].
14. Kraut R, Olson J, Banaji M, Bruckman A, Cohen J, Couper M. Psychological research online: Report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet. Am Psychol 2004;59(2):105-117. [doi: 10.1037/0003-066X.59.2.105] [Medline: 14992637].
15. Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc 2005 Apr;53(4):695-699. [doi: 10.1111/j.1532-5415.2005.53221.x] [Medline: 15817019].
16. Cutting JE, DeLong JE, Nothelfer CE. Attention and the evolution of Hollywood film. Psychol Sci 2010 Mar;21(3):432-439. [doi: 10.1177/ 0956797610361679] [Medline: 20424081].
17. Brainard DH. The Psychophysics Toolbox. Spat Vis 1997;10(4):433-436. [Medline: 9176952].
18. Zeimpekis D, Gallopoulos E. Design of a MATLAB toolbox for term-document matrix generation. : SIAM; 2005 Presented at: Workshop on Clustering High Dimensional Data and its Applications; April 20-23, 2005; Newport Beach, CA p. 38-48.
19. U.S. Census Bureau. Age and Sex Composition 2010. 2011. URL: http://www.census.gov/prod/cen2010/briefs/c2010br-03.pdf [accessed 2013-03-14] [WebCite Cache ID 6F7WG1fqR].
20. U.S. Census Bureau. Overview of Race and Hispanic Origin 2010. 2011. URL: http://www.census.gov/prod/cen2010/briefs/c2010br-02.pdf [accessed 2013-03-14] [WebCite Cache ID 6F7WpfJsj].
21. U.S. Census Bureau. Educational Attainment of the Population 18 Years and Over, by Age, Sex, Race, and Hispanic Origin 2011. 2012. URL: http://www.census.gov/hhes/socdemo/education/data/cps/2011/tables.html [accessed 2013-03-14] [WebCite Cache ID 6F7WeVZUT].
22. Ross J, Irani L, Silberman M, Zaldivar A, Tomlinson B. Who are the crowdworkers? Shifting demographics in Mechanical Turk. 2010 Presented at: ACM SIGCHI Conference on Human Factors in Computing Systems; April 10-15, 2010; Atlanta, GA.
23. Janssen DP. Twice random, once mixed: Applying mixed models to simultaneously analyze random effects of language and participants. Behav Res Methods 2012 Mar;44(1):232-247. [doi: 10.3758/s13428-011-0145-1] [Medline: 21858733].
24. Zhai H, Lingren T, Deleger L, Li Q, Kaiser M, Stoutenborough L, et al. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. J Med Internet Res 2013;15(4):e73 [FREE Full text] [doi: 10.2196/jmir.2426] [Medline: 23548263].
25. Saunders DR, Bex PJ, Woods RL. A novel, objective measure of information acquisition from video is correlated with blur. 2012 Presented at: The Association for Research in Vision and Ophthalmology Annual Meeting; May 6-10, 2012; Fort Lauderdale, FL.
26. Woods RL, Satgunam P. Television, computer and portable display device use by people with central vision impairment. Ophthalmic Physiol Opt 2011 May;31(3):258-274 [FREE Full text] [doi: 10.1111/j.1475-1313.2011.00833.x] [Medline: 21410501].
27. Griffin ZM, Spieler DH. Observing the what and when of language production for different age groups by monitoring speakers' eye movements. Brain Lang 2006 Dec;99(3):272-288. [doi: 10.1016/j.bandl.2005.08.003] [Medline: 16290041].
28. Mason W, Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behav Res Methods 2012 Mar;44(1):1-23. [doi: 10.3758/s13428-011-0124-6] [Medline: 21717266].
29. Von Ahn L. Games with a purpose. Computer 2006;39(6):92-94.
30. Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: An online game for analyzing images of infected thick blood smears. J Med Internet Res 2012;14(6):e167 [FREE Full text] [doi: 10.2196/jmir.2338] [Medline: 23196001].
31. Crump MJ, McDonnell JV, Gureckis TM. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLoS One 2013;8(3):e57410 [FREE Full text] [doi: 10.1371/journal.pone.0057410] [Medline: 23516406].