The viability of crowdsourcing for survey research

Behavior Research Methods

Volumn 43, Issue 3, 2011, Pages 800-813

  Behrend, Tara S ^a   Sharek, David J ^b   Meade, Adam W ^b   Wiebe, Eric N ^b  

^a GEORGE WASHINGTON UNIVERSITY   (United States)

^b NORTH CAROLINA STATE UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Industrial organizational psychology; Mechanical Turk; Personality; Sampling; Survey research

Indexed keywords

ARTICLE; BEHAVIORAL RESEARCH; HUMAN; INFORMATION PROCESSING; METHODOLOGY; PATIENT SELECTION; PERSONALITY; RESEARCH SUBJECT;

BEHAVIORAL RESEARCH; DATA COLLECTION; HUMANS; PATIENT SELECTION; PERSONALITY; RESEARCH DESIGN; RESEARCH SUBJECTS;

EID: 80052833420     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-011-0081-0     Document Type: Article

References (54)

1. Amazon. com. (2010). Mechanical Turk, Retrieved from http://www. mturk. com/.
2. Anderson, N. R. (2003). Applicant and recruiter reactions to new technology in selection: A critical review and agenda for future research. International Journal of Selection and Assessment, 11, 121-136. doi: 10. 1111/1468-2389. 00235.
3. Barchard, K., & Williams, J. (2008). Practical advice for conducting ethical online experiments and questionnaires for United States psychologists. Behavior Research Methods, 40, 1111-1128. doi: 10. 3758/BRM. 40. 4. 1111.
4. Booth-Kewley, S., Edwards, J. E., & Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference? The Journal of Applied Psychology, 77, 562-566.
5. Cohen, J. (1969). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
6. Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., et al. (2009). How well do line drawings depict shape? ACM Transactions on Graphics, 28, 1-9. doi: 10. 1145/1531326. 1531334.
7. Cole, M. S., Bedeian, A. G., & Field, H. S. (2006). The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership. Organizational Research Methods, 9, 339-368. doi: 10. 1177/1094428106287434.
8. Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349-354.
9. Dandurand, F., Shultz, T. R., & Onishi, K. H. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior Research Methods, 40, 428-434.
10. DeBeuckalaer, A., & Lievens, F. (2009). Measurement equivalence of paper-and-pencil and internet organizational surveys: A large-scale examination in 16 countries. Applied Psychology, 58, 336-361. doi: 10. 1111/j. 1464-0597. 2008. 00350. x.
11. Embretson, S. E., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
12. Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11, 93-103. doi: 10. 1177/014662168701100107.
13. Garland, K. J., & Noyes, J. M. (2004). Computer experience: A poor predictor of computer attitudes. Computers in Human Behavior, 20, 823-840. doi: 10. 1016/j. chb. 2003. 11. 010.
14. Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe vol. 7 (pp. 7-28). Tilburg, The Netherlands: Tilburg University Press.
15. Goldberg, L. R., & Kilkowski, J. M. (1985). The prediction of semantic consistency in self-descriptions: Characteristics of persons and terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48, 82-98. doi: 10. 1037/0022-3514. 48. 1. 82.
16. Gosling, S. D., Sandy, C. J., John, O. P., & Potter, J. (2010). Wired but not WEIRD: The promise of the Internet in reaching more diverse samples. Brain and Behavior Sciences, 33, 94-95. doi: 10. 1017/S0140525X10000300.
17. Heilman, M., & Smith, N. A. (2010). Rating computer-generated questions with Mechanical Turk. Paper presented at the NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk, Los Angeles, CA.
18. Henrich, J., Heine, S. J., & Norenzayan, A. (2010a). Most people are not WEIRD. Nature, 466, 2929. doi: 10. 1038/466029a.
19. Henrich, J., Heine, S. J., & Norenzayan, A. (2010b). The weirdest people in the world? The Behavioral and Brain Sciences, 33, 61-135. doi: 10. 1017/S0140525X0999152X.
20. Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), Retrieved from http://www. wired. com/wired/archive/14. 06/crowds. html.
21. Ipeirotis, P. G. (2010). Demographics of Mechanical Turk. Center for digital economy research working papers, 10. Retrieved from http://hdl. handle. net/2451/29585.
22. Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39, 103-129. doi: 10. 1016/j. jrp. 2004. 09. 009.
23. Kittur, A., Chi, E., Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. Paper presented at the Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy.
24. Kleemann, F., Voß, G., & Rieder, K. (2008). Un(der)paid innovators: The commercial utilization of consumer work through crowdsourcing. Science Technology & Innovation Studies, 4, 5-26.
25. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of board of scientific affairs advisory group on the conduct of research on the Internet. The American Psychologist, 59, 105-117. doi: 10. 1037/0003-066X. 59. 2. 105.
26. Landy, F. J. (2008). Stereotypes, bias, and personnel decisions: strange and stranger. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 379-392. doi: 10. 1111/j. 1754-9434. 2008. 00071. x.
27. Lautenschlager, G. J., & Meade, A. W. (2008). AlphaTest: A windows program for tests of hypotheses about coefficient alpha. Applied Psychological Measurement, 32, 502-503. doi: 10. 1177/0146621607312307.
28. Little, G., Chilton, L. B., Goldman, M., Miller, R. C. (2009). TurKit: Tools for iterative tasks on Mechanical Turk. Paper presented at the ACM SIGKDD Workshop on Human Computation, Paris, France.
29. Locke, E. A. (Ed.). (1986). Generalizing from laboratory to field settings. Lexington, MA: Lexington Books.
30. Manfreda, K. L., Bosnjak, M., Berzelak, J., Haas, I., & Vehovar, V. (2008). Web surveys versus other modes. International Journal of Market Research, 50, 79-104.
31. Meade, A. W. (2010). A taxonomy of effect size measures for the differential item functioning of items and scales. The Journal of Applied Psychology, 95, 728-743.
32. Meade, A. W., Michels, L. C., & Lautenschlager, G. J. (2007). Are Internet and paper-and-pencil personality tests truly comparable? An experimental design measurement invariance study. Organizational Research Methods, 10, 322-345. doi: 10. 1177/1094428106289393.
33. Meyerson, P., & Tryon, W. W. (2003). Validating internet research: A test of the psychometric equivalence of internet and in-person samples. Behavior Research Methods, Instruments, & Computers, 35, 614-620.
34. Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. F. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: are personality, biodata, and situational judgment tests comparable? PersonnelPsychology, 56, 733-752. doi: 10. 1111/j. 1744-6570. 2003. tb00757. x.
35. Potosky, D. (2007). The Internet knowledge (iKnow) measure. Computers in Human Behavior, 23, 2760-2777. doi: 10. 1016/j. chb. 2006. 05. 003.
36. Potosky, D., & Bobko, P. (1998). The computer understanding and experience scale: A self-report measure of computer experience. Computers in Human Behavior, 14, 337-348.
37. Raju, N. S., van der Linden, W., & Fleer, P. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.
38. Reips, U. D. (2002). Standards for internet-based experimenting. Experimental Psychology, 49, 243-256. doi: 10. 1027//1618-3169. 49. 4. 243.
39. Richman, W., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. The Journal of Applied Psychology, 84, 754-775.
40. Rivas, L., Gabriel, E., Stark, S., & Chernyshenko, O. S. (2009). The effects of referent item parameters on differential item functioning detection using the free baseline likelihood ratio test. Applied Psychological Measurement, 33, 251-265.
41. Rosnow, R. L., & Rosenthal, R. (1976). The volunteer subject revisited. Australian Journal of Psychology, 28, 97-108.
42. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph, 17.
43. Sharek, D. (2010). The influence of flow in the measure of engagement. Unpublished Master's thesis. Retrieved from http://www. lib. ncsu. edu/theses/available/etd-02182010-132225/.
44. Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43, 155-167. doi: 10. 3758/s13428-010-0039-7.
45. Stanton, J. M. (1998). An empirical assessment of data collection using the internet. Personnel Psychology., 51, 709-725.
46. Stanton, J. M., & Rogelberg, S. G. (2001). Using internet/intranet web pages to collect organizational research data. Organizational Research Methods, 4, 200-217.
47. Thissen, D. (2001). Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill: Universityof North Carolina at Chapel Hill, L. L. Thurstone Psychometric Laboratory.
48. Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ: Erlbaum.
49. Thompson, L. F., & Surface, E. A. (2007). Employee surveys administered online: Attitudes toward the medium, nonresponse, and data representativeness. Organizational Research Methods, 10, 241-261. doi: 10. 1177/1094428106/294696.
50. Tippins, N. T. (2002). Organization development and IT: Practicing OD in the virtual world. In J. Waclawski & A. H. Church (Eds.), Organization development: A data-driven approach to organizational change (pp. 245-265). San Francisco: Jossey-Bass.
51. Truell, A. D., Bartlett, J. E., & Alexander, M. W. (2002). Response rate, speed, and completeness: A comparison of Internet-based and mail surveys. Behavior Research Methods, Instruments, & Computers, 34, 46-49.
52. Turk and rescue. (2007). Economist, 384(8547), 97.
53. VandeWalle, D. (1997). Development and validation of a work domain goal orientation instrument. Educational and Psychological Measurement, 57, 995-1015. doi: 10. 1177/0013164497057006009.
54. Ward, E. A. (1993). Generalizability of psychological research from undergraduates to employed adults. The Journal of Social Psychology, 133, 513-519.