Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample

Sociological Science

Volumn 1, Issue , 2014, Pages 292-310

  Weinberg, Jill D ^a,b   Freese, Jeremy ^a   McElhattan, David ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

^b AMERICAN BAR FOUNDATION   (United States)

Author keywords

Amazon Mechanical Turk; Crowdsource; Factorial survey; Knowledge networks; Online experiment; Vignette experiment

Indexed keywords

EID: 85027982511     PISSN: None     EISSN: 23306696     Source Type: Journal    
DOI: 10.15195/v1.a19     Document Type: Article

References (41)

1. Alexander, Cheryl S. and Henry Jay Becker. 1978. "The Use of Vignettes in Survey Research." Public Opinion Quarterly 42(1):93-104. http://dx.doi.org/10.1086/268432
2. Amazon Mechanical Turk. 2014. "Service Summary." Seattle, Washington. Retrieved April 20, 2014 https://requester.mturk. com/tour.
3. Atzmüller, Christiane; Steiner, Peter M. 2010. "Experimental Vignette Studies in Survey Research." European Journal of Research Methods for the Behavioral and Social Sciences 6(3):128-38. http://dx.doi.org/10.1027/1614-2241/a000014
4. Auspurg, Katrin, Thomas Hinz, and Stefan Liebig. 2009. "Complexity, Learning Effects, and the Plausibility of Vignettes in Factorial Surveys." Paper presented at the American Sociological Association Annual Meeting, August 7-11, San Francisco, California.
5. Baker, Reg, Stephen J. Blumberg, J. Michael Brick, Mick P. Couper, Melanie Courtright, J. Michael Dennis, Don Dillman, Martin R. Frankel, Philip Garland, Robert M. Grovers, Courtney Kennedy, Jon Krosnick, and Paul J. Lavrakas. 2010. "Research Synthesis: AAPOR Report on Online Panels." Public Opinion Quarterly 74(4):711-81. http://dx.doi.org/10.1093/poq/nfq048
6. Berinsky, Adam J., Gregory A. Huber and Gabriel S. Lenz. 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk." Political Analysis 20(3): 351-68. http://dx.doi.org/10.1093/pan/mpr057
7. Buhrmester, Michael, Tracey Kwang, and Samuel D. Gosling. 2011. "Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High- Quality, Data?" Perspectives on Psychological Science 6(1): 3-5. http://dx.doi.org/10. 1177/1745691610393980
8. Chandler, Jesse, Pam Mueller, and Gabriele Paolacci. 2014. "Nonnaïveté Among Amazon Mechanical Turk Workers: Consequences and Solutions For Behavioral Researchers." Behavior Research Methods 46(1):112-30. http://dx.doi.org/10.3758/s13428-013-0365-7
9. Downs, Julie S., Mandy B. Holbrook, Steve Sheng, and Lorrie Faith Cranor. 2010. "Are Your Participants Gaming the System? Screening Mechanical Turk Workers." In Proceedings of the 28th International Conference on Human Factors in Computing Systems (2399-2402). New York: ACM.
10. Druckman, James D. and Cindy D. Kam. 2011. "Students as Experimental Participants: A Defense for the 'Narrow Data Base." ' Pp. 41-57 in Cambridge Handbook of Experimental Political Science, edited by James D. Druckman, Donald P. Green, James H. Kuklinski and Arthur Lupia. New York: Cambridge University Press.
11. Gätcher, Simon. 2010. "(Dis)advantages of Student Subjects: What is Your Research Question." Behavior and Brain Sciences 33(2-3): 92-93. http://dx.doi.org/10.1017/S0140525X10000099
12. Goodman, Joseph K., Cynthia E. Cryder, and Amar Cheema. 2012. "Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples." Journal of Behavioral Decision Making.
13. Gosling, Samuel D., Simine Vazie, Sanjay Srivastava, and Oliver P. John. 2004. "Should We Trust Web-Based Studies?" American Psychologist 59(2): 93-104. http://dx.doi. org/10.1037/0003-066X.59.2.93
14. Horton, John J. and Lydia B. Chilton. 2010. "The Labor Economics of Paid Crowdsourcing." Proceedings of the 11th ACM Conference on Electronic Commerce 2010. Retrieved July 18, 2014 from http://ssrn.com/abstract$=$1596874.
15. Horton, John J., David G. Rand and Richard J. Zeckhauser. 2011. "The Online Laboratory: Conducting Experiments in a Real Labor Market." Experimental Economics 14(3):399-425. http://dx.doi.org/10.1007/s10683-011-9273-9
16. Ipeirotis, Panos. 2010. "Demographics of Mechanical Turk. (CeDER Working Paper- 10-01). New York University. Retrieved July 18, 2014 from http://hdl.handle.net/2451/29585.
17. Jackson, Michelle and D. R. Cox. 2013. "The Principles of Experimental Design and Their Application to Sociology." Annual Review of Sociology 39: 27-49. http://dx.doi.org/10. 1146/annurev-soc-071811-145443
18. Kazai, Gabriella, Jaap Kamps, and Natasa Milic- Frayling. 2012. "Worker Types and Personality Traits of Crowdsourcing Relevance Labels" in Proceedings of 20th International Conference on Information and Knowledge Management (CIKM). ACM: New York.
19. Komarov, Steven, Katharina Reinecke, and Krzysztof Z. Gajos. 2013 "Crowdsourcing Performance Evaluations of User Interfaces" in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM: New York.
20. Lynch Jr., John G. 1982. "On the External Validity of Experiments in Consumer Research." Journal of Consumer Research 9(3):225-39.http://dx.doi.org/10.1086/208919
21. Mason, Winter and Siddarth Suri. 2012. "Conducting Behavioral Research on Amazon's Mechanical Turk." Behavior Research Methods 44(1): 1-23. http://dx.doi.org/10.3758/s13428-011-0124-6
22. Mullinix, Kevin J., James N. Druckman, and Jeremy Freese. "When Convenience Samples Yield Generalizable Estimates in Social Science Experiments." Unpublished manuscript.
23. Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.
24. Nelson, Thomas E., Rosalee A. Clawson, and Zoe M. Oxley. 1997. "Media Framing of a Civil Liberties Conflict and Its Effects on Tolerance." American Political Science Review 91(3):567-83. http://dx.doi.org/10.2307/2952075
25. Paolacci, Gabriele, Jesse Chandler, and Panos Ipeirotis. 2010. "Running Experiments on Amazon Mechanical Turk." Judgment and Decision Making 5(5) 411-19.
26. Peterson, Robert A. 2001. "On the Use of College Students in Social Science Research: Insights From a Second Order Meta-Analysis." Journal of Consumer Research 28 (3): 450-61. http://dx.doi.org/10.1086/323732
27. Phelan, Jo C., Bruce G. Link, and Naumi M. Feldman. 2013. "The Genomic Revolution and Beliefs about Essential Racial Differences: A Backdoor to Eugenics?" American Sociological Review 78(2): 167-91. http://dx.doi.org/10.1177/0003122413476034
28. Rand, David G. 2012. "The Promise of Mechanical Turk: How Online Labor Markets Can Help Theorists Run Behavioral Experiments." Journal of Theoretical Biology 299(21): 172-79. http://dx.doi.org/10.1016/j.jtbi. 2011.03.004
29. Regnerus, Mark. 2012. "How different are the adult children of parents who have same-sex relationships? Findings from the New Family Structures Study." Social Science Research. 41(4): 752-70. http://dx.doi.org/10.1016/j.ssresearch.2012.03.009
30. Reips, Ulf-Dietrich. 2002. "Standards for Internet-Based Experimenting." Experimental Psychology 49(4): 243-256.
31. Reips, Ulf-Dietrich. 2000. "The Web Experiment Method: Advantages, Disadvantages, and Solutions." Psychological Experiments on the Internet. Ed. Birnbaum, Michael H. San Diego: Academic Press. 89-118.
32. Rossi, Peter H. and Steven L. Nock. eds. 1982. Measuring Social Judgments: The Factorial Survey Approach. Beverly Hills, CA: SAGE Publications.
33. Sears, David O. 1986. "College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature." Journal of Personality and Social Psychology 51(3): 515-30. http://dx.doi. org/10.1037/0022-3514.51.3.515
34. Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
35. Shapiro, Danielle N., Jesse Chandler, and Pam A. Mueller. 2013. "Using Mechanical Turk to Study Clinical Populations." Clinical Psychological Science 1(2):213-20. http://dx.doi. org/10.1177/2167702612469015
36. Sniderman, Paul M. and Thomas Piazza. 2002. Black Pride and Black Prejudice. Princeton, NJ: Princeton University Press.
37. Tuckel, Peter and Harry O'Neill. 1995. "A Profile of Telephone Answering Machine Owners and Screeners" Proceedings of the Section on Survey Research Methods of the American Statistical Association. Retrieved July 18, 2014 from http://www.amstat.org/sections/srms/proceedings/papers/1995_201.pdf.
38. Webster, Murray and Jane Sell. 2007. Laboratory Experiments in the Social Sciences. London: Academic Press.
39. Weinberg, Jill D. n.d. "Identity and Social Acceptability: Public Perceptions of Reasonable Accommodation." Manuscript in Progress (on file with author).
40. Weinberg, Jill D., Laura Beth Nielsen, and Jeremy Freese 2012. "Rub My Shoulders But Don't Send Me Emails: Public Perceptions of Sexual Harassment." Paper presented at Annual Meeting of the Law and Society Association, June 5-8, 2012, Honolulu, HI.
41. Zelditch Jr, Morris. 1969. "Can You Really Study an Army in the Laboratory." Pp. 528-39 in A Sociological Reader on Complex Organizations Edited by Amitai Etzioni. Austin, Texas: Holt, Rinehart, and Winston.