Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection

SAGE Open

Volumn 7, Issue 2, 2017, Pages

  Casey, Logan S ^a   Chandler, Jesse ^b,c   Levine, Adam Seth ^d   Proctor, Andrew ^e   Strolovitch, Dara Z ^e  

^a HARVARD SCHOOL OF PUBLIC HEALTH   (United States)

^b UNIVERSITY OF MICHIGAN   (United States)

^c MATHEMATICA POLICY RESEARCH   (United States)

^d Department of Obstetrics Gynecology   (United States)

^e PRINCETON UNIVERSITY   (United States)

Author keywords

data collection; political behavior psychology; political methodology; political science; politics and social sciences; psychology; reliability and validity; research methodology and design; research methods; social sciences

Indexed keywords

EID: 85021748541     PISSN: None     EISSN: 21582440     Source Type: Journal    
DOI: 10.1177/2158244017712774     Document Type: Article

References (62)

1. Aguinis H., Lawal S. O., (2013). eLancing: A review and research agenda for bridging the science–practice gap. Human Resource Management Review, 23, 6-17. doi:10.1016/j.hrmr.2012.06.003
2. American National Election Studies. (2012). The ANES 2012 Time Series Study [dataset]. Stanford University and University of Michigan [producers]. Available from www.electionstudies.org
3. Arechar A. A., Kraft-Todd G. T., Rand D. G., (2016). Turking overtime: How participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. Retrieved from https://ssrn.com/abstract=2836946
4. Aviv A. L., Zelenski J. M., Rallo L., Larsen R. J., (2002). Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences, 33, 487-496. doi:10.1016/S0191-8869(01)00199-4
5. Benjamini Y., Hochberg Y., (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B: Methodological, 57, 289-300. doi:10.2307/2346101
6. Berinsky A. J., Huber G. A., Lenz G. S., (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351-368. doi:10.1093/pan/mpr057
7. Brawley A. M., Pury C. L., (2016). Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54, 531-546.
8. Bureau of Labor Statistics, U.S. Department of Labor. (2016). Reasons people give for not being in the labor force, 2004 and 2014 on the Internet. The Economics Daily. Retrieved from https://www.bls.gov/opub/ted/2016/reasons-people-give-for-not-being-in-the-labor-force-2004-and-2014.htm
9. Casler K., Bickel L., Hackett E., (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156-2160. doi:10.1016/j.chb.2013.05.009
10. Chandler J., Mueller P., Paolacci G., (2014). Nonnaive among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavioral Research Methods Science, 46, 112-130.
11. Chandler J., Paolacci G., Peer E., Mueller P., Ratliff K. A., (2015). Using nonnative participants can reduce effect sizes. Psychological Science, 26, 1131-1139. doi:10.1177/0956797615585115
12. Chandler J., Shapiro D., (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53-81. doi:10.1146/annurev-clinpsy-021815-093623
13. Cooper H., Baumgardner A. H., Strathman A., (1991). Do students with different characteristics take part in psychology experiments at different times of the semester? Journal of Personality, 59, 109-127. doi:10.1111/j.1467-6494.1991.tb00770.x
14. Corrigan P. W., Bink A. B., Fokuo J. K., Schmidt A., (2015). The public stigma of mental illness means a difference between you and me. Psychiatry Research, 226, 186-191. doi:10.1016/j.psychres.2014.12.047
15. Couper M. P., (2000). Review: Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494. doi:10.1086/318641
16. de Bruijne M., Wijnant A. (2014a). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78, 951-962. doi:10.1093/poq/nfu046
17. de Bruijne M., Wijnant A. (2014b). Mobile response in web panels. Social Science Computer Review, 32, 728-742. doi:10.1177/0894439314525918
18. Dietrich B. J., Lasley S., Mondak J. J., Remmel M. L., Turner J., (2012). Personality and legislative politics: The Big Five trait dimensions among U.S. state legislators. Political Psychology, 33, 195-210.
19. Ebersole C. R., Atherton O. E., Belanger A. L., Skulborstad H. M., Allen J., Banks J. B... Nosek B. A., (2016, June 28). Many labs 3: Evaluating participant pool quality across the academic semester via replication. Retrieved from osf.io/ct89g
20. Filion F. L., (1975). Estimating bias due to nonresponse in mail surveys. Public Opinion Quarterly, 39, 482-492. doi:10.1086/268245
21. Flores A., Herman J. L., Gates G. J., Brown T. N. T., (2016). How many adults identify as transgender in the United States? The Williams Institute. Retrieved from https://williamsinstitute.law.ucla.edu/research/how-many-adults-identify-as-transgender-in-the-united-states/
22. Fraley R. C., Vazire S., (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE, 9(10), e109019. doi:10.1371/journal.pone.0109019
23. Gannon M. J., Nothern J. C., Carroll S. J., (1971). Characteristics of nonrespondents among workers. Journal of Applied Psychology, 55, 586-588. doi:10.1037/h0031907
24. Gates G., (2014). LGBT demographics: Comparisons among population-based surveys. The Williams Institute. Retrieved from https://williamsinstitute.law.ucla.edu/wp-content/uploads/lgbt-demogs-sep-2014.pdf
25. Gates G., Newport F., (2012, October 18). Special report: 3.4% of U.S. adults identify as LGBT. Retrieved from http://opiniontoday.com/2012/10/18/special-report-3-4-of-u-s-adults-identify-as-lgbt/
26. Gerber A. S., Huber G. A., Doherty D., Dowling C. M., (2011). The Big Five personality traits in the political arena. Annual Review of Political Science, 14, 265-287. doi:10.1146/annurev-polisci-051010-111659
27. Goldin G., Darlow A., (2013). TurkGate (Version 0.4. 0) [Software]. Retrieved from http://gideongoldin.github.io/TurkGate/
28. Gosling S. D., Rentfrow P. J., Potter J., (2014). Norms for the Ten Item Personality Inventory (Unpublished data). Retrieved from http://gosling.psy.utexas.edu/scales-weve-developed/ten-item-personality-measure-tipi/.
29. Gosling S. D., Rentfrow P. J., Swann W. B., (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37, 504-528.
30. Guidelines for academic requesters. (2014). Version 1.1. Retrieved from https://irb.northwestern.edu/sites/irb/files/documents/guidelinesforacademicrequesters.pdf
31. Haupert M. L., Gesselman A. N., Moors A. C., Fisher H. E., Garcia J. R., (2016). Prevalence of experiences with consensual nonmonogamous relationships: Findings from two national samples of single Americans. Journal of Sex & Marital Therapy, 22, 1-17.
32. Hillygus D. S., Jackson N., Young M., (2014). Professional respondents in nonprobability online panels. In Callegaro M., Baker R., Bethlehem J., Göritz A. S., Krosnick J. A., Lavrakas P. J., (Eds.), Online panel research (pp. 219-237). John Wiley. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/9781118763520.ch10/summary
33. Ho D. E., Imai K., King G., Stuart E. A., (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199-236. doi:10.1093/pan/mpl013
34. Howe L. D., Hargreaves J. R., Ploubidis G. B., De Stavola B. L., Huttly S. R. A., (2011). Subjective measures of socio-economic position and the wealth index: A comparative analysis. Health Policy and Planning, 26, 223-232. doi:10.1093/heapol/czq043
35. John O. P., Srivastava S., (1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In Handbook of personality: Theory and research (Vol. 2, pp. 102-138). New York: The Guilford Press.
36. Komarov S., Reinecke K., Gajos K. Z., (2013). Crowdsourcing performance evaluations of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 207-216). New York, NY: ACM. doi:10.1145/2470654.2470684
37. Krupnikov Y., Levine A. S., (2014). Cross-sample comparisons and external validity. Journal of Experimental Political Science, 1, 59-80. doi:10.1017/xps.2014.7
38. Lakkaraju K., (2015). A study of daily sample composition on Amazon Mechanical Turk. In Agarwal N., Xu K., Osgood N., (Eds.), Social computing, behavioral-cultural modeling, and prediction (pp. 333-338). Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-3-319-16268-3_39
39. Litman L., Robinson J., Abberbock T., (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49, 433-442. doi:10.3758/s13428-016-0727-z
40. Mavletova A., (2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31(6), 725-743. doi:10.1177/0894439313485201
41. Moore P., (2015). A third of young Americans say they aren’t 100% heterosexual. YouGov. Retrieved from https://today.yougov.com/news/2015/08/20/third-young-americans-exclusively-heterosexual/
42. Mullinix K. J., Leeper T. J., Druckman J. N., Freese J., (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2, 109-138. doi:10.1017/XPS.2015.19
43. Paolacci G., Chandler J., (2014). Inside the turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. doi:10.1177/0963721414531598
44. Paolacci G., Chandler J., Ipeirotis P. G., (2010). Running experiments on Amazon Mechanical Turk (SSRN Scholarly Paper No. ID 1626226). Rochester, NY: Social Science Research Network. Retrieved from http://papers.ssrn.com/abstract=1626226
45. Peer E., Brandimarte L., Samat S., Acquisti A., (2017). Beyond the turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153-163.
46. Peer E., Vosgerau J., Acquisti A., (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023-1031. doi:10.3758/s13428-013-0434-y
47. Peterson R. A., Merunka D. R., (2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67, 1035-1041. doi:10.1016/j.jbusres.2013.08.010
48. Pew Research Center. (2016a). A new estimate of the U.S. Muslim population. Retrieved from http://www.pewresearch.org/fact-tank/2016/01/06/a-new-estimate-of-the-u-s-muslim-population/
49. Pew Research Center. (2016b). 10 facts about atheists. Retrieved from http://www.pewresearch.org/fact-tank/2016/06/01/10-facts-about-atheists/
50. Ravallion M., Lokshin M., (1999). Subjective economic welfare. The World Bank. Retrieved from http://elibrary.worldbank.org/doi/abs/10.1596/1813-9450-2106
51. Reidy D. E., Berke D. S., Gentile B., Zeichner A., (2014). Man enough? Masculine discrepancy stress and intimate partner violence. Personality and Individual Differences, 68, 160-164. doi:10.1016/j.paid.2014.04.021
52. Shapiro D. N., Chandler J., Mueller P. A., (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1(2), 213-220. doi:10.1177/2167702612469015
53. Sigman R., Lewis T., Yount N. D., Lee K., (2014). Does the length of fielding period matter? Examining response scores of early versus late responders. Journal of Official Statistics, 30, 651-674. doi:10.2478/jos-2014-0042
54. Sommer J., Diedenhofen B., Musch J., (2016). Not to be considered harmful: Mobile-device users do not spoil data quality in web surveys. Social Science Computer Review, 35(3): 378-387. doi:10.1177/0894439316633452
55. Stewart N., Ungemach C., Harris A. J. L., Bartels D. M., Newell B. R., Paolacci G., Chandler J., (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479-491.
56. Struminskaya B., Weyandt K., Bosnjak M., (2015). The effects of questionnaire completion using mobile devices on data quality: Evidence from a probability-based general population panel. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology, 9, 261-292. doi:10.12758/mda.2015.014
57. Sturgis P., Allum N., Brunton-Smith I., (2009). Attitudes over time: The psychology of panel conditioning. In Lynn P., (Ed.), Methodology of longitudinal surveys (pp. 113-126). John Wiley. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/9780470743874.ch7/summary
58. U.S. Census Bureau. (2011-2015). American community survey 5-year estimates. Retrieved from https://www.census.gov/data/developers/data-sets/acs-5year.html
59. U.S. Census Bureau. (2016). Current population survey, annual social and economic supplement. Retrieved from https://www.census.gov/cps/data/cpstablecreator.html
60. Voigt L. F., Koepsell T. D., Daling J. R., (2003). Characteristics of telephone survey respondents according to willingness to participate. American Journal of Epidemiology, 157, 66-73. doi:10.1093/aje/kwf185
61. Weinberg J., Freese J., McElhattan D., (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292-310. doi:10.15195/v1.a19
62. Wells T., Bailey J., Link M., (2013). Filling the void: Gaining a better understanding of tablet-based surveys. Survey Practice, 6(1). Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/25