Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies

Journal of Experimental Political Science

Volumn 1, Issue 2, 2014, Pages 120-131

  Clifford, Scott ^a   Jerit, Jennifer ^b  

^a UNIVERSITY OF HOUSTON   (United States)

^b STONY BROOK UNIVERSITY   (United States)

Author keywords

attention; lab experiment; Mode effects; online experiment; political knowledge; social desirability

Indexed keywords

EID: 84962153074     PISSN: 20522630     EISSN: 20522649     Source Type: Journal    
DOI: 10.1017/xps.2014.5     Document Type: Article

References (31)

1. Bargh, J., and Chartrand, T. L. 2000. The Mind in the Middle: A Practical Guide to Priming and Automaticity Research. In Handbook of Research Methods in Social and Personality Psychology eds. H. T. Reis and C. M. Judd. New York: Cambridge University Press, 253-85.
2. Berinsky, A. J. 1999. Can We Talk? Self-Presentation and the Survey Response. Political Psychology 25 (4): 643-59.
3. Berinsky, A. J., Huber, G. A., and Lenz, G. S. 2012. Using Mechanical Turk as a Subject Recruitment Tool for Experimental Research. Political Analysis 20 (3): 351-68.
4. Berinsky, A. J., Margolis, M., and Sances, M. 2014. Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Internet Surveys. American Journal of Political Science Forthcoming.
5. Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., and Monroe, K. 1992. MMPI-2 Random Responding Indices: Validation Using a Self-Report Methodology. Psychological Assessment 4 (3): 340-45.
6. Boster, F., and Shulman, H. 2014. Political Knowledge Test Performance as a Function of Venue, Time Pressure, and Performance Norms. Working Paper, North Central College.
7. Cassese, E. C., Huddy, L., Hartman, T. K., Mason, L., and Weber, C. R. 2013. Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments. PS: Political Science and Politics 46 (4): 775-84.
8. Chang, L., and Krosnick, J. A. 2009. National Surveys via RDD Telephone Interviewing versus the Internet. Public Opinion Quarterly 73 (4): 641-78.
9. Chang, L., and Krosnick, J. A. 2010. Comparing Oral Interviewing with Self-Administered Computerized Questionnaires. Public Opinion Quarterly 74 (1): 154-67.
10. Coppock, A., and Green, D. P. 2013. Assessing the Correspondence Between Experimental Results Obtained in the Lab and Field:AReview of Recent Social Science Research. Working Paper, Columbia University.
11. Evans, D. C., Garcia, D. J., Garcia, D. M., and Baron, R. S. 2003. In the Privacy of Their Own Homes: Using the Internet to Assess Racial Bias. Personality and Social Psychology Bulletin 29 (2): 273-84.
12. Gerber, A. 2011. Field Experiments in Political Science. In Handbook of Experimental Political Science eds. J. Druckman, D. P. Green, J. H. Kuklinski, and A. Lupia. New York: Cambridge University Press, 115-38.
13. Goodman, J. K., Cryder, C. E., and Cheema, A. 2013. Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making 26 (3): 213-24.
14. Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., and De Shon, R. P. 2012. Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business Psychology 27: 99-114.
15. Jerit, J., Barabas, J., and Clifford, S. 2013. Comparing Contemporaneous Laboratory and Field Experiments on Media Effects. Public Opinion Quarterly 77 (1): 256-82.
16. Lelkes, Y., Krosnick, J.A., Marx, D.M., Judd, C.N., and Park, B. 2012. Complete Anonymity Compromises the Accuracy of Self-Reports. Journal of Experimental Social Psychology 48: 1291-99.
17. Lodge, M., and Taber, C. S. 2013. The Rationalizing Voter. New York: Cambridge University Press.
18. Meade, A. W., and Craig, S. B. 2012. Identifying Careless Responses in Survey Data. Psychological Methods 17 (3): 437-55.
19. McConahay, J. G. 1986. Modern Racism, Ambivalence, and the Modern Racism Scale. In Prejudice, Discrimination, and Racism eds. J. F. Dovidio and S. L. Gaertner. New York: Academic Press, 91-125.
20. McDermott, R. 2002. Experimental Methods in Political Science. Annual Review of Political Science 5: 31-61.
21. Morton, R. B., and Williams, K. C. 2010. Experimental Political Science and the Study of Causality. New York: Cambridge University Press.
22. Mutz, D. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.
23. Oppenheimer, D., Meyvis, T., and Davidenko, N. 2009. Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power. Journal of Experimental Social Psychology 45: 867-72.
24. Prior, M. 2009. Improving Media Effects Research Through Better Measurement of News Exposure. Journal of Politics 71 (3): 893-908.
25. Sargis, E.G., Skitka, L. J., and McKeever, W. 2014. The Internet as Psychological Laboratory Revisited: Best Practices, Challenges, and Solutions. In The Social Net: The Social Psychology of the Internet ed. Y. Amichai-Hamberger. Oxford: Oxford University Press. In Press.
26. Tourangeau, R., and Smith, T. W. 1996. Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context. Public Opinion Quarterly 60 (2): 275-304.
27. Tourangeau, R., and Yan, T. 2007. Sensitive Survey Questions. Psychological Bulletin 133 (5): 859-83.
28. Vavreck, L. 2012. The Myth of Cheating on Self-Completed Surveys. http://today. yougov.com/news/2012/04/17/myth-cheating-self-completed-surveys/ (Accessed June 10, 2013).
29. Warren, J. 2012. Fake Orgasms and the Tea Party: Just Another Political Science Convention. The Atlantic Online. http://www.theatlantic.com/politics/archive/2012/04/fake-orgasms-and-the-tea-party-just-another-political-scienceconvention/255909/ (Accessed June 10, 2013).
30. Weigold, A., Weigold, I. K., and Russell, E. J. 2013. Examination of the Equivalence of Self-Report Survey-Based Paper-and-Pencil and Internet Data Collection Methods. Psychological Methods 18 (1): 53-70.
31. Weinberger, J., and Westen, D. 2008. RATS, We Should Have Used Clinton: Subliminal Priming in Political Campaigns. Political Psychology 29 (5): 631-51.