The impact of speeding on data quality in nonprobability and freshly recruited probability-based online panels

Online Panel Research: A Data Quality Perspective

Volumn , Issue , 2014, Pages 238-262

  Greszki, Robert ^a   Meyer, Marco ^a   Schoen, Harald ^a  

^a UNIVERSITY OF BAMBERG   (Germany)

Author keywords

Data quality; Freshly random recruited respondents; Inattentive respondent; Measurement error; Nonprobability online panel; Quality indicators; Response times; Satisficing; Speeding; Web surveys

Indexed keywords

EID: 84927794426     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781118763520.ch11     Document Type: Chapter

References (68)

1. Alesina, A.,&Rosenthal, H. (1995). Partisan politics, divided government, and the economy. NewYork:Cambridge University Press.
2. Baker, R., Blumberg, S. J., Brick, M., Couper, M. P., Courtright,M., Dennis, J. M., et al. (2010). Research synthesis: AAPOR report on online panels. Public Opinion Quarterly, 74, 711-781.
3. Baker, R., & Downes-Le Guin, T. (2007). Separating the wheat from the chaff: Ensuring data quality in internet samples. In M. Trotman, T. Burrell, L. Gerrard, K. Anderton, G. Basi, M. Couper, K. Moris, et al. (Eds.), Proceedings of the Fifth International Conference of the Association for Survey Computing: The challenges of a changing world (pp. 157-166). Berkeley, UK: ASC.
4. Balden,W. (2008). An unwanted impact. Retrieved June 28, 2013, from: http://www.quirks.com/articles /2008/20080103.aspx?searchID=765725748&sort=5&pg=1.
5. Bassili, J. N. (1993). Response latency versus certainty as indexes of the strength of voting intentions in a CATI survey. Public Opinion Quarterly, 57, 54-61.
6. Bassili, J. N. (1996). The how and the why of response latency measurement in telephone surveys. In N. Schwarz, & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 319-346). San Francisco, CA: Jossey-Bass.
7. Bassili, J. N., & Fletcher, J. F. (1991). Response-time measurement in survey research: A method for CATI and a new look at nonattitudes. Public Opinion Quarterly, 55, 331-346.
8. Bassili, J. N.,&Scott, B. S. (1996). Response latency as a signal to question problems in survey research. Public Opinion Quarterly, 60, 390-399.
9. Berinsky, A. J., Margolis, M., & Sances, M.W. (2012). Separating the shirkers from the workers? Making sure respondents pay attention on internet surveys. Paper presented at the NYU CESS 5th Annual Experimental Political Science Conference, New York.
10. Bernstein, R., Chadha, A., & Montjoy, R. (2001). Overreporting voting: Why it happens and why it matters. Public Opinion Quarterly, 65, 22-44.
11. Bethlehem, J., & Biffignandi, S. (2012). Handbook of web surveys. Hoboken, NJ: John Wiley & Sons, Inc.
12. Blais, A. (2000). To vote or not to vote: The merits and limits of rational choice theory. Pittsburgh, PA:University of Pittsburgh Press.
13. Callegaro,M.,Yang,Y., Bhola, D. S.,Dillman,D.A.,&Chin, T. (2009). Response latency as an indicator of optimizing in online questionnaires. Bullétin de Méthodologie Sociologique, 103, 5-25.
14. Chang, L., & Krosnick, J. A. (2010). Comparing oral interviewing with self-administered computerized questionnaires: An experiment. Public Opinion Quarterly, 74, 154-167.
15. Coen, T., Lorch, J., & Piekarski, L. (2005). The effects of survey frequency on panelists' responses. Paper presented at the Worldwide Panel Research Conference ESOMAR, Amsterdam.
16. Cohen, J., Cohen, P.,West, S. G., & Aiken, L.S. (2003). Applied multiple regression/correlation analysis for the behavioral science. New York: Routledge.
17. Couper, M. P. (2000). Usability evaluation of computer-assisted survey instruments. Social Science Computer Review, 18, 384-396.
18. Couper, M. P. (2005). Technology trends in survey data collection. Social Science Computer Review, 23, 486-501.
19. DeBell, M.,Krosnick, J. A.,&Lupia, A. (2010). Methodology report and user's guide for the 2008-2009 ANES panel study. Palo Alto, CA and Ann Arbor, MI: Stanford University and the University of Michigan.
20. De Leeuw, E. D. (1992). Data quality in mail, telephone and face to face surveys. Amsterdam:TT-Publikaties.
21. Draisma, S., & Dijkstra, W. (2004). Response latency and (para)linguistic expressions as indicators of response error. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questions (pp. 131-147). Hoboken, NJ: John Wiley & Sons, Inc.
22. Duch, R. M., & Stevenson, R. T. (2008). The economic vote: How political and economic institutions condition election results. New York: Cambridge University Press.
23. Faas, T., & Schoen, H. (2006). Putting a questionnaire on the web is not enough: A comparison of online and offline surveys conducted in the context of the German Federal Election 2002. Journal of Official Statistics, 22, 177-190.
24. Fraley, R. C. (2004). How to conduct behavioral research over the internet: A beginner's guide to HTML and CGI/Perl. New York: The Guilford Press.
25. Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349-360.
26. GESIS - Leibniz-Institute for the Social Science (2009a). German Longitudinal Election Study 2009:Langfrist-Online-Tracking, T6: ZA5339, Version 2.0.0, Studienbeschreibung. Retrieved August 20, 2012, from: http://info1.gesis.org/dbksearch19/SDesc2.asp?no=5339&tab=3&ll=10¬abs=&af =&nf=1&search=gles&search2=&db=E.
27. GESIS - Leibniz-Institute for the Social Science (2009b). German Longitudinal Election Study 2009:Short-term Campaign Panel: ZA5305, Version 3.0.0, Study Description (ZA5305_sd_v3-0-0.pdf). Retrieved August 20, 2012, from: http://info1.gesis.org/dbksearch19/SDesc2.asp?no=5305&tab=3&ll=10¬abs=&af=&nf=1&search=gles&search2=&db=E.
28. Greszki, R. P., Meyer, M., & Schoen, H. (2012). Speeding in web surveys: A serious threat to data quality? Paper presented at the annual European Political Science Association Conference EPSA, Berlin.
29. Groves, R. M. (1991). Survey errors and survey costs. New York: John Wiley & Sons, Inc.
30. Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology. Hoboken, NJ: John Wiley & Sons, Inc.
31. Hansen, S. B. (1975). Participation, political structure, and concurrence. The American Political Science Review, 69, 1181-1199.
32. Heerwegh, D. (2003). Explaining response latencies and changing answers using client-side paradata from a web survey. Social Science Computer Review, 21, 360-373.
33. Heerwegh, D. (2011). Internet survey paradata. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social and behavioral research and the internet (pp. 325-348). New York: Routledge.
34. Johnson, M. (2004). Timepieces: Components of survey question response latencies. Political Psychology, 25, 679-702.
35. Kaczmirek, L. (2009). Human-survey interaction: Usability and nonresponse in online surveys. Cologne: Herbert von Halem Verlag.
36. Kahn, R. L., & Cannell, C. F. (1957). The dynamics of interviewing. New York: JohnWiley & Sons, Inc.
37. Kaminska, O., Goeminne, B., & Swyngedouw, M. (2006). Satisficing in early versus late responses to a mail survey. Leuven, NL: Katholieke Universiteit Leuven.
38. Kaminska, O., McCutcheon, A. L., & Billiet, J. (2010). Satisficing among reluctant respondents in a cross-national context. Public Opinion Quarterly, 74, 956-984.
39. Knapton, K., & Garlick, R. (2007). Catch me if you can. Retrieved August 28, 2013, from: http://www .quirks.com/articles/2007/20071107.aspx?searchID=765736159&sort=5&pg=1.
40. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236.
41. Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201-219.
42. Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., et al. (2002). The impact of "no opinion" response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371-403.
43. Krosnick, J. A., Narayan, S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. In M. T. Braverman, & J. K. Slater (Eds.), Advances in survey research (pp. 29-44). San Francisco, CA:Jossey-Bass.
44. Malhotra, N. (2008). Completion time and response order effects in web surveys. Public Opinion Quarterly, 72, 914-934.
45. Meade, A. W., & Craig, S.B. (2011). Identifying careless responses in survey data. Paper presented at the 26th Annual Meeting of the Society for Industrial and Organizational Psychology. Chicago, IL.
46. Meyer, M., & Schoen, H. (in press). Response latencies and attitude-behavior consistency in a direct democratic setting: Evidence from a subnational referendum in Germany. Political Psychology.
47. Miller, J. (2006). Research reveals alarming incidence of "undesirable" online panelists. Retrieved June 28, 2013, from: http://www.burke.com/Library/Articles/Jeff%20Miller%20RCR%20PDF.pdf.
48. Miller, J., & Baker-Prewitt, J. (2009). Beyond "trapping" the undesirable panelist: The use of red herrings to reduce satisficing. Retrieved June 28, 2013, from: http://www.burke.com/Library /Conference/Beyond%20Trapping%20the%20Undesirable%20Panelist_FINAL.pdf.
49. Narayan, S., & Krosnick, J.A. (1996). Education moderates some response effects in attitude measurement. Public Opinion Quarterly, 60, 58-88.
50. Puleston, J., & Sleep D. (2008). A look at the impact of boredom on the respondent experience. Retrieved June 28, 2013, from: http://www.quirks.com/articles/2008/20081106.aspx?searchID =765759022&sort=5&pg=1.
51. Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114, 510-532.
52. Rattinger, H., Rossteutscher, S., Schmitt-Beck, R.,&Wessels, B. (2009). German Longitudinal Election Study: Long-term Online Tracking, T6 (GLES 2009). Cologne:GESIS Leibnitz-Institute for the Social Science.
53. Respondi AG (2009a). Our answers to the 26 ESOMAR questions for determining the quality of online samples and online panels. Retrieved August 20, 2012, from: http://www.respondi.com.
54. Respondi AG (2009b). Panel book. Retrieved August 20, 2012, from: http://www.respondi.com.
55. Rossmann, J. (2010). Data quality in web surveys of the German Longitudinal Election Study 2009. Paper presented at the 3rd ECPR Graduate Conference, Dublin.
56. Schaeffer, N. C., Dykema, J.,&Maynard, D.W. (2010). Interviewers and interviewing. In P. T. Marsden, & J. D. Wright (Eds.), Handbook of survey research (pp. 437-470). Bingley, UK: Emerald.
57. Schoen, H. (2004). Online-Umfragen - schnell, billig, aber auch valide? Ein Vergleich zweier Internetbefragungen mit persönlichen Interviews zur Bundestagswahl 2002. ZA-Information, 54, 27-52.
58. Schuman, H., & Presser, S. (1996). Question and answers in attitude surveys. Thousand Oaks, CA:Sage.
59. Silver, B. D., Anderson, B. A., & Abramson, P. R. (1986). Who overreports voting? The American Political Science Review, 80, 613-624.
60. Smith, R.,&Brown,H.H. (2005).Assessing the quality of data from online panels:Moving forwardwith confidence. Retrieved November 14, 2012, from: http://www.harrisinteractive.com/servicebureau /pubs/HI_Quality_of_Data_White_Paper.pdf.
61. Topoel, V., Das, M., & Van Soest, A. (2008). Effects of design in web surveys: Comparing trained and fresh respondents. Public Opinion Quarterly, 72, 985-1007.
62. Tourangeau, R. (1984). Cognitive science and survey methods. In T. B. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73-100). Washington, DC: National Academy Press.
63. Tourangeau, R. (1987). Attitude measurement: A cognitive perspective. In H. J. Hippler, N. Schwarz, & S. Sudman (Eds.), Social information processing and survey methodology (pp. 149-162). New York:Springer.
64. Tourangeau, R. (2004). Survey research and societal change. Annual Review of Psychology, 55, 775-801.
65. Tourangeau, R., Couper, M. P., & Conrad, F. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68, 368-393.
66. Tourangeau, R., & Rasinski, K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314.
67. Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York:Cambridge University Press.
68. Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22, 51-68.