Analyzing Paradata to Investigate Measurement Error

Improving Surveys with Paradata: Analytic Uses of Process Information

Volumn , Issue , 2013, Pages 73-95

  Yan, Ting ^a   Olson, Kristen ^b  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF NEBRASKA LINCOLN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85017386478     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781118596869.ch4     Document Type: Chapter

References (75)

1. Bassili, J. (1993). Response Latency Versus Certainty as Indexes of the Strength of Voting Intentions in a Cati Survey. The Public Opinion Quarterly, 57(1): 54-61.
2. Bassili, J. (1995). Response Latency and the Accessibility of Voting Intentions: What Contributes to Accessibility and how it Affects Vote Choice. Personality and Social Psychological Bulletin, 21(7): 686-695.
3. Bassili, J. (1996). The How and Why of Response Latency Measurement in Telephone Surveys. In Schwarz, N. and Sudman, S., editors, Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research, pages 319-346. Jossey-Bass Publishers.
4. Bassili, J. and Fletcher, J. (1991). Response-Time Measurement in Survey Research a Method for CATI and a New Look at Nonattitudes. Public Opinion Quarterly, 55(3): 331-346.
5. Bassili, J. and Krosnick, J.A. (2000). Do Strength-Related Attitude Properties Determine Susceptibility to Response Effects? New Evidence from Response Latency, Attitude Extremity, and Aggregate Indices. Political Psychology, 21(1): 107-132.
6. Bassili, J. and Scott, B. (1996). Response Latency as a Signal to Question Problems in Survey Research. Public Opinion Quarterly, 60(3): 390-399.
7. Belli, R., Lepkowski, J.M., and Kabeto, M. (2001). The Respective Roles of Cognitive Processing Difficulty and Conversational Rapport on the Accuracy of Retrospective Reports of Doctor's Office Visits. In Cynamon, M., and Kulka, R., editors, Seventh Conference of Health Survey Research Methods, pages 197-203. DHHS Publication No. (PHS) 01-1013. US Government Printing Office, Hyattsville, MD.
8. Benkí, J.R., Broome, J., Conrad, F.G., Groves, R.M., and Kreuter, F. (2011). Effects of Speech Rate, Pitch, and Pausing on Survey Participation Decisions. Paper presented at the American Association for Public Opinion Research Annual Meeting, Phoenix, AZ.
9. Biemer, P.P. and Lyberg, L.E. (2003). Introduction to Survey Quality. Wiley and Sons, Inc., New York.
10. Brick, J., Tubbs, E., Collins, M., Nolin, M., Cantor, D., Levin, K., and Carnes, Y. (1997). Telephone Coverage Bias and Recorded Interviews in the 1993 National Household Education Survey (NHES: 93), U.S. Department of Education, Washington, DC.
11. Callegaro, M., Yang, Y., Bhola, D., Dillman, D., and Chin, T. (2009). Response Latency as an Indicator of Optimizing in Online Questionnaires. Bulletin de Methodologie Sociologique, 103(1): 5-25.
12. Caspar, R. and Couper, M.P. (1997). Using Keystroke Files to Assess Respondent Difficulties. Proceedings of the ASA, Survey Research Methods Section, pages 239-244.
13. Conrad, F.G., Couper, M.P., Tourangeau, R., and Peytchev, A. (2006). Use and Non-use of Clarification Features in Web Surveys. Journal of Official Statistics, 22: 245-269.
14. Conrad, F.G., Schober, M., and Coiner, T. (2007). Bringing Features of Human Dialogue to Web Surveys. Applied Cognitive Psychology, 21(2): 165-187.
15. Conrad, F.G., Schober, M., and Dijkstra, W. (2004). Non-verbal Cues of Respondents' Need for Clarification in Survey Interviews. Proceedings of the ASA, Section on Survey Methods Research, pages 4934-4950.
16. Conrad, F.G., Schober, M., and Dijkstra, W. (2008). Cues of Communication Difficulty in Telephone Interviews. In Lepkowski, J.M., Tucker, C., Brick, J.M., Leeuw, E.D.D., Japec, L., Lavrakas, P.J., Link, M.W., and Sangster, R.L., editors, Advances in Telephone Survey Methodology, pages 212-230. Wiley and Sons, Inc.
17. Conrad, F.G., Tourangeau, R., Couper, M.P., and Zhang, C. (2011). Interactive Interventions in Web Surveys can Increase Response Accuracy. Paper presented at Annual Conference of the American Association for Public Opinion Research.
18. Couper, M.P. (1998). Measuring Survey Quality in a CASIC Environment. Proceedings of the Survey Research Methods Section, ASA, pages 41-49.
19. Couper, M.P. (2000). Usability Evaluation of Computer-Assisted Survey Instruments. Social Science Computer Review, 18(4): 384-396.
20. Couper, M.P. (2009). The Role of Paradata in Measuring and Reducing Measurement Error in Surveys. Paper Presented at NCRM Network for Methodological Innovation 2009: The Use of Paradata in UK Social Surveys.
21. Couper, M.P., Hansen, S., and Sadosky, S. (1997a). Evaluating Interviewer Use of CAPI Technology. In Lyberg, L., Biemer, P., Collins, M., De Leeuw, E., Dippo, C., Schwarz, N., and Trewin, D., editors, Survey Measurement and Process Quality, pages 267-285. Wiley and Sons, Inc.
22. Couper, M.P., Horm, J., and Schlegel, J. (1997b). Using Trace Files to Evaluate the National Health Interview Survey CAPI Instrument. Proceedings of Survey Research Methods Section, ASA, pages 825-829.
23. Couper, M.P., Sadosky, S., and Hansen, S. (1994). Measuring Interviewer Performance Using CAPI. Survey Research Center.
24. Couper, M.P. and Schlegel, J. (1998). Evaluating the NHIS CAPI Instrument Using Trace Files. Proceedings of the ASA, Survey Research Methods Section, pages 923-927.
25. Couper, M.P., Tourangeau, R., and Marvin, T. (2009). Taking the Audio Out of Audio-CASI. Public Opinion Quarterly, 73(2): 281-303.
26. Couper, M.P. and Kreuter, F. (2013). Using Paradata to Explore Item Level Response Times in Surveys. Journal of the Royal Statistical Society, A: Statistics in Society. 176(1): 271-286.
27. Draisma, S. and Dijkstra, W. (2004). Response Latency and (Para)Linguistic Expressions as Indicators of Response Error. In Presser, S., Rothgeb, J.M., Couper, M.P., Lessler, J.T., Martin, E., Martin, J., and Singer, E., editors, Methods for Testing and Evaluating Survey Questionnaires, pages 131-147. Wiley and Sons, Inc.
28. Duff, B., Hanmer, M.J., Park, W., and White, I.K. (2007). Good Excuses: Understanding Who Votes With An Improved Turnout Question. Public Opinion Quarterly, 71(1): 67-90.
29. Dykema, J., Lepkowski, J.M., and Blixt, S. (1997). The Effect of Interviewer and Respondent Behavior on Data Quality: Analysis of Interaction Coding in a Validation Study. In Lyberg, L., Biemer, P., Collins, M., de Leeuw, E.D., Dippo, C., Schwarz, N., and Trewin, D., editors, Survey Measurement and Process Quality, pages 287-310. Wiley and Sons, Inc., New York.
30. Fazio, R.H. (1990). A Practical Guide to the Use of Response Latency in Social Psychological Research. In Hendrick, C. and Clark, M.S., editors, Review of Personality and Social Psychology, Research Methods in Personality and Social Psychology, volume 11, pages 74-97. Sage Publications.
31. Fletcher, J.F. (2000). Two-Timing: Politics and Response Latencies in a Bilingual. Political Psychology.
32. Fowler, F. (2011). Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions. In Madans, J., Miller, K., Maitland, A., and Willis, G., editors, Question Evaluation Methods: Contributing to the Science of Data Quality, pages 7-21. Wiley and Sons, Inc.
33. Galesic, M. and Bosnjak, M. (2009). Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey. Public Opinion Quarterly, 73(2): 349-360.
34. Garbarski, D., Schaeffer, N.C., and Dykema, J. (2011). Are Interactional Behaviors exhibited when the Self-reported Health Question is asked Associated with Health Status? Social Science Research, 40(4): 1025-1036.
35. Grant, J.T., Mockabee, S.T., and Monson, J.Q. (2010). Campaign Effects on the Accessibility of Party Identification. Political Research Quarterly, 63(4): 811-821.
36. Groves, R.M. (1989). Survey Errors and Survey Costs. Wiley and Sons, Inc., New York.
37. Hansen, S., Fuchs, M., and Couper, M.P. (1997). CAI Instrument Usability Testing. Proceedings of Survey Research Methods Section, ASA, pages 1023-1028.
38. Healey, B. (2007). Drop Downs and Scroll Mice: The Effect of Response Option Format and Input Mechanism Employed on Data Quality in Web Surveys. Social Science Computer Review, 25(1): 111-128.
39. Heerwegh, D. (2002). Describing Response Behavior in Websurveys Using Client Side Paradata. Paper presented at the International Workshop on Web Surveys held at ZUMA, Mannheim, Germany, October 25, 2002.
40. Heerwegh, D. (2003). Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey. Social Science Computer Review, 21(3): 360-373.
41. Heerwegh, D. and Loosveldt, G. (2000). An Evaluation of the Effect of Response Formats on Data Quality in Web Surveys. Social Science Computer Review, 20(4): 471-484.
42. Herzog, A.R., Rodgers, W.L., and Kulka, R.A. (1988). Interviewing Older Adults. Mode Comparison Using Data from a Face-to-Face Survey and a Telephone Resurvey. Public Opinion Quarterly, 52(1): 84-99.
43. Hill, D.H. (1993). Response and Sequencing Errors in Surveys: A Discrete Contagious Regression Analysis. Journal of the ASA, 88(423): 775-781.
44. Huckfeldt, R., Levine, J., Morgan, W., and Sprague, J. (1998). Election Campaigns, Social Communication, and the Accessibility of Perceived Discussant Preference. Political Behavior, 20(4): 263-294.
45. Huckfeldt, R., Levine, J., Morgan, W., and Sprague, J. (1999). Accessibility and the Political Utility of Partisan and Ideological Orientations. American Journal of Political Science, 43(3): 888-911.
46. Huckfeldt, R. and Sprague, J. (2000). Political Consequences of Inconsistency: The Accessibility and Stability of Abortion Attitudes. Political Psychology, pages 57-79.
47. Hurtado, A. (1994). Does Similarity Breed Respect: Interviewer Evaluations of Mexican-Descent Respondents in a Bilingual Survey. Public Opinion Quarterly, 58(1): 77-95.
48. Jans, M.E. (2010). Verbal Paradata and Survey Error: Respondent Speech, Voice, and Question-Answering Behavior Can Predict Income Item Nonresponse. PhD thesis, University of Michigan, Ann Arbor, MI.
49. Johnson, M. (2004). Timepieces: Components of Survey Question Response Latencies. Political Psychology, 25(5): 679-702.
50. Johnson, T.P., Cho, Y.I.K., Holbrook, A.L., O'Rourke, D., Warnecke, R.B., and Chavez, N. (2006). Cultural Variability in the Effects of Question Design Features on Respondent Comprehension of Health Surveys. Annals of Epidemiology, 16(9): 661-668.
51. Kaminska, O., McCutcheon, A.L., and Billiet, J. (2010). Satisficing Among Reluctant Respondents in a Cross-National Context. Public Opinion Quarterly, 74(5): 956-984.
52. Krosnick, J.A. (1999). Survey Research. Annual Review of Psychology, 50(3): 537-567.
53. Krosnick, J.A. (1991). Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Applied Cognitive Psychology, 5(3): 213-236.
54. Lavine, H., Borgida, E., and Sullivan, J.L. (2000). On the Relationship between Attitude Involvement and Attitude Accessibility: Toward a Cognitive-Motivational Model of Political Information Processing. Political Psychology, 21(1): 81-106.
55. Lenzner, T., Kaczmirek, L., and Lenzner, A. (2010). Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment. Applied Cognitive Psychology, 24(7): 1003-1020.
56. Lepkowski, J.M., Couper, M.P., Hansen, S.E., Landers, W., McGonagle, K.A., and Shlegel, J. (1998). CAPI Instrument Evaluation: Behavior Coding, Trace Files and Usability Methods. Proceedings of Survey Research Methods Section, ASA, pages 917-922.
57. Lepkowski, J.M., Siu, V., and Fisher, J. (2000). Event History Analysis of Interviewer and Respondent Survey Behavior. Metodoloski Zvezki, 15: 3-20.
58. Lynn, P. and Nicolaas, G. (2010). Making Good Use of Survey Paradata. Survey Practice.
59. Malhotra, N. (2008). Completion Time and Response Order Effects in Web Surveys. Public Opinion Quarterly, 72(5): 914-934.
60. Mathiowetz, N.A. (1998). Respondent Expressions of Uncertainty: Data Source for Imputation. Public Opinion Quarterly, 62(1):47.
61. Mockovak, W. and Powers, R. (2008). The Use of Paradata for Evaluating Interviewer Training and Performance. In Proceedings of the Section on Survey Research Methods of the ASA, pages 1386-1389.
62. Mulligan, K., Grant, T., Monson, Q., and Mockabee, S. (2003). Response Latency Methodology for Survey Research: Measurement and Modeling Strategies. Political Analysis, 11(3): 289-301.
63. Olson, K. and Peytchev, A. (2007). Effect of Interviewer Experience on Interview Pace and Interviewer Attitudes. Public Opinion Quarterly, 71(2): 273-286.
64. Ongena, Y.P. and Dijkstra, W. (2006). Methods of Behavior Coding of Survey Interviews. Journal of Official Statistics, 22(3): 419-451.
65. Ratcliff, R. (1993). Methods for Dealing with Reaction Time Outliers. Psychological Bulletin, 114(3): 510-532.
66. Raudenbush, S. and Bryk, A. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods. 2nd edition. Sage Publications.
67. Savage, S.J. and Waldman, D.M. (2008). Learning and Fatigue during Choice Experiments: A Comparison of Online and Mail Survey Modes. Journal of Applied Econometrics, 23(3): 351-371.
68. Schaeffer, N.C. and Dykema, J. (2011). Response 1 to Fowler's Chapter: Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions. In Madans, J., Miller, K., A., M., and Willis, G., editors, Question Evaluation Methods: Contributing to the Science of Data Quality, pages 23-39. Wiley and Sons, Inc.
69. Smyth, J.D., Dillman, D.A., Christian, L.M., and Stern, M.J. (2006). Comparing Check-all and Forced-choice Question Formats in Web Surveys. Public Opinion Quarterly, 70(1): 66-77.
70. Stern, M.J. (2008). The Use of Client-Side Paradata in Analyzing the Effects of Visual Layout on Changing Responses in Web Surveys. Field Methods, 20(4): 377-398.
71. Stieger, S. and Reips, U-D. (2010). What Are Participants Doing While Filling in an Online Questionnaire: A Paradata Collection Tool and an Empirical Study. Computers in Human Behavior, 26(6): 1488-1495.
72. Sykes, W. and Collins, M. (1992). Anatomy of the Survey Interview. Journal of Official Statistics, 8(3): 277-291.
73. Tourangeau, R., Couper, M.P., and Conrad, F.G. (2004). Spacing, Position, and Order-Interpretive Heuristics for Visual Features of Survey Questions. Public Opinion Quarterly, 68(3): 368-393.
74. Tourangeau, R., Rips, L.J., and Rasinski, K.A. (2000). The Psychology of Survey Response. Cambridge University Press.
75. Yan, T. and Tourangeau, R. (2008). Fast Times and Easy Questions: The Effects of Age, Experience and Question Complexity on Web Survey Response Times. Applied Cognitive Psychology, 22(1): 51-68.