Detecting and understanding interviewer effects on survey data by using a cross-classified mixed effects location–scale model

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 180, Issue 2, 2017, Pages 551-568

  Brunton Smith, Ian ^a   Sturgis, Patrick ^b   Leckie, George ^c  

^a UNIVERSITY OF SURREY   (United Kingdom)

^b UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^c UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

Interviewer effect; Measurement error; Mixed effects location scale model; Stat JR software; Understanding society

Indexed keywords

EID: 84969933379     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/rssa.12205     Document Type: Article

References (48)

1. Anderson, B. A., Silver, B. D. and Abramson, P. R. (1988) The effects of race of the interviewer on measures of electoral participation by blacks in SRC national election studies. Publ. Opin. Q., 52, 53–83.
2. Bailar, B. A., Bailey, L. and Stevens, J. (1977) Measures of interviewer bias and variance. J. Marktng Res., 24, 337–343.
3. Biemer, P. P. and Stokes, S. L. (1985) Optimal design of interviewer variance experiments in complex surveys. J. Am. Statist. Ass., 80, 158–166.
4. Brunton-Smith, I. and Sturgis, P. (2011) Do neighbourhoods generate fear of crime?: an empirical test using the British Crime Survey. Criminology, 49, 331–369.
5. Brunton-Smith, I., Sturgis, P. and Williams, J. (2012) Is success in obtaining contact and cooperation correlated with the magnitude of interviewer variance? Publ. Opin. Q., 76, 265–286.
6. Burton, J., Knies, G. and Al Baghal, T. (2014) Understanding society: interviewer survey 2014. User Guide v1.1. Colchester: UK Household Longitudinal Survey.
7. Cannell, C. F., Miller, P. V. and Oksenberg, L. (1981) Research on interviewing techniques. In Sociological Methodology (ed. S. Leinhardt), pp. 389–437. San Francisco: Jossey-Bass.
8. Charlton, C. M. J., Michaelides, D. T., Parker, R. M. A., Cameron, B., Szmaragd, C., Yang, H., Zhang, Z., Frazer, A. J., Goldstein, H., Jones, K., Leckie, G., Moreau, L. and Browne, W. J. (2013) Stat-JR version 1.0. Centre for Multilevel Modelling, University of Bristol and Electronics and Computer Science, University of Southampton. (Available from http://www.bristol.ac.uk/cmm/software/statjr/.)
9. Collins, M. (1980) Interviewer variability: a review of the problem. J. Markt Res. Soc., 22, 77–95.
10. Davis, P. and Scott, A. (1995) The effect of interviewer variance on domain comparisons. Surv. Methodol., 21, 99–106.
11. Durrant, G. B., Groves, R. M., Staetsky, L. and Steele, F. (2010) Effects of interviewer attitudes and behaviors on refusal in household surveys. Publ. Opin. Q., 74, 1–36.
12. Fellegi, I. P. (1964) Response variance and its estimation. J. Am. Statist. Ass., 59, 1016–1041.
13. Fellegi, I. P. (1974) An improved method of estimating the correlated response variance. J. Am. Statist. Ass., 69, 496–501.
14. Finkel, S. E., Guterbock, T. M. and Borg, M. J. (1991) Race-of-interviewer effects in a preelection poll: Virginia, 1989. Publ. Opin. Q., 55, 313–330.
15. Gelman, A. (2007) Struggles with survey weighting and regression modelling. Statist. Sci., 22, 153–164.
16. Goldberg, L. (1990) An alternative “description of personality”: the big-Five factor structure. J. Personlty Socl Psychol., 59, 1216–1229.
17. Goldstein, H. (2011) Multilevel Statistical Models. Chichester: Wiley.
18. Hansen, M. H., Hurwitz, W. N. and Bershad, M. A. (1961) Measurement errors in censuses and surveys. Bull. Int. Statist. Inst., 38, 359–374.
19. Hedeker, D., Mermelstein, R. J. and Demirtas, H. (2008) An application of a mixed-effects location scale model for analysis of ecological momentary assessment (EMA) data. Biometrics, 64, 627–634.
20. Hedeker, D. and Nordgren, R. (2013) MIXREGLS: a program for mixed-effects location scale analysis. J. Statist. Softwr., 52, 1–38.
21. Hirschman, A. O. (1964) The paternity of an index. Am. Econ. Rev., 54, 761.
22. Hox, J. J. (1994) Hierarchical regression models for interviewer and respondent effects. Sociol. Meth. Res., 22, 300–318.
23. Hox, J. J., de Leuww, E. D. and Kreft, I. I. G. (1991) The effect of interviewer and respondent characteristics on the quality of survey data: a multilevel model. In Measurement Errors in Surveys (eds P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz and S. Sudman), pp. 439–462. New York: Wiley.
24. Huddy, L., Billig, J., Bracciodieta, J., Hoeffer, L., Moynihan, P. and Puglian, P. (1997) The effect of interviewer gender on the survey response. Polit. Behav., 19, 197–220.
25. Hughes, A., Chromy, J., Giacoletti, K. and Odom, D. (2002) Impact of interviewer experience on respondent reports of substance use. In Redesigning an Ongoing National Household Survey: Methodological Issues, Substance Abuse and Mental Health Services Administration (eds J. Gfroerer, J. Eyerman and J. Chromy), pp. 161–184. Rockville: Office of Applied Studies.
26. Jäckle, A., Lynn, P., Sinibaldi, J. and Tipping, S. (2013) The effect of interviewer experience, attitudes, personality and skills on respondent cooperation with face-to-face surveys. Surv. Res. Meth., 7, 1–15.
27. Kane, E. W. and Macaulay, L. J. (1993) Interviewer gender and gender attitudes. Publ. Opin. Q., 57, 1–28.
28. Kish, L. (1962) Studies of interviewer variance for attitudinal variables. J. Am. Statist. Ass., 57, 92–115.
29. Knies, G. (2014) Understanding society: the UK Household Longitudinal Study Waves 1-4.User Manual. Institute for Social and Economic Research, Colchester.
30. Leckie, G., French, R., Charlton, C. and Browne, W. (2014) Modeling heterogeneous variance-covariance components in two-level models. J. Educ. Behav. Statist., 39, 307–332.
31. Lunn, D., Jackson, C., Best, N., Thomas, A. and Spiegelhalter, D.(2012) The BUGS Book: a Practical Introduction to Bayesian Analysis. Boca Raton: Chapman and Hall–CRC.
32. Mangione, T., Fowler, F. J. and Louis, T. A. (1992) Question characteristics and interviewer effects. J. Off. Statist., 8, 293–307.
33. Martin, D. (2001) Geography for the 2001 Census in England and Wales. London: Office for National Statistics.
34. O'Muircheartaigh, C. (1976) Response errors in an attitudinal sample survey. Qual. Quant., 26, 97–115.
35. O'Muircheartaigh, C. and Campanelli, P. (1998) The relative impact of interviewer effects and sample design effects on survey precision. J. R. Statist. Soc. A, 161, 63–77.
36. Pickery, J., Loosveldt, G. and Carton, A. (2001) The effects of interviewer and respondent characteristics on response behaviour in panel surveys: a multilevel approach. Sociol. Meth. Res., 29, 509–523.
37. Rao, J. N. K., Verret, F. and Hidiroglou, M. A. (2013) A weighted composite likelihood approach to inference for two-level models from survey data. Surv. Methodol., 39, 263–282.
38. Schaeffer, N. C. (1980) Evaluating race of interviewer effects in a national survey. Sociol. Meth. Res., 8, 400–419.
39. Schaeffer, N. C., Dykema, J. and Maynard, D. W. (2010) Interviewers and interviewing. In Handbook of Survey Research, 2nd edn (eds P. V. Marsden and J. D. Wright). Bingley: Emerald.
40. Schnell, R. and Kreuter, F. (2005) Separating interviewer and sampling point effects. J. Off. Statist., 21, 389–410.
41. Scott, A. and Jessop, C. (2013) UK Household Longitudinal Study (UKHLS) Wave 3. Technical Report. NatCen, London.
42. Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002) Bayesian measures of model complexity and fit (with discussion). J. R. Statist. Soc. B, 64, 583–639.
43. Sturgis, P. and Luff, R. (2015) Audio-recording of open-ended survey questions: a solution to the problem of interviewer transcription In Survey Measurements: Techniques, Data Quality and Sources of Error (ed. U. Engel), pp. 42–57. Chicago: University of Chicago Press.
44. van Tilburg, T. (1998) Interviewer effects in the measurement of personal network size: a nonexperimental study. Sociol. Meth. Res., 26, 300–328.
45. Turner, M., Sturgis, P., Martin, D. and Skinner, C. (2014) Can interviewer personality, attitudes and experience explain the design effect in face-to-face surveys? In Improving Survey Methods: Lessons from Recent Research (eds U. Engel, B. Jann, P. Lynn, A. Scherpenzeel and P. Sturgis). Abingdon: Routledge.
46. West, B. T. and Elliott, M. R. (2014) Frequentist and Bayesian approaches for comparing interviewer variance components in two groups of survey interviewers. Surv. Methodol., 40, 163–188.
47. West, B. T., Kreuter, F. and Jaenichen, U. (2013) Interviewer effects in face-to-face surveys: a function of sampling, measurement error or nonresponse? J. Off. Statist., 29, 277–297.
48. West, B. T. and Olson, K. (2010) How much of interviewer variance is really nonresponse error variance? Publ. Opin. Q., 74, 1004–1026.