Identifying sources of error in cross-national questionnaires: Application of an error source typology to cognitive interview data

Journal of Official Statistics

Volumn 27, Issue 4, 2011, Pages 569-599

  Fitzgerald, Rory ^a   Widdop, Sally ^a   Gray, Michelle ^a   Collins, Debbie ^b  

^a CITY UNIVERSITY   (United Kingdom)

^b NATIONAL CENTRE FOR SOCIAL RESEARCH   (United Kingdom)

Author keywords

Cognitive interviewing; Cross National error source typology (CNEST); Cross national research; European Social Survey; Questionnaire design; Translation

Indexed keywords

EID: 84857098848     PISSN: 0282423X     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (45)

1. National Error Source Typology (CNEST); translation; European Social Survey.
2. Beatty, P.C. (2004). The Dynamics of Cognitive Interviewing. Methods for Testing and Evaluating Survey Questionnaires, S. Presser, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Martin, and E. Singer (eds). New Jersey: Wiley, 45-66.
3. Beatty, P.C. and Willis, G.B. (2007). Research Synthesis: the Practice of Cognitive Interviewing. Public Opinion Quarterly, 71, 287-311.
4. Behr, D., Harkness, J., Fitzgerald, R., and Widdop, S. (2008). ESS Round 4 Questionnaire and Translation Queries. London: European Social Survey, Centre for Comparative Social Surveys, City University.
5. Carrasco, L. (2003). The American Community Survey (ACS) en Español: Using Cognitive Interviews to Test the Functional Equivalence of Questionnaire Translations. Washington, DC: Statistical Research Division, U.S. Census Bureau.
6. Collins, D. (2003). Pretesting Survey Instruments: An Overview of Cognitive Methods. Quality of Life Research, 12, 219-227.
7. Collins, D. (2007). Analysing and Interpreting Cognitive Interview Data: A Qualitative Approach. Proceedings of the 6th Questionnaire Evaluation Standard for Testing Conference. Ottawa: Statistics Canada.
8. Conrad, F.G. and Blair, J. (1996). From Impressions to Data: Increasing the Objectivity of Cognitive Interviews. Proceedings of the American Statistical Association, Section on Survey Research Methods, 1, 1-9. Alexandria, VA: American Statistical Association.
9. Conrad, F.G. and Blair, J. (2009). Sources of Error in Cognitive Interviews. Public Opinion Quarterly, 73, 32-55.
10. DeMaio, T.J. and Landreth, A. (2004). Cognitive Interviews: Do Different Methods Produce Different Results?. Methods for Testing and Evaluating Survey Questionnaires, S. Presser, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Martin, and E. Singer (eds). Hoboken, NJ: John Wiley and Sons, 89-108.
11. Enerstvedt, R.G. (1989). The Problem of Validity in Social Science. Issues of Validity in Qualitative Research, S. Kvale (ed.). Sweden: Studentlitteratur.
12. Fitzgerald, R., Widdop, S., Gray, M., and Collins, D. (2009). Testing for Equivalence Using Cross-national Cognitive Interviewing. London: Centre for Comparative Social Surveys, City University Working Paper 2009/01.
13. Forsyth, B.H., Stapleton-Kudela, M., Levin, K., Lawrence, D., and Willis, G.B. (2007). Methods for Translating an English-Language Survey Questionnaire on Tobacco Use into Mandarin, Cantonese, Korean and Vietnamese. Field Methods, 19, 264-283.
14. Gerber, E. (1999). The View from Anthropology: Ethnography and the Cognitive Interview. Cognition and Survey Research, M. Sirken, D. Herrmann, S. Schechter, N. Schwarz, J. Tanur, and R. Tourangeau (eds). New York: Wiley.
15. Gerber, E. and Wellens, T. (1997). Perspectives on Pre-testing: "Cognition" in the Cognitive Interview? Bulletin de Methodologie Sociologique, 55, 18-39.
16. Goerman, P. (2006). Cognitive Interview Methodology Revisited: Development of Best Practices for Pretesting Spanish Survey Instruments. Paper presented at the Meeting of the American Association for Public Opinion Research, Montreal, Canada.
17. Goerman, P.L. and Caspar, R. (2007). A New Methodology for the Cognitive Testing of Translated Materials: Testing the Source Version as a Basis for Comparison. Paper presented at the American Association for Public Opinion Research Conference, May 17-20, Anaheim, CA and submitted to 2007 JSM Proceedings, Statistical Computing Section [CD-ROM], Alexandria, VA: American Statistical Association, 3949-3956.
18. Guba, E.G. and Lincoln, Y.S. (1994). Competing Paradigms in Qualitative Research. The Handbook of Qualitative Research, N.K. Denzin and Y.S. Lincoln (eds). Thousand Oaks, CA: Sage.
19. Harkness, J. (2007). Improving the Quality of Translations. Measuring Attitudes Crossnationally: Lessons from the European Social Survey, R. Jowell, C. Roberts, R. Fitzgerald, and G. Eva (eds). London: Sage.
20. Harkness, J., De Vijer, F., and Johnson, T. (2003). Questionnaire Design in Comparative Research. Cross-cultural Survey Methods, J. Harkness, F. De Vijer, and P. Mohler (eds). Hoboken, NJ: John Wiley and Sons.
21. Jowell, R. and Eva, G. (2009). Happiness is Not Enough: Cognitive Judgements as Indicators of National Well-being. Social Indicators Research, 91, 317-328.
22. Knafl, K., Deatrick, J., Gallo, A., Holcombe, G., Bakitas, M., Dixon, J., and Grey, M. (2007). The Analysis and Interpretation of Cognitive Interviews for Instrument Development. Research in Nursing and Health, 30, 224-234.
23. Kudela, M.S., Levin, K., Tseng, M., Hum, M., Lee, S., Wong, C., McNutt, S., and Lawrence, D. (2004). Tobacco Use Cessation Supplement to the Current Population Survey Chinese, Korean, and Vietnamese Translations: Results of Cognitive Testing. Final Report submitted to the National Cancer Institute, Rockville, MD.
24. Kudela, M.S., Forsyth, B.H., Levin, K., Lawrence, D., and Willis, G.B. (2006). Cognitive Interviewing versus Behaviour Coding. Paper presented at the American Association of Public Opinion Research, Montreal, Canada.
25. Levin, K., Willis, G.B., Forsyth, B.H., Norberg, A., Kudela, M.S., Stark, D., and Thompson, F.E. (2009). Using Cognitive Interviews to Evaluate the Spanish-Language Translation of a Dietary Questionnaire. Survey Research Methods, 3, 13-25.
26. Madill, A., Jordan, A., and Shirley, C. (2000). Objectivity and Reliability in Qualitative Analysis: Realist, Contextualist and Radical Constructionist Epistemologies. British Journal of Psychology, 91, 1-20.
27. Miles, M.B. and Huberman, A.M. (1994). Qualitative Data Analysis: An Expanded Sourcebook (Second Edition). Newbury Park; CA: Sage Publications.
28. Miller, K., Willis, G.B., Eason, C., Moses, L., and Canfield, B. (2005). Interpreting Results of Cross-Cultural Interviews: A Mixed Methods Approach. Methodological Aspects in Cross National Research, J.H.P. Hoffmeyer-Zlotnik and J.A. Harkness (eds). ZUMA, Mannheim, 79-92. Available at http://www.gesis.org/fileadmin/upload/forschung/publikationen/zeitschriften/zuma_nachrichten_spezial/znspezial11.pdf?download=true
29. Miller, K., Fitzgerald, R., Caspar, R., Dimov, M., Gray, M., Nunes, C., Padilla, J.L., Pruefer, P., Schöbi, N., Schoua-Glusberg, A., Widdop, S., and Willson, S. (2008). Design and Analysis of Cognitive Interviews for Cross-National Testing. Paper presented at the International Conference on Survey Methods in Multinational, Multiregional, and Multicultural Contexts (3MC), Berlin, 25-29 June, Germany.
30. Nápoles-Springer, A.M., Santoyo-Olsson, J., O'Brien, H., and Stewart, A.L. (2006). Using Cognitive Interviews to Develop Surveys in Diverse Populations. Medical Care, 44 (Suppl. 3), S21-S30.
31. Patton, M.Q. (2002). Qualitative Research and Evaluation Methods (Third Edition). Thousand Oaks, CA: Sage.
32. Ritchie, J. and Spencer, L. (1994). Qualitative Data Analysis for Applied Policy Research. Analysing Qualitative Data, A. Bryman and R. Burgess (eds). London: Routledge.
33. Saris, W. and Gallhofer, I. (2007). Can Questions Travel Successfully? Measuring Attitudes Cross-nationally: Lessons from the European Social Survey, R. Jowell, C. Roberts, R. Fitzgerald, and G. Eva (eds). London: Sage.
34. Schoua-Glusberg, A. (2006). Eliciting Education Level in Spanish Interviews. Paper presented to the American Association of Public Opinion Research, Montreal, Canada.
35. Spencer, L., Ritchie, J., and O'Connor, W. (2003). Analysis: Practices, Principles and Processes. Qualitative Research Practice, J. Ritchie and J. Lewis (eds). London: Sage, 199-218.
36. Tucker, C. (1997). Methodological Issues Surrounding the Application of Cognitive Psychology in Survey Research. Bulletin de Methodologie Sociologique, 11, 67-92.
37. Widdop, S.E. (2007). European Social Survey Codebook: Round 1 and Round 2. London: Centre for Comparative Social Surveys, City University.
38. Willis, G.B. (2004). Cognitive Interviewing Revisited: A Useful Technique, in Theory? Methods for Testing and Evaluating Survey Questionnaires, S. Presser, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Martin, and E. Singer (eds). Hoboken, NJ: John Wiley and Sons, 23-44.
39. Willis, G.B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.
40. Willis, G.B., DeMaio, T., and Harris-Kojetin, B. (1999). Is the Bandwagon Headed to the Methodological Promised Land? Evaluating the Validity of Cognitive Interviewing Techniques. Cognition and Survey Research, M. Sirken, D. Herrmann, S. Schechter, N. Schwarz, J. Tanur, and R. Tourangeau (eds). New York: Wiley, 133-153.
41. Willis, G.B., Lawrence, D., Kudela, M., and Levin, K. (2005a). The Use of Cognitive Interviewing to Study Cultural Variation in Survey Response. Paper presented to the Questionnaire Evaluation Standards Workshop (QUEST) on Questionnaire Design and Testing, Heerlen, the Netherlands, 19-21 April.
42. Willis, G.B., Lawrence, D., Thompson, F.W., Kudela, M., Levin, K., and Miller, K. (2005b). The Use of Cognitive Interviewing to Evaluate Translated Survey Questions: Lessons Learned. Proceedings of the Federal Committee on Statistical Methodology Research Conference. Arlington, VA.
43. Willis, G.B., Lawrence, D., Hartman, A., Kudela, M.S., Levin, K., and Forsyth, B. (2007). Translation of a Tobacco Survey into Spanish and Asian Languages: The Tobacco Use Supplement to the Current Population Survey. Nicotine & Tobacco Research, 10, 1075-1084.
44. Willis, G.B. and Zahnd, E. (2007). Questionnaire Design from a Cross-cultural Perspective: An Empirical Investigation of Koreans and Non-Koreans. Journal of Health Care for the Poor and Underserved, 18, 197-217.
45. Willson, S. and Miller, K. (2005). Using Qualitative Methods to Improve Measurement Error in Quantitative Data. Paper presented at the annual meeting of the American Sociological Association, Philadelphia, August 12.