Validating sensitive questions: A comparison of survey and register data

Journal of Official Statistics

Volumn 31, Issue 1, 2015, Pages 31-59

  Kirchner, Antje ^a  

^a INSTITUTE FOR EMPLOYMENT RESEARCH IAB   (Germany)

Author keywords

Randomized response technique; Social desirability; Unemployment benefit II; Validation data; Welfare receipt

Indexed keywords

EID: 84925152144     PISSN: 0282423X     EISSN: 20017367     Source Type: Journal    
DOI: 10.1515/JOS-2015-0002     Document Type: Article

References (75)

1. AAPOR - The American Association for Public Opinion Research 2011. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 7th Ed. Lanexo: AAPOR.
2. Angrist, J.D., G.W. Imbens, and D.B. Rubin. 1996. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association 91: 444–455. doi: http://dx.doi.org/10.1080/01621459.1996.10476902.
3. Aquilino, W.S. 1994. “Interview Mode Effects in Surveys of Drug and Alcohol Use: A Field Experiment.” Public Opinion Quarterly 58: 210–240. DOI: http://dx.doi.org/10.1086/269419.
4. Bartus, T. 2005. “Estimation of Marginal Effects Using Margeff.” The Stata Journal 5: 309–329.
5. Biemer, P.P. 2010. “Overview of Design Issues: Total Survey Error.” In Handbook of Survey Research, edited by P.P. Biemer, P.V. Marsden, and J.D. Wright, 27–57. Bingley: Emerald Publishing Group Limited.
6. Boeije, H. and G.J.L.M. Lensvelt-Mulders. 2002. “Honest by Chance: A Qualitative Interview Study to Clarify Respondents’ (Non-)compliance with Computer-Assisted-Randomized Response.” Bulletin de Methodologie Sociologique 75: 24–39.
7. Boruch, R.F. 1971. “Assuring Confidentiality of Responses in Social Research: A Note on Strategies.” The American Sociologist 6: 308–311.
8. Bradburn, N., S. Sudman, and B. Wansink. 2004. Asking Questions. Revised Edition. San Francisco: Jossey-Bass.
9. Bullock, H.E. 2006. “Attributions for Poverty: A Comparison of Middle-Class and Welfare Recipient Attitudes.” Journal of Applied Social Psychology 29: 2059–2082. DOI: http://dx.doi.org/10.1111/j.1559-1816.1999.tb02295.x.
10. Böckenholt, U., S. Barlas, and P.G.M. van der Heijden. 2009. “Do Randomized-Response Designs Eliminate Response Biases? An Empirical Study of Non-Compliance Behavior.” Journal of Applied Econometrics 24: 377–392. DOI: http://dx.doi.org/10.1002/jae.1052.
11. Böckenholt, U. and P.G.M. van der Heijden. 2007. “Item Randomized-Response Models for Measuring Noncompliance: Risk-Return Perceptions, Social Influences, and Self-Protective Responses.” Psychometrika 72: 245–262. DOI: http://dx.doi.org/10.1007/s11336-005-1495-y.
12. Cialdini, R.B. 2007. “Descriptive Social Norms as Underappreciated Sources of Social Control.” Psychometrika 72: 263–268. DOI: http://dx.doi.org/10.1007/s11336-006-1560-6.
13. Clark, S.J. and R.A. Desharnais. 1998. “Honest Answers to Embarrassing Questions: Detecting Cheating in the Randomized Response Model.” Psychological Methods 3: 160–168. DOI: http://dx.doi.org/10.1037/1082-989X.3.2.160.
14. Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences. Vol. 2. Hillshale, NJ: Erlbaum.
15. Coutts, E. and B. Jann. 2011. “Sensitive Questions in Online Surveys: Experimental Results for the Randomized Response Technique (RRT) and the Unmatched CountTechnique (UCT).” Sociological Methods & Research 40: 169–193. DOI: http://dx.doi.org/10.1177/0049124110390768.
16. Coutts, E., B. Jann, I. Krumpal, and A.-F. Naher. 2011. “Plagiarism in Student Papers: Prevalence Estimates Using Special Techniques for Sensitive Questions.” Journal of Economics and Statistics (Jahrbücher für Nationalökonomie und Statistik) 231: 749–760.
17. De Jong, M.G., R. Pieters, and S. Stremersch. 2012. “Analysis of Sensitive Questions Across Cultures: An Application of Multigroup Item Randomized Response Theory to Sexual Attitudes and Behavior.” Journal of Personality and Social Psychology 19: 153–176. DOI: http://dx.doi.org/10.1037/a0029394.
18. De Leeuw, E.D. and J. van der Zouwen. 1988. “Data Quality in Telephone and Face to Face Surveys: A Comparative Metaanalysis.” In Telephone Survey Methodology, edited by R.M. Groves, P.P. Biemer, L.E. Lyberg, J.T. Massey, W.L. Nicholls, and J. Waksberg, 283–299. New York: John Wiley & Sons, Inc.
19. De Schrijver, A. 2012. “Sample Survey on Sensitive Topics: Investigating Respondents’ Understanding and Trust in Alternative Versions of the Randomized Response Technique.” Journal of Research Practice 8: 1–17.
20. Fidler, D.S. and R.E. Kleinknecht. 1977. “Randomized Response versus Direct Questioning: Two Data-Collection Methods for Sensitive Information.” Psychological Bulletin 84: 1045–1049. doi: http://dx.doi.org/10.1037/0033-2909.84.5.1045.
21. Fox, J.A. and P.E. Tracy. 1986. Randomized Response: A Method for Sensitive Surveys. Beverly Hills: Sage Publications.
22. Ganzeboom, H.B.G., P.M. De Graaf, and D.J. Treiman. 1992. “A Standard International Socio-Economic Index of Occupational Status.” Social Science Research 21: 1–56. DOI: http://dx.doi.org/10.1016/0049-089X(92)90017-B.
23. Greenberg, B.G., A.L.A. Abul-Ela, W.R. Simmons, and D.G. Horvitz. 1969. “The Unrelated Question Randomized Response Model: Theoretical Framework.” Journal of the American Statistical Association 64: 520–539. DOI: http://dx.doi.org/10.1080/01621459.1969.10500991.
24. Greenberg, B.G., R.R. Kuebler Jr., J.R. Abernathy, and D.G.G. Horvitz. 1971. “Application of the Randomized Response Technique in Obtaining Quantitative Data.” Journal of the American Statistical Association 66: 243–250. DOI: http://dx.doi.org/10.1080/01621459.1971.10482248.
25. Groves, R.M. 2004 [1989]. Survey Error and Survey Costs. Hoboken: Wiley & Sons.
26. Groves, R.M., F.J. Fowler, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2009. Survey Methodology. Hoboken: Wiley & Sons.
27. Hausman, J. 2001. “Mismeasured Variables in Econometric Analysis: Problems From the Right and Problems from the Left.” The Journal of Economic Perspectives 15: 57–67.
28. Hendrickx, J. 2002. “ISKO: Stata Module to Recode 4 Digit ISCO-88 Occupational Codes, Statistical Software Components s425802.” Boston College Department of Economics. revised 20 Oct 2004. Available at: https://ideas.repec.org/c/boc/bocode/s425802.html (accessed February 14, 2015).
29. Holbrook, A.L., M.C. Green, and J.A. Krosnick. 2003. “Telephone Versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires. Comparisons of Respondent Satisficing and Social Desirability Response Bias.” Public Opinion Quarterly 67: 79–125. DOI: http://dx.doi.org/10.1086/346010.
30. Holbrook, A.L. and J.A. Krosnick. 2010. “Measuring Voter Turnout by Using the Randomized Response Technique: Evidence Calling into Question the Method’s Validity.” Public Opinion Quarterly 74: 328–343. DOI: http://dx.doi.org/10.1093/poq/nfq012.
31. Holtgraves, T. 2004. “Social Desirability and Self-Reports: Testing Models of Socially Desirable Responding.” Personality and Social Psychology Bulletin 30: 161–172. DOI: http://dx.doi.org/10.1177/0146167203259930.
32. Horvitz, D.G., B.V. Shah, and W.R. Simmons. 1967. “The Unrelated Question Randomized Response Model.” In Proceedings of the Social Statistics Section. American Statistical Association, 65–72.
33. Höglinger, M., B. Jann, and A. Diekmann. 2014. Sensitive Questions in Online Surveys: An Experimental Evaluation of the Randomized Response Technique and the Crosswise Model. University of Bern Social Science Working Paper No. 9, 1–51. Available at: ftp://repec.sowi.unibe.ch/files/wp9/hoeglinger-jann-diekmann-2014.pdf (accessed September 17, 2014).
34. Jacobebbinghaus, P. and S. Seth. 2007. “The German Integrated Employment Biographies Sample IEBS.” Schmollers Jahrbuch 127: 335–342.
35. Jann, B. 2011. “Rrlogit: Stata module to estimate logistic regression for randomized response data.” Statistical SoftwareComponents, Boston College Department of Economics. Available at: https://ideas.repec.org/c/boc/bocode/s456203.html (accessed February 14,2015).
36. Jann, B., J. Jerke and I. Krumpal. 2012. “Asking Sensitive Questions Using the Crosswise Model. An Experimental Survey Measuring Plagiarism.” Public Opinion Quarterly 71: 32–49. DOI: http://dx.doi.org/10.1093/poq/nfr036.
37. Kirchner, A. 2014. Techniques for Asking Sensitive Question in Labor Market Surveys. IAB-Bibliothek Dissertationen, 348. Bielefeld: Bertelsmann. Available at: http://edoc.ub.uni-muenchen.de/17192/1/Kirchner_Antje.pdf ?(accessed February 14, 2015).
38. Kirchner, A., I. Krumpal, M. Trappmann, and H. von Hermanni. 2013. “Messung und Erklärung von Schwarzarbeit in Deutschland – Eine empirische Befragungsstudie unter besonderer Berücksichtigung des Problems der sozialen Erwünschtheit.” Zeitschrift für Soziologie 42: 291–314.
39. Korndörfer, M., I. Krumpal, and S.C. Schmukle. 2014. “Measuring and Explaining Tax Evasion: Improving Self-Reports Using the Crosswise Model.” Journal of Economic Psychology 45: 18–32. DOI: http://dx.doi.org/10.1016/j.joep.2014.08.001.
40. Kreuter, F., G. Müller, and M. Trappmann. 2010. “Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data.” Public Opinion Quarterly 74: 880–906. DOI: http://dx.doi.org/10.1093/poq/nfq060.
41. Kreuter, F., G. Müller, and M. Trappmann. 2014. “A Note on Mechanisms Leading to Lower Data Quality of Late or Reluctant Respondents.” Sociological Methods and Research 43: 452-464. DOI: http://dx.doi.org/10.1177/0049124113508094.
42. Krosnick, J.A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5: 213–236. DOI: http://dx.doi.org/10.1002/acp.2350050305.
43. Krumpal, I. 2012. “Estimating the Prevalence of Xenophobia and Anti-Semitism in Germany: A Comparison of Randomized Response and Direct Questioning.” Social Science Research 41: 1387–1403. DOI: http://dx.doi.org/10.1016/j.ssresearch.2012.05.015.
44. Kuk, A.Y.C. 1990. “Asking Sensitive Questions Indirectly.” Biometrika 77: 436–438. DOI: http://dx.doi.org/10.1093/biomet/77.2.436.
45. Lamb, C.W. and D.E. Stem. 1978. “An Empirical Validation of the Randomized Response Technique.” Journal of Marketing Research 15: 616–621.
46. Landsheer, J.A., P.G.M. van der Heijden, and G. van Gils. 1999. “Trust and Understanding. Two Psychological Aspects of Randomized Response. A Study of a Method for Improving the Estimate of Social Security Fraud.” Quality & Quantity 33: 1–12. DOI: http://dx.doi.org/10.1023/A:1004361819974.
47. Lara, D., S.G. García, C. Ellertson, C. Camlin, and J. Suárez. 2006. “The Measure of Induced Abortion Levels in Mexico Using Random Response Technique.” Sociological Methods & Research 35: 279–301. DOI: http://dx.doi.org/10.1177/0049124106290442.
48. Lara, D., J. Strickler, C.D. Olavarrieta, and C. Ellertson. 2004. “Measuring Induced Abortion in Mexico.” Sociological Methods & Research 32: 529–558. DOI: http://dx.doi.org/10.1177/0049124103262685.
49. Lee, R.M. 1993. Doing Research on Sensitive Topics. London: Sage.
50. Lensvelt-Mulders, G.J.L.M., J.J. Hox, P.G.M. van der Heijden, and C.J.M. Maas. 2005. “Meta-Analysis of Randomized Response Research: Thirty-Five Years of Validation.” Sociological Methods & Research 33: 319–348 .http://dx.doi.org/10.1177/0049124104268664.
51. Lensvelt-Mulders, G.J.L.M., J.J. Hox, and P.G.M. Van der Hejden. 2005b. “How to Improve the Efficiency of Randomized Response Designs.” Quality & Quantity 39: 253–265. DOI: http://dx.doi.org/10.1007/s11135-004-0432-3.
52. Lensvelt-Mulders, G.J.L.M., P.G.M. Van der Heijden, O. Laudy, and G. van Gils. 2006. “A Validation of Computer-Assisted Randomized Response Survey to Estimate the Prevalence of Undeclared Work in Social Security.” Journal of the Royal Statistical Society (Series A) 169: 305–318.
53. Locander, W., S. Sudman, and N. Bradburn. 1976. “An Investigation of Interview Method. Threat and Response Distortion.” Journal of the American Statistical Association 71: 269–275. DOI: http://dx.doi.org/10.1080/01621459.1976.10480332.
54. Maddala, G.S. 1983. Limited Dependent and Qualitative Variables in Econometrics. New York: Cambridge University Press.
55. Mangat, N.S. 1994. “An Improved Randomized Response Strategy.” Journal of the Royal Statistical Society (Series B) 56: 93–95.
56. Mangat, N.S. and R. Singh. 1990. “An Alternative Randomized Response Procedure.” Biometrika 77: 439–442. DOI: http://dx.doi.org/10.1093/biomet/77.2.439.
57. Manzoni, A., J.K. Vermunt, R. Luijkx, and R. Muffels. 2010. “Memory Bias in Retrospectively Collected Employment Careers: A Model-Based Approach to Correct for Measurement Error.” Sociological Methodology 40: 39–73. DOI: http://dx.doi.org/10.1111/j.1467-9531.2010.01230.x.
58. Mood, C. 2010. “Logistic Regression: Why We Cannot Do What We Think We Can Do, and What We Can Do About it.” European Sociological Review 26: 67–82. DOI: http://dx.doi.org/10.1093/esr/jcp006.
59. Moors, J.J.A. 1971. “Optimization of the Unrelated Randomized Response Model.” Journal of the American Statistical Association 66: 627–629. DOI: http://dx.doi.org/10.1080/01621459.1971.10482320.
60. Moshagen, M., E.B. Hilbig, E. Erdfelder, and A. Moritz. 2014. “An Experimental Validation Method for Questioning Techniques That Assess Sensitive Issues.” Experimental Psychology 61: 48–54. DOI: http://dx.doi.org/10.1027/1618-3169/a000226.
61. Ostapczuk, M., M. Moshagen, Z. Zhao, and J. Musch. 2009. “Assessing Sensitive Attributes Using the Randomized Response Technique: Evidence for the Importance of Response Symmetry.” Journal of Educational and Behavioral Statistics 43: 267–287. DOI: http://dx.doi.org/10.3102/1076998609332747.
62. Ostapczuk, M., J. Musch, and M. Moshagen. 2011. “Improving Self-Report Measures of Medication Non-Adherence Using a Cheating Detection Extension of the Randomized-Response Technique.” Statistical Methods in Medical Research 20: 489–503. DOI: http://dx.doi.org/10.1177/0962280210372843.
63. Tourangeau, R. and K.A. Rasinski. 1988. “Cognitive Processes Underlying Context Effects in Attitude Measurement.” Psychological Bulletin 103: 299–314. DOI: http://dx.doi.org/10.1037/0033-2909.103.3.299.
64. Tourangeau, R., L.J. Rips, and K. Rasinski. 2000. The Psychology of Survey Response. New York: Cambridge University Press.
65. Tourangeau, R. and T. Yan. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin 133: 859–883. DOI: http://dx.doi.org/10.1037/0033-2909.133.5.859.
66. Tracy, P.E. and J.A. Fox. 1981. “The Validity of Randomized Response for Sensitive Measurements.” American Sociological Review 46: 187–200.
67. Trappmann, M., S. Gundert, C. Wenzig, and D. Gebhardt. 2010. “PASS: A Household Panel Survey for Research on Unemployment and Poverty.” Schmollers Jahrbuch. Zeitschrift fu¨r Wirtschafts- und Sozialwissenschaften 130: 609–622.
68. Umesh, U.N. and R.A. Peterson. 1991. “A Critical Evaluation of the Randomized Response Method: Applications, Validation, and Research Agenda.” Sociological Methods & Research 20: 104–138. DOI: http://dx.doi.org/10.1177/0049124191020001004.
69. Van den Hout, A., U. Böckenholt, and P.G.M. van der Heijden. 2010. “Estimating the Prevalence of Sensitive Behavior and Cheating with Dual Design for Direct Questioning and Randomized Response.” Journal of the Royal Statistical Society: Series C (Applied Statistics) 59: 723–736. DOI: http://dx.doi.org/10.1111/j.1467-9876.2010.00720.x.
70. Van der Heijden, P.G.M., G. van Gils, J. Bouts, and J.J. Hox. 2000. “A Comparison of Randomized Response, Computer-Assisted Self-Interview, and Face-to-Face Direct Questioning: Eliciting Sensitive Information in the Context of Welfare and Unemployment Benefit.” Sociological Methods & Research 28: 505–537. DOI: http://dx.doi.org/10.1177/0049124100028004005.
71. Warner, S.L. 1965. “Randomized-Response: A Survey Technique for Eliminating Evasive Answer Bias.” Journal of the American Statistical Association 60: 63–69. DOI: http://dx.doi.org/10.1080/01621459.1965.10480775.
72. Weissman, A.N., R.A. Steer, and D.S. Lipton. 1986. “Estimating Illicit Drug Use Through Telephone Interviews and the Randomized Response Technique.” Drug and Alcohol Dependence 18: 225–233. DOI: http://dx.doi.org/10.1016/0376-8716(86)90054-2.
73. Wolter, F. 2012. Heikle Fragen in Interviews. Eine Validierung der Randomized Response-Technik. Springer VS
74. Wolter, F. and P. Preisendorfer. 2013. “Asking Sensitive Questions: An Evaluation of the Randomized Response Technique versus Direct Questioning Using Individual Validation Data.” Sociological Methods & Research 42: 321–353. DOI: http://dx.doi.org/10.1177/0049124113500474.
75. Yu, J.-W., G.L. Tian, and M.L. Tang. 2008. “Two New Models for Survey Sampling With Sensitive Characteristic: Design and Analysis.” Metrika 67: 251–263. DOI: http://dx.doi.org/10.1007/s00184-007-0131-x.