Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions

Sociological Methods and Research

Volumn 42, Issue 3, 2013, Pages 227-263

  Klausch, Thomas ^a   Hox, Joop J ^a   Schouten, Barry ^a,b  

^a UTRECHT UNIVERSITY   (Netherlands)

^b Statistics Netherlands ^*  (Netherlands)

Author keywords

categorical data modeling; confirmatory factor analysis; measurement effects; measurement equivalence; mixed mode surveys; mode effects

Indexed keywords

EID: 84884567112     PISSN: 00491241     EISSN: 15528294     Source Type: Journal    
DOI: 10.1177/0049124113500480     Document Type: Article

References (75)

1. Alwin Duane F.Margins of Error. Hoboken, NJ: John Wiley; 2007:.
2. Andrews Frank M."Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach.".Public Opinion Quarterly. 1984;48 (2): 409-42.
3. Asparouhov Tihomir."Sampling Weights in Latent Variable Modeling.".Structural Equation Modeling: A Multidisciplinary Journal. 2005;12:411-34.
4. AsparouhovTihomirMuthénBengt. 2006. "Robust Chi Square Difference Testing with mean and variance adjusted test statistics."Retrieved June 12, 2012 (http://www.statmodel.com/download/webnotes/webnote10.pdf).
5. Biemer Paul,Stokes LynneMeasurement Errors in Surveys, Wiley Series in Probability and Statistics. Biemer Paul P.Groves Robert M.Lyberg Lars E.Mathiowetz Nancy A.Sudman Seymour, ed. Hoboken, NJ: John Wiley; 1991:487-517.
6. Billiet Jaak B.,Davidov Eldad."Testing the Stability of an Acquiescence Style Factor Behind Two Interrelated Substantive Variables in a Panel Design.".Sociological Methods & Research. 2008;36:542-62.
7. Billiet Jaak B.,McClendon McKee J."Modeling Acquiescence in Measurement Models for Two Balanced Sets of Items.".Structural Equation Modeling: A Multidisciplinary Journal. 2000;7:608-28.
8. Blalock H. M."A Causal Approach to Nonrandom Measurement Errors.".The American Political Science Review. 1970;64:1099-111.
9. Bollen Kenneth A.Structural Equations with Latent Variables. New York: John Wiley; 1989:.
10. Bollen Kenneth A.,Paxton Pamela."Detection and Determinants of Bias in Subjective Measures.".American Sociological Review. 1998;63:465-78.
11. Bowling Ann."Mode of Questionnaire Administration can have Serious Effects on Data Quality.".Journal of Public Health. 2005;27:281-91.
12. Braunsberger Karin,Wybenga Hans,Gates Roger."A Comparison of Reliability between Telephone and Web-Based Surveys.".Journal of Business Research. 2007;60:758-64.
13. Buchanan Tom,Johnson John A.,Goldberg Lewis R."Implementing a Five-Factor Personality Inventory for Use on the Internet.".European Journal of Psychological Assessment. 2005;21:115-27.
14. Chang Linchiat,Krosnick Jon A."National Surveys Via RDD Telephone Interviewing Versus the Internet.".Public Opinion Quarterly. 2009;73:641-78.
15. Chen Fang Fang."Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance.".Structural Equation Modeling: A Multidisciplinary Journal. 2007;14:464-504.
16. Christian Leah Melani,Dillman Don A.,Smyth Jolene D."The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys.".Advances in Telephone Survey Methodology. 2008;:250-75.
17. Cole Michael S.,Bedeian Arthur G.,Feild Hubert S."The Measurement Equivalence of Web-Based and Paper-and-Pencil Measures of Transformational Leadership.".Organizational Research Methods. 2006;9:339-68.
18. Davis Darren W."Nonrandom Measurement Error and Race of Interviewer Effects among African Americans.".The Public Opinion Quarterly. 1997;61:183-207.
19. de Beuckelaer Alain,Lievens Filip."Measurement Equivalence of Paper-and-Pencil and Internet Organisational Surveys: A Large Scale Examination in 16 Countries.".Applied Psychology. 2009;58:336-61.
20. de Leeuw Edith DData Quality in Mail, Telephone, and Face to Face surveys. Amsterdam, the Netherlands: TT-Publicaties; 1992:.
21. de Leeuw Edith D."To Mix or Not to Mix Data Collection Modes in Surveys.".Journal of Official Statistics. 2005;21:233-55.
22. de Leeuw Edith DInternational Handbook of Survey Methodology. Leeuw Edith D. deHox Joop J.Dillman Don A., ed. New York: Taylor & Francis; 2008:113-35.
23. de Leeuw Edith D.,Hox Joop J.,Scherpenzeel AnnetteJSM Proceedings, Survey Research Methods Section. Alexandria, VA: American Statistical Association; 2011:6305-14.
24. de Leeuw Edith D.,Mellenbergh Gideon J.,Hox. Joop J."The Influence of Data Collection Method on Structural Models.".Sociological Methods & Research. 1996;24:443-72.
25. Deutskens Elisabeth,de Ruyter Ko,Wetzels Martin."An Assessment of Equivalence between Online and Mail Surveys in Service Research.".Journal of Service Research. 2006;8:346-55.
26. Dillman Don A.,Phelps Glenn,Tortora Robert,Swift Karen,Kohrell Julie,Berck Jodi."Response Rate and Measurement Differences in Mixed-Mode Surveys using Mail, Telephone, Interactive Voice Response (IVR) and the Internet.".Social Science Research. 2009;38:1-18.
27. Dillman Don A.,Smyth Jolene D.,Christian Leah MelaniInternet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley; 2009:.
28. French Brian F.,Holmes Finch W."Multigroup Confirmatory Factor Analysis: Locating the Invariant Referent Sets.".Structural Equation Modeling: A Multidisciplinary Journal. 2008;15:96-113.
29. Fricker Scott,Galesic Mirta,Tourangeau Roger,Yan Ting."An Experimental Comparison of Web and Telephone Surveys.".Public Opinion Quarterly. 2005;69:370-92.
30. Fuller WayneMeasurement Error Models. New York: John Wiley; 1987:.
31. Gerbing David W.,Anderson James C."On the Meaning of within-Factor Correlated Measurement Errors.".Journal of Consumer Research. 1984;11:572-80.
32. Green Donald Philip,Citrin Jack."Measurement Error and the Structure of Attitudes: Are Positive and Negative Judgments Opposites?".American Journal of Political Science. 1994;38:256-81.
33. Greene Jessica,Speizer Howard,Wiitala Wyndy."Telephone and Web: Mixed-Mode Challenge.".Health Services Research. 2008;43:230-48.
34. Groves Robert M.,Fowler Floyd. J.,Couper Mick P.,Lepkowski James M.,Singer Eleanor,Tourangeau RogerSurvey Methodology. 2010. Survey Methodology. 2nd ed. Hoboken, NJ: John Wiley.Hoboken, NJ: John Wiley; 2010:.
35. Guo Shenyang,Fraser Mark W.Propensity Score Analysis. Thousand Oaks, CA: Sage; 2010:.
36. Heerwegh Dirk,Loosveldt Geert."Face-to-Face versus Web Surveying in a High-Internet-Coverage Population.".Public Opinion Quarterly. 2008;72:836-46.
37. Heerwegh Dirk,Loosveldt Geert."Assessing Mode Effects in a National Crime Victimization Survey using Structural Equation Models: Social Desirability Bias and Acquiescence.".Journal of Official Statistics. 2011;27:49-63.
38. Holbrook Allyson L.,Green Melanie C.,Krosnick Jon A."Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias.".Public Opinion Quarterly. 2003;67:79-125.
39. Jäckle Annette,Roberts Caroline,Lynn Peter."Assessing the Effect of Data Collection Mode on Measurement.".International Statistical Review. 2010;78:3-20.
40. Jöreskog Karl Gustav."Simultaneous Factor Analysis in Several Populations.".Psychometrika. 1971;36:409-26.
41. Kamata Akihito,Bauer Daniel J."A Note on the Relation between Factor Analytic and Item Response Theory Models.".Structural Equation Modeling: A Multidisciplinary Journal. 2008;15:136-53.
42. Kankaraš Miloš,Vermunt Jeroen K.,Moors Guy."Measurement Equivalence of Ordinal Items: A Comparison of Factor Analytic, Item Response Theory, and Latent Class Approaches.".Sociological Methods & Research. 2011;40:279-310.
43. Kaplan David."An Extension of the Propensity Score Adjustment Method for the Analysis of Group Differences in MIMIC Models.".Multivariate Behavioral Research. 1999;34:467-92.
44. Kim Eun Sook,Yoon Myeongsun."Testing Measurement Invariance: A Comparison of Multiple-Group Categorical CFA and IRT.".Structural Equation Modeling: A Multidisciplinary Journal. 2011;18:212-28.
45. Krosnick Jon A."Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.".Applied Cognitive Psychology. 1991;5:213-36.
46. Link Michael W.,Mokdad Ali H."Alternative Modes for Health Surveillance Surveys: An Experiment with Web, Mail, and Telephone.".Epidemiology. 2005;16:701-04.
47. Lord Frederic M.,Norvick Melvin R.Statistical theories of mental test scores. Reading, MA: Addison-Wesley; 1968:.
48. Meade Adam W.,Lautenschlager Gary J."A Comparison of Item Response Theory and Confirmatory Factor Analytic Methodologies for Establishing Measurement Equivalence/Invariance.".Organizational Research Methods. 2004;7:361-88.
49. Mellenbergh Gideon J."Item Bias and Item Response Theory.".International Journal of Educational Research. 1989;13:127-43.
50. Mellenbergh Gideon J.Research Methodology in the Social, Behavioral and Life Sciences,. Adèr HermannMellenbergh Gideon, ed. London, UK: Sage; 1999:168-87.
51. Meredith William."Measurement Invariance, Factor Analysis and Factorial Invariance.".Psychometrika. 1993;58:525-43.
52. Millsap Roger E.Statistical Approaches to Measurement Invariance. New York: Routledge; 2011:.
53. Millsap Roger E.,Yun-Tein Jenn."Assessing Factorial Invariance in Ordered-Categorical Measures.".Multivariate Behavioral Research. 2004;39:479-515.
54. Morgan Stephen L.,Winship ChirstopherCounterfactuals and Causal Inference. Cambridge, MA: Cambridge University Press; 2007:.
55. Morren Meike,Gelissen John P. T. M.,Vermunt Jeroen K."Dealing with Extreme Response Style in Cross-Cultural Research: A Restricted Latent Class Factor Analysis Approach.".Sociological Methodology. 2011;41:13-47.
56. Muthén Bengt."A General Structural Equation Model with Dichotomous, Ordered Categorical, and Continuous Latent Variable Indicators.".Psychometrika. 1984;49:115-32.
57. MuthénBengt O.MuthénLinda. 2010. "IRT in Mplus."Retrieved June 12, 2012 (http://www.statmodel.com/download/MplusIRT2.pdf).
58. MuthénBengt O.AsparouhovTihomir. 2002. "Latent Variable Analysis with Categorical Outcomes: Multiple-Group and Growth Modeling in Mplus."Retrieved July 12, 2012 (http://www.statmodel.com/download/webnotes/CatMGLong.pdf).
59. MuthénBengt O.du ToitS. H. C.SpisicD. 1997. "Robust Inference Using Weighted Least Squares and Quadratic Estimating Equations in Latent Variable Modeling with Categorical and Continuous Outcomes."Retrieved July 12, 2012 (http://www.gseis.ucla.edu/faculty/muthen/articles/Article_075.pdf).
60. Oort Frans J."Simulation Study of Item Bias Detection with Restricted Factor Analysis.".Structural Equation Modeling: A Multidisciplinary Journal. 1998;5:107-24.
61. Rosenbaum Paul R."Model-Based Direct Adjustment.".Journal of the American Statistical Association. 1987;82:387-94.
62. Rosenbaum Paul R.,Rubin Donald B."The Central Role of the Propensity Score in Observational Studies for Causal Effects.".Biometrika. 1983;70:41-55.
63. Saris Willem E.,Andrews Frank M.Measurement Errors in Surveys, Wiley Series in Probability and Statistics,. Biemer Paul P.Groves Robert M.Lyberg Lars E.Mathiowetz Nancy A.Sudman Seymour, ed. Hoboken, NJ: John Wiley; 1991:487-517.
64. Saris Willem E.,Gallhofer Irmtraud."Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.".Survey Research Methods. 2007;1:29-43.
65. Scherpenzeel Annette C.,Saris Willem E."The Validity and Reliability of Survey Questions A Meta-Analysis of MTMM Studies.".Sociological Methods & Research. 1997;25:341-83.
66. Schonlau Matthias,Zapert Kinga,Simon Lisa Payne,Sanstad Katherine Haynes,Marcus Sue M.,Adams John,Spranca Mark,Kan Hongjun,Turner Rachel,Berry Sandra H."A Comparison between Responses from a Propensity-Weighted Web Survey and an Identical RDD Survey.".Social Science Computer Review. 2004;22:128-38.
67. Skrondal Anders,Rabe-Hesketh SophiaGeneralized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models. Boca Raton, FL: Chapman & Hall/CRC; 2004:.
68. Stark Stephen,Chernyshenko Oleksandr S.,Drasgow Fritz."Detecting Differential Item Functioning with Confirmatory Factor Analysis and Item Response Theory: Toward a Unified Strategy.".Journal of Applied Psychology. 2006;91:1292-306.
69. Tourangeau Roger,Rips Lance J.,Rasinski KennethThe Psychology of Survey Response. Cambridge MA: Cambridge University Press; 2000:.
70. Tourangeau Roger,Couper Mick P.,Conrad Frederick."Spacing, Position, and Order Interpretive Heuristics for Visual Features of Survey Questions.".Public Opinion Quarterly. 2004;68:368-93.
71. Vannieuwenhuyze Jorre T. A.,Loosveldt Geert."Evaluating Relative Mode Effects in Mixed-Mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.".Sociological Methods & Research. 2013;42:82-104.
72. Vannieuwenhuyze Jorre T. A.,Loosveldt Geert,Molenberghs Geert."A Method for Evaluating Mode Effects in Mixed-mode Surveys.".Public Opinion Quarterly. 2010;74:1027-45.
73. Vandenberg Robert J.,Lance Charles E."A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research.".Organizational Research Methods. 2000;3 (1): 4-70.
74. Welkenhuysen-Gybels Jerry,Billiet Jaak,Cambré Bart."Adjustment for Acquiescence in the Assessment of the Construct Equivalence of Likert-Type Score Items.".Journal of Cross-Cultural Psychology. 2003;34:702-22.
75. Yoon Myeongsun,Millsap Roger E."Detecting Violations of Factorial Invariance Using Data-Based Specification Searches: A Monte Carlo Study.".Structural Equation Modeling: A Multidisciplinary Journal. 2007;14:435-63.