A new approach to evaluating the quality of measurement instruments: The split-ballot MTMM design

Sociological Methodology

Volumn 34, Issue , 2004, Pages 311-347

  Saris, Willem E ^a,b,c   Satorra, Albert ^d   Coenders, Germà ^e  

^a ESADE BUSINESS SCHOOL   (Spain)

^b UNIVERSITAT RAMON LLULL   (Spain)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

^d UNIVERSITAT POMPEU FABRA   (Spain)

^e UNIVERSITY OF GIRONA   (Spain)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 16244366781     PISSN: 00811750     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.0081-1750.2004.00155.x     Document Type: Article

References (49)

1. Allison, P. D. 1987. "Estimation of Linear Models with Incomplete Data." Pp. 71-103 in Sociological Methodology, Vol 17, edited by C. C. Clogg. Washington, DC: American Sociological Association.
2. Althauser, R. P., T. A. Heberlein, and R. A. Scott. 1971. "A Causal Assessment of Validity: The Augmented Multitrait-Multimethod Matrix." Pp. 151-69 in Causal Models in the Social Sciences, edited by H. M. Blalock, Jr. Chicago: Aldine.
3. Alwin, D. 1974. "An Analytic Comparison of Four Approaches to the Interpretation of Relationships in the Multitrait-Multimethod Matrix." Pp. 79-105 in Sociological Methodology, edited by H. L. Costner. San Francisco: Jossey-Bass.
4. Andrews, F. M. 1984. "Construct Validity and Error Components of Survey Measures. A Structural Modeling Approach." Public Opinion Quarterly 48:409-42.
5. Arminger, G., and M.E. Sobel. 1990. "Pseudo-Maximum Likelihood Estimation of Mean and Covariance Structures with Missing Data." Journal of the American Statistical Association 85:195-203.
6. Bagozzi, R. P., and Y. Yi. 1991. "Multitrait-Multimethod Matrices in Consumer Research." Journal of Consumer Research 17:426-39.
7. Billiet, J., G. Loosveldt, and L. Waterplas. 1986. "Het Survey-Interview Onderzocht: Effecten van het Ontwerp en Gebruik van Vragenlijsten op de Kwaliteit van de Antwoorden" (Research on surveys: effects of the design and use of questionnaires on the quality of the responses). Leuven, Belgium: Sociologisch Onderzoeksinstituut KU Leuven.
8. Brannick, M. T., and P. E. Spector. 1990. "Estimation Problems in the Block-Diagonal Model of the Multitrait-Multimethod Matrix." Applied Psychological Measurement 14:325-39.
9. Browne M. W. 1984. "The Decomposition of Multitrait-Multimethod Matrices." British Journal of Mathematical and Statistical Psychology 37:1-21.
10. Bunting, B., G. Adamson, and P. K. Mulhall. 2002. "A Monte Carlo Examination of an MTMM Model with Planned Incomplete Data Structures." Structural Equation Modeling 9:369-89.
11. Campbell, D. T., and D. W. Fiske. 1959. "Convergent and Discriminant Validation by the Multitrait Multimethod Matrices." Psychological Bulletin 56: 81-105.
12. Campbell, D. T., and E. J. O'Connell. 1967. "Method Factors in Multitrait-Multimethod Matrices: Multiplicative Rather Than Additive?" Multivariate Behavioral Research 2:409-26.
13. Coenders, G., and W. E. Saris. 1995. "Categorization and Quality: The Choice Between Pearson and Polychoric Correlations." Pp. 125-44 in The Multitrait-Multimethod Approach to Evaluate Measurement Instruments, edited by W. E. Saris and A. Münnich. Budapest: Eötvös University Press.
14. Coenders, G., A. Satorra, and W. E. Saris. 1997. "Alternative approaches to Structural Modeling of Ordinal Data: A Monte Carlo Study." Structural Equation Modeling 4:261-82.
15. _. 1998. "Relationship Between a Restricted Correlated Uniqueness Model and a Direct Product Model for Multitrait-Multimethod Data." Pp. 151-72 in Advances in Methodology, Data Analysis and Statistics, Metodoloki Zvezki, Vol 14. Ljubljana, Slovenia.
16. _. 2000. "Testing Nested Additive, Multiplicative and General Multitrait-Multimethod Models." Structural Equation Modeling 7:219-50.
17. Corten I., W. E. Saris, G. Coenders, W. van der Veld, C. Albers, and C. Cornelis. 2002. "The Fit of Different Models for Multitrait-Multimethod Experiments." Structural Equation Modeling 9:213-32.
18. Cudeck, R. 1988. "Multiplicative Models and MTMM Matrices." Journal of Educational Statistics 13, 131-47.
19. Eid, M. 2000. "Multitrait-Multimethod Model with Minimal Assumptions." Psychometrika 65:241-61.
20. European Social Survey. 2002. European Social Survey Round 1: Report of the First Year. London: Natcen.
21. Grayson, D., and H. W. Marsh. 1994. "Identification with Deficient Rank Loading Matrices in Confirmatory Analysis: Multitrait-Multimethod Models." Psychometrika 59:121-34.
22. Jöreskog, K. G. 1971. "Simultaneous Factor Analysis in Several Populations." Psychometrika 34:409-26.
23. _. 1990. "New Developments in LISREL: Analysis of Ordinal Variables Using Polychoric Correlations and Weighted Least Squares." Quality and Quantity 24:387-404.
24. Kenny, D. A. 1976. "An Empirical Application of Confirmatory Factor Analysis to the Multitrait-Multimethod Matrix." Journal of Experimental Social Psychology 12:247-52.
25. Kenny, D. A., and Kashy, D. A. 1992. "Analysis of the Multitrait-Multimethod Matrix by Confirmatory Factor Analysis." Psychological Bulletin 112:165-72.
26. Kogovšek, T., A. Ferligoj, G. Coenders, and W. E. Saris. 2002. "Estimating Reliability and Validity of Personal Support Measures: Full Information ML Estimation with Planned Incomplete Data." Social Networks 24:1-20.
27. Költringer, R. 1995. "Measurement Quality in Austria Personal Interviews." Pp. 207-25 in The Multitrait-Multimethod Approach to Evaluate Measurement Instruments, edited by W.E. Saris and A. Münnich. Budapest: Eötvös University Press.
28. Marsh, H. W. 1989. "Confirmatory Factor Analysis of Multitrait-Multimethod Data: Many Problems and Few Solutions." Applied Psychological Measurement 13:335-61.
29. Marsh, H. W., and M. Bailey. 1991. "Confirmatory Factor Analyses of Multitrait-Multimethod Data: Comparison of the Behavior of Alternative Models." Applied Psychological Measurement 15:47-70.
30. van Meurs, A., and W. E. Saris. 1990. "Memory Effects in MTMM Studies." Pp. 134-46 in Evaluation of Measurement Instruments by Meta-analysis of Multitrait-Multimethod Studies, edited by W. E. Saris and A. van Meurs. Amsterdam: North Holland.
31. Muthén, B. 1984. "A General Structural Equation Model with Dichotomous, Ordered Categorical, and Continuous Latent Variable Indicators." Psycho-metrika 49:115-32.
32. Muthén, B., D. Kaplan, and M. Hollis. 1987. "On Structural Equation Modeling with Data That Are Not Missing Completely at Random." Psychometrika 52:431-62.
33. Olsson, U. 1979. "Maximum-Likelihood Estimation of the Polychoric Correlation Coefficient." Psychometrika 44:443-60.
34. Rodgers, W. L., F. M. Andrews, and A. R. Herzog. 1992. "Quality of Survey Measures: A Structural Modeling Approach." Journal of Official Statistics 8:251-75.
35. Saris, W. E. 1990. "The Choice of a Model for Evaluation of Measurement Instruments." Pp. 118-29 in Evaluation of Measurement Instruments by Meta-analysis of Multitrait Multimethod Matrices, edited by W. E. Saris and A. van Meurs. Amsterdam: North Holland.
36. _. 1998. "A New Approach for Evaluation of Measurement Instruments: The Split-Ballot MTMM Design." Presented at the International Conference on Methodology and Statistics, Preddvor, Slovenia, September 20-22, 1998.
37. Saris, W. E., and F. M. Andrews. 1991. "Evaluation of Measurement Instruments Using a Structural Modeling Approach." Pp. 575-99 in Measurement Errors in Surveys, edited by P. P. Biemer et al. New York: Wiley.
38. Saris, W. E. and I. N. Gallhofer. Forthcoming. "Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions."
39. Saris, W. E., and J. Krosnick. Forthcoming. "Comparing Questions with Agree/Disagree Response Options to Questions with Construct Specific Response Options."
40. Saris, W. E., and A. Munnich. 1995. The Multitrait-Multimethod Approach to Evaluate Measurement Instruments. Budapest: Eötvös University Press.
41. Saris, W. E., T. Van Wijk, and A. Scherpenzeel. 1998. "Validity and Reliability of Subjective Social Indicators: The Effect of Different Measures of Association." Social Indicators Research 45:173-99.
42. Satorra, A. 1992. "Asymptotic Robust Inferences in the Analysis of Mean and Covariance Structures." Pp. 249-78 in Sociological Methodology, Vol. 22, edited by P. V. Marsden. Cambridge, MA: Basil Blackwell.
43. Satorra, A. 1993. "Asymptotic Robust Inferences in Multi-Sample Analysis of Augmented-Moment Structures." Pp. 211-29 in Multivariate Analysis: Future Directions, Vol, 2, edited by C. M. Cuadras and C. R. Rao. Amsterdam: Elsevier.
44. Satorra, A. 2001. "Goodness of Fit Testing of Structural Equation Models with Multiple Group Data and Nonnormality." Chap. 12 in Structural Equation Modeling: Present and Future, edited by R. Cudeck, S. du Toit, and D. Sörbom. Lincolnwood, IL: Scientific Software International. SSI.
45. Scherpenzeel, A. C. 1995. "A Question of Quality. Evaluating Survey Questions by Multitrait-Multimethod Studies." Ph.D. dissertation, University of Amsterdam, Leidschendam, the Netherlands.
46. Scherpenzeel, A., and W. E. Saris. 1997. "The Validity and Reliability of Survey Questions: A Meta-analysis of MTMM Studies." Sociological Methods and Research 25:341-83.
47. Schuman, H., and S. Presser, 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Order, and Context. New York: Academic Press.
48. Werts, C. E., and R. L. Linn, 1970. "Path Analysis. Psychological Examples." Psychological Bulletin 74:193-212.
49. Wothke, W. 1996. "Models for Multitrait-Multimethod Matrix Analysis. Pp. 7-56 in Advanced Structural Equation Modeling. Issues and Techniques, edited by G. C. Marcoulides and R. E. Schumacker. Mahwah, NJ: Lawrence Erlbaum.