A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

Multivariate Behavioral Research

Volumn 51, Issue 1, 2016, Pages 67-85

  Koch, Tobias ^a   Schultze, Martin ^b   Jeon, Minjeong ^c   Nussbeck, Fridtjof W ^d   Praetorius, Anna Katharina ^e   Eid, Michael ^b  

^a LEUPHANA UNIVERSITY OF LÜNEBURG   (Germany)

^b FREIE UNIVERSITÄT BERLIN   (Germany)

^c OHIO STATE UNIVERSITY   (United States)

^d BIELEFELD UNIVERSITY   (Germany)

^e GERMAN INSTITUTE FOR INTERNATIONAL EDUCATIONAL RESEARCH   (Germany)

Author keywords

Bayesian analysis; cross classification; MTMM modeling; structurally different and interchangeable methods

Indexed keywords

HUMAN; MODEL; MULTITRAIT MULTIMETHOD; SCIENTIST; VARIANCE; ACHIEVEMENT; ALGORITHM; BAYES THEOREM; CROSS-SECTIONAL STUDY; MULTIVARIATE ANALYSIS; PROCEDURES; PSYCHOLOGY; SCHOOL TEACHER; SELF CONCEPT; STATISTICAL MODEL; STUDENT; THINKING;

ACHIEVEMENT; ALGORITHMS; BAYES THEOREM; CROSS-SECTIONAL STUDIES; HUMANS; MODELS, STATISTICAL; MULTIVARIATE ANALYSIS; SCHOOL TEACHERS; SELF CONCEPT; STUDENTS; THINKING;

EID: 84958794078     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2015.1101367     Document Type: Article

References (81)

1. T.Asparouhov,, & B.Muthén, (2012). General random effect latent variable modeling: Random subjects, items, contexts, and parameters. Retrieved from http://www.statmodel.com/download/NCMErevision2.pdf
2. D.Bauer,, A.L.Howard,, R.E.Baldasaro,, P.J.Curran,, A.M.Hussong,, L.Chassin,, & R.A.Zucker, (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. doi: 10.1037/a0032475
3. S.N.Beretvas, (2011). Cross-classified and multiple membership models. In J.Hox & R.J.Kyle (Eds.), Handbook of advanced multilevel analysis (pp. 313–334). New York, NY: Routledge Taylor & Francis Group.
4. H.D.Bull,, M.Schultze,, C.Geiser,, & H.Scheithauer, (2013). The role of cognitive empathy in the association between social intelligence and relational aggression: A MTMM analysis of informant agreement in the assessment of empathy deficits in relationally aggressive adolescents. (Manuscript submitted for publication.)
5. H.D.Bull,, M.Schultze,, & H.Scheithauer, (2009). School-based prevention of bullying and relational aggression: The fairplayer.manual. European Journal of Developmental Science, 3(3), 312–317. doi: 10.1007/s11121-009-0128-y
6. D.T.Campbell,, & D.W.Fiske, (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. doi: 10.1037/h0046016
7. H.Carretero-Dios,, M.Eid,, & W.Ruch, (2011). Analyzing multitrait-mulitmethod data with multilevel confirmatory factor analysis: An application to the validation of the state-trait cheerfulness inventory. Journal of Research in Personality, 45, 153–164. doi:10.1016/j.jrp.2010.12.007
8. L.Castro-Schilo,, K.F.Widaman,, & K.J.Grimm, (2013). Neglect the structure of multitrait-multimethod data at your peril: Implications for associations with external variables. Structural Equation Modeling: A Multidisciplinary Journal, 20(2), 181–207. doi:10.1080/10705511.2013.769385
9. S.-J.Cho,, I.Partchev,, & P.De Boeck, (2012). Parameter estimation of multiple item response profile model. British Journal of Mathematical and Statistical Psychology, 65, 438–466. doi:10.1111/j.2044-8317.2011.02036.x
10. S.-J.Cho,, & S.Rabe-Hesketh, (2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics & Data Analysis, 55, 12–25. doi:10.1016/j.csda.2010.04.015
11. J.M.Conway,, & A.I.Huffcutt, (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331–360. doi:10.1207/s15327043hup1004\_2
12. S.M.Curtis, (2012). mcmcplots: Create plots from MCMC output [Computer software manual]. Retrieved from http://CRAN.R-project.org/package=mcmcplots
13. E.Danay,, & M.Ziegler, (2011). Is there really a single factor of personality? A multirater approach to the apex of personality. Journal of Research in Personality, 45, 560–567. doi:10.1016/j.jrp.2011.07.003
14. L.Dumenci, (2000). Multitrait-multimethod analysis. In S.D.Brown & H.E.A.Tinsley (Eds.), Handbook of applied multivariate statistics and mathematerial modeling (pp. 583–611). San Diego, CA: Academic Press.
15. M.Eid, (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241–261. doi:10.1007/BF02294377
16. M.Eid,, & E.Diener, (2006). Introduction: The need for multimethod measurement in psychology. In M.Eid & E.Diener (Eds.), Handbook of psychological measurement: A multimethod perspective (pp. 3–8). Washington, DC: American Psychological Association.
17. M.Eid,, T.Lischetzke,, & F.Nussbeck, (2006). Structural equation models for multitrait-multimethod data. In M.Eid & E.Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 289–299). Washington, DC: American Psychological Association.
18. M.Eid,, T.Lischetzke,, F.W.Nussbeck,, & L.I.Trierweiler, (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38–60. doi:10.1037/1082-989X.8.1.38
19. M.Eid,, F.W.Nussbeck,, C.Geiser,, D.A.Cole,, M.Gollwitzer,, & T.Lischetzke, (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230–253. doi:10.1037/a0013219
20. A.Fielding,, & H.Goldstein, (2006). Cross-classified and multiple membership structures in multilevel models: An introduction and review (Tech. Rep.). Research Report RR791. London, UK: Department for Education and Skills.
21. T.Gaugler,, & M.G.Akritas, (2011). Testing for interaction in two-way random and mixed effects models: The fully nonparametric approach. Biometrics, 67, 1314–1320. doi:10.1111/j.1541-0420.2011.01579.x
22. C.Geiser,, M.Eid,, S.G.West,, T.Lischetzke,, & F.W.Nussbeck, (2012). A comparison of method effects in two confirmatory factor models for structurally different methods. Structural Equation Modeling, 19(3), 409–436. doi:10.1080/10705511.2012.687658
23. C.Geiser,, M.Eid,, & F.W.Nussbeck, (2008). On the meaning of the latent variables in the CT-C(M-1) model: A comment on Maydeu-Olivares and Coffman (2006). Psychological Methods, 13, 49–57. doi:10.1037/1082-989X.13.1.49
24. C.Geiser,, T.Koch,, & M.Eid, (2014). Data-generating mechanisms versus constructively defined latent variables in multitrait-multimethod analysis: A comment on Castro-Schilo, Widaman, and Grimm (2013). Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 509–523. doi:10.1080/10705511.2014.919816
25. A.Gelman,, J.B.Carlin,, H.S.Stern,, D.B.Dunson,, A.Vehtari,, & D.B.Rubin, (2004). Bayesian data analysis (2nd ed.). New York, NY: Chapman & Hall.
26. A.Gelman,, & J.Hill, (2007). Data analysis using regression and multilevel/hierarchical models. New York, NY: Cambridge University Press.
27. A.Gelman,, & D.B.Rubin, (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472. doi:10.1214/ss/1177011136
28. J.Ghorpade, (2000). Managing five paradoxes of 360-degree feedback. The Academy of Management Executive, 14, 140–150. doi:10.5465/AME.2000.2909846
29. H.Goldstein, (1994). Multilevel cross-classified models. Sociological Methods & Research, 22, 364–375. doi:10.1177/0049124194022003005
30. H.Goldstein, (2011). Cross-classified data structures. In W.A.Shewhart & S.S.Wilks (Eds.), Multilevel statistical models (pp. 243–254). Chichester, UK: John Wiley & Sons. doi:10.1002/9780470973394.ch12
31. J.Hox,, & C.Maas, (2006). Multilevel models for multimethod measures. In M.Eid & E.Diener (Eds.), Handbook of psychological measurement: A multimethod perspective (pp. 269–282). Washington, DC: American Psychological Association.
32. W.T.Hoyt, (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5, 64–86. doi:10.1037/1082-989X.5.1.64
33. M.Jeon,, & S.Rabe-Hesketh, (2012). Profile-likelihood approach for estimating generalized linear mixed models with factor structures. Journal of Educational and Behavioral Statistics, 37, 518–542. doi:10.3102/1076998611417628
34. M.Jeon,, & F.Rijmen, (2014). Recent developments in maximum likelihood estimation of MTMM models for categorical data. Frontiers in Psychology, 5(269), 1–7. doi:10.3389/fpsyg.2014.00269
35. M.W.Julian, (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8(3), 325–352. doi:10.1207/S15328007SEM0803\_1
36. A.Kelava, (2009). Multikollinearität in nicht-linearen latenten Strukturgleichungsmodellen [Multicollinearity in non-linear structural equation models]. (Doctoral dissertation). Goethe University, Frankfurt, Germany. Retrieved from http://publikationen.ub.uni-frankfurt.de/frontdoor/index/index/docId/6267
37. D.A.Kenny, (1976). An empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology, 12(3), 247–252. doi:10.1016/0022-1031(76)90055-X
38. D.A.Kenny, (1995). The multitrait-multimethod matrix: Design, analysis, and conceptual issues. In P.Shrout & S.T.Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp. 111–124). Hillsdale, NJ: Lawrence Erlbaum.
39. D.A.Kenny,, & D.A.Kashy, (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 112(1), 165–172. doi: 10.1037/0033-2909.112.1.165
40. T.Koch, (2013). Multilevel structural equation modeling of multitrait-multimethod-multioccasion data. (Doctoral dissertation). Freie Universität Berlin, Germany. Retrieved from http://www.diss.fu-berlin.de/diss/servlets/MCRFileNodeServlet/FUDISS_derivate_000000013668/Thesis_Koch_gedreht.pdf
41. T.Koch,, M.Eid,, & K.Lochner, (in press). Multitrait-multimethod-analysis: The psychometric foundation of CFA-MTMM models. In P.Irwing, T.Booth, & D.Hughes (Eds.), The Wiley handbook of psychometric testing. London, UK: John Wiley & Sons.
42. T.Koch,, M.Schultze,, M.Eid,, & C.Geiser, (2014). A longitudinal multileve CFA-MTMM model for interchangeable and structurally different methods. Frontiers in Psychology, 5(311), 1–9. doi:10.3389/fpsyg.2014.00311
43. M.Kunter,, G.Schümer,, C.Artelt,, J.Baumert,, E.Klieme,, M.Neubrand,,.. M.Weiß, (2003). Pisa 2000 - Dokumentation der Erhebungsinstrumente [PISA 2000: Documentation of the measures].
44. Luo, W., & Kwok, O.-M. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182–212. doi: 10.1080/00273170902794214
45. C.J.Maas,, G.J.Lensvelt-Mulders,, & J.J.Hox, (2009). A multilevel multitrait-multimethod analysis. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 72–77. doi: http://dx.doi.org/10.1027/1614-2241.5.3.72
46. J.Mahlke,, M.Schultze,, T.Koch,, M.Eid,, R.Eckert,, & F.Brodbeck, (2015). A multilevel CFA–MTMM approach for multisource feedback instruments: Presentation and application of a new statistical model. Structural Equation Modeling: A Multidisciplinary Journal. doi:10.1080/10705511.2014.990153
47. H.W.Marsh,, & D.Grayson, (1995). Latent variable models of multitrait-multimethod data. In R.H.Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (p. 177–198). Thousand Oaks, CA: Sage Publications.
48. H.W.Marsh,, & L.A.Roche, (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187–1197. doi:10.1037/0003-066X.52.11.1187
49. H.W.Marsh,, Z.Wen,, & K.-T.Hau, (2004). Structural equation models of latent interactions: Evaluation of alternative estimation strategies and indicator construction. Psychological Methods, 9, 275–300. doi:10.1037/1082-989X.9.3.275
50. T.Meiser,, & M.A.Steinwascher, (2014). Different kinds of interchangeable methods in multitrait-multimethod analysis: A note on the multilevel CFA-MTMM model by Koch et al. (2014). Frontiers in Psychology, 5(615), 1–3. doi:10.3389/fpsyg.2014.00615
51. H.Moosbrugger,, K.Schermelleh-Engel,, & A.Klein, (1997). Methodological problems of estimating latent interaction effects. Methods of psychological research online, 2, 95–111.
52. B.Muthén,, & T.Asparouhov, (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313–335. doi: http://dx.doi.org/10.1037/a0026802
53. F.W.Nussbeck,, M.Eid,, & T.Lischetzke, (2006). Analysing multitrait-multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results? British Journal of Mathematical and Statistical Psychology, 59, 195–213. doi:10.1348/000711005X67490
54. G.Pham,, T.Koch,, A.Helmke,, F.-W.Schrader,, T.Helmke,, & M.Eid, (2012). Do teachers know how their teaching is perceived by their pupils? Procedia - Social and Behavioral Sciences 46, 3368–3374. doi:10.1016/j.sbspro.2012.06.068
55. M.Plummer, (2003). Jags: A program for analysis of Bayesian graphical models using gibbs sampling [Computer software manual]. Retrieved from http://citeseer.ist.psu.edu/plummer03jags.html
56. S.Pohl,, & R.Steyer, (2010). Modeling common traits and method effects in multitrait-multimethod analysis. Multivariate Behavioral Research 45(1), 45–72. doi: 10.1080/00273170903504729
57. S.Pohl,, R.Steyer,, & K.Kraus, (2008). Modelling method effects as individual causal effects. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171(1), 41–63. doi:10.1111/j.1467-985X.2007.00517.x
58. A.-K.Praetorius,, V.-D.Berner,, H.Zeinz,, A.Scheunpflug,, & M.Dresel, (2013). Judgment confidence and judgment accuracy of teachers in judging self-concepts of students. The Journal of Educational Research, 106, 64–76. doi:10.1080/00220671.2012.667010
59. R Core Team. (2014). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
60. S.Rabe-Hesketh,, A.Skrondal,, & A.Pickles, (2004, October). GLLAMM manual [Computer software manual]. Working Paper 160. Berkeley, CA: U.C. Berkeley Division of Biostatistics Working Paper Series. Retrieved from http://biostats.bepress.com/ucbbiostat/paper160
61. S.Rabe-Hesketh,, A.Skrondal,, & A.Pickles, (2005). Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects. Journal of Econometrics, 128, 301–323. doi:10.1016/j.jeconom.2004.08.017
62. S.W.Raudenbush, (1993). A crossed random effects model for unbalanced data with applications in cross-sectional and longitudinal research. Journal of Educational and Behavioral Statistics, 18, 321–349. doi:10.3102/10769986018004321
63. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models (2nd Ed.). Thousand Oaks, CA: Sage.
64. F.Rijmen,, F.Tuerlinckx,, P.De Boeck,, & P.Kuppens, (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185–205. doi:10.1037/1082-989X.8.2.185
65. H.Sahai,, & M.M.Ojeda, (2005). Two-way crossed classification with interaction. Vol. II: Unbalanced data theory, methods, applications, and data analysis. New York, NY: Springer. doi:10.1007/978-0-8176-8168-5\_4
66. C.Schöne,, O.Dickhäuser,, B.Spinath,, & J.Stiensmeier-Pelster, (2002). Skalen zur Erfassung des schulischen Selbstkonzepts (SESSKO) [Scales for the measurement of academic self-concept.]. Göttingen, Germany: Hofgrefe.
67. M.Schultze, (2012). Evaluating what the crowd says. A longitudinal structural equation model for exchangeable and structurally different methods for evaluating interventions. Diploma thesis, Freie Universität, Berlin, Germany.
68. M.Schultze,, T.Koch,, & M.Eid, (2015). The effects of nonindependent rater sets in multilevel–multitrait–multimethod models. Structural Equation Modeling: A Multidisciplinary Journal, 22, 439–448. doi:10.1080/10705511.2014.937675
69. R.E.Schumacker,, & G.A.Marcoulides, (1998). Interaction and nonlinear effects in structural equation modeling. Mahwah, NJ: Lawrence Erlbaum Associates.
70. S.R.Searle, (2006). Linear models for unbalanced data (Wiley series in probability and statistics). Hoboken, NJ: John Wiley & Sons.
71. S.R.Searle,, G.Casella,, & C.E.McCulloch, (2009). Variance components (Wiley series in probability and statistics). (Vol. 391). Hoboken, NJ: John Wiley & Sons.
72. P.E.Shrout, (1995). Measuring the degree of consensus in personality judgments. In P.E.Shrout & S.T.Fiske (Eds.), Personality research, methods and theory: A festschrift honoring Donald W. Fiske (pp. 79–92). Hillsdale, NJ: Lawrence Erlbaum Associates.
73. D.J.Spiegelhalter,, N.G.Best,, B.P.Carlin,, & A.Van Der Linde, (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 583–639. doi:10.1111/1467-9868.00353
74. R.Steyer, (1989). Models of classical psychometric test theory as stochastic measurement models: Representation, uniqueness, meaningfulness, identifiability, and testability. Methodika, 3, 25–60.
75. R.Steyer,, & M.Eid, (2001). Messen und Testen [Measurement and testing.] (2nd Ed.). Heidelberg, Germany: Springer.
76. Y.-S.Su,, & M.Yajima, (2015). R2jags: A package for running jags from r [Computer software manual]. Retrieved from http://CRAN.R-project.org/package=R2jags (R package version 0.05-01)
77. K.F.Widaman, (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9(1), 1–26. doi:10.1177/014662168500900101
78. F.J.Yammarino, (2003). Modern data analytic techniques for multisource feedback. Organizational Research Methods, 6, 6–14. doi:10.1177/1094428102239423
79. F.J.Yammarino,, & L.E.Atwater, (1997). Do managers see themselves as others see them? Implications of self-other rating agreement for human resources management. Organizational Dynamics, 25, 35–44. doi:10.1016/S0090-2616(97)90035-8
80. A.Ziegler,, M.Dresel,, B.Schober,, & H.Stöger, (2008). Motivationstestbatterie für Schülerinnen und Schüler der Jahrgangsstufen 5-10 (MTB 5-10) [Motivational test battery for students pertaining to grade 5-10.]. Unpublished instrument. Ulm, Germany: Universität.
81. D.W.Zimmermann, (1975). Probability spaces, hilbert spaces, and the axioms of test theory. Psychometrika, 40, 395–412. doi: 10.1007/BF02291765