Ignoring Clustering in Confirmatory Factor Analysis: Some Consequences for Model Fit and Standardized Parameter Estimates

Multivariate Behavioral Research

Volumn 49, Issue 6, 2014, Pages 518-543

  Pornprasertmanit, Sunthud ^a   Lee, Jaehoon ^a   Preacher, Kristopher J ^b  

^a TEXAS TECH UNIVERSITY   (United States)

^b VANDERBILT UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84914152543     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2014.933762     Document Type: Article

References (100)

1. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
2. Ansari, A., Jedidi, K., & Dube, L. (2002). Heterogeneous factor analysis models: A Bayesian approach. Psychometrika, 67, 49–78.
3. Ansari, A., Jedidi, K., & Jagpal, S. (2000). A hierarchical Bayesian methodology for treating heterogeneity in structural equation models. Marketing Science, 19, 328–347.
4. Asparouhov, T. (2005). Sampling weights in latent variable modeling. Structural Equation Modeling, 12, 411–434.
5. Asparouhov, T., & Muthén, B. (2005). Multivariate statistical modeling with survey data. Proceedings of the Federal Committee on Statistical Methodology Research Conference. Retrieved from http://www.fcsm.sites.usa.gov/files/2014/05/2005FCSM_Asparouhov_Muthen_IIA.pdf
6. Babakus, E., Bienstock, C.C., & Van Scotter, J.R. (2004). Linking perceived quality and customer satisfaction to store traffic and revenue growth. Decision Sciences, 35, 713–737.
7. Bell, S.J., Mengüç, B., & Stefani, S.L. (2004). When customers disappoint: A model of relational internal marketing and customer complaints. Journal of the Academy of Marketing Science, 32, 112–126.
8. Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
9. Bollen, K.A. (1989). Structural equations with latent variables. New York, NY: Wiley.
10. Breevaart, K., Bakker, A.B., Demerouti, E., & Hetland, J. (2012). The measurement of state work engagement: A multilevel factor analytic study. European Journal of Psychological Assessment, 28, 305–312.
11. Cassidy, D.J., Hestenes, L.L., Hegde, A., Hestenes, S., & Mims, S. (2005). Measurement of quality in preschool child care classrooms: An exploratory and confirmatory factor analysis of the early childhood environment rating scale-revised. Early Childhood Research Quarterly, 20, 345–360.
12. Chen, Q., Kwok, O.M., Luo, W., & Willson, V.L. (2010). The impact of ignoring a level of nesting structure in multilevel growth mixture models: A Monte Carlo study. Structural Equation Modeling, 17, 570–589.
13. Cheung, M.W.-L., & Au, K. (2005). Applications of multilevel structural equation modeling to cross-cultural research. Structural Equation Modeling, 12, 598–619.
14. Cheung, M.W.-L., Leung, K., & Au, K. (2006). Evaluating multilevel models in cross-cultural research: An illustration with social axioms. Journal of Cross-Cultural Psychology, 37, 522–541.
15. Chou, C., Bentler, P.M., & Pentz, M.A. (2000). A two-stage approach to multilevel structural equation models: Application to longitudinal data. In T.D. Little, K.U. Schnabel, & J. Baumert (Eds.), Modeling longitudinal and multilevel data: Practical issues, applied approaches and specific examples (pp. 33–50). Mahwah, NJ: Erlbaum.
16. Collins, C.J., & Smith, K.G. (2006). Knowledge exchange and combination: The role of human resource practices in the performance of high-technology firms. Academy of Management Journal, 49, 544–560.
17. Collins, L.M., Schafer, J.L., & Kam, C.M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330–351.
18. Detert, J.R., Schroeder, R.G., & Cudeck, R. (2003). The measurement of quality management culture in schools: Development and validation of the SQMCS. Journal of Operations Management, 21, 307–328.
19. Dyer, N.G., Hanges, P.J., & Hall, R.J. (2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. Leadership Quarterly, 16, 149–167.
20. Ebesutani, C., Okamura, K., Higa-McMillan, C., & Chorpita, B.F. (2011). A psychometric analysis of the positive and negative affect schedule for children-parent version in a school sample. Psychological Assessment, 23, 406–416.
21. Ehrhart, M.G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61–94.
22. Flora, D.B., & Curran, P.J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466–491.
23. Floyd, F.J., & Widaman, K.F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299.
24. Garb, H.N., Wood, J.M., & Fiedler, E.R. (2011). A comparison of three strategies for scale construction to predict a specific behavioral outcome. Assessment, 18, 399–411.
25. Geldhof, G.J., Preacher, K.J., & Zyphur, M.J. (2014). Reliability estimation in multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72–91.
26. Gibson, C.B., & Birkinshaw, J. (2004). The antecedents, consequences, and mediating role of organizational ambidexterity. Academy of Management Journal, 47, 209–226.
27. Glisson, C., & James, L.R. (2002). The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior, 23, 767–794.
28. Gong, Y., Chang, S., & Cheung, S.-Y. (2010). High performance work system and collective OCB: A collective social exchange perspective. Human Resource Management Journal, 20, 119–137.
29. González-Romá, V., Peiró, J.M., & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength. Journal of Applied Psychology, 87, 465–473.
30. Hägglund, G. (1982). Factor analysis by instrumental variables methods. Psychometrika, 47, 209–222.
31. Han, J., Chou, P., Chao, M., & Wright, P.M. (2006). The HR competencies-HR effectiveness link: A study in Taiwanese high-tech companies. Human Resource Management, 45, 391–406.
32. Hatami, G., Motamed, N., & Ashrafzadeh, M. (2010). Confirmatory factor analysis of Persian adaptation of multidimensional students’ life satisfaction scale (MSLSS). Social Indicators Research, 98, 265–271.
33. Håvold, J.I. (2007). National cultures and safety orientation: A study of seafarers working for Norwegian shipping companies. Work and Stress, 21, 173–195.
34. Hoegl, M., & Gemuenden, H.G. (2001). Teamwork quality and the success of innovative projects: A theoretical concept and empirical evidence. Organization Science, 12, 435–449.
35. Hoegl, M., Parboteeah, K.P., & Munson, C.L. (2003). Team-level antecedents of individuals’ knowledge networks. Decision Sciences, 34, 741–770.
36. Hoogland, J.J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling. Sociological Methods & Research, 26, 329–367.
37. Jackson, C.J., Levine, S.Z., & Furnham, A. (2003). Gray's model of personality and aggregate level factor analysis. European Journal of Personality, 17, 397–411.
38. Jak, S., Oort, F.J., & Dolan, C.V. (2013). A test for cluster bias: Detecting violations of measurement invariance across clusters in multilevel data. Structural Equation Modeling, 20, 265–282.
39. Jedidi, K., & Ansari, A. (2001). Bayesian structural equation models for multilevel data. In G.A. Marcoulides & R.E. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 129–157). Mahwah, NJ: Erlbaum.
40. Jöreskog, K.G., & Sörbom, D. (1981). LISREL V: Analysis of linear structural relationships by maximum likelihood and least squares methods: Chicago, IL: National Educational Resources Chicago.
41. Julian, M.W. (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8, 325–352.
42. Kaplan, D., & Ferguson, A.J. (1999). On the utilization of sample weights in latent variable models. Structural Equation Modeling, 6, 305–321.
43. Keller, R.T. (2001). Cross-functional project groups in research and new product development: Diversity, communications, job stress, and outcomes. Academy of Management Journal, 44, 547–555.
44. Kim, E.S., Kwok, O., & Yoon, M. (2012). Testing factorial invariance in multilevel data: A Monte Carlo study. Structural Equation Modeling, 19, 250–267.
45. King, J.E., & Figueredo, A.J. (1997). The five-factor model plus dominance in chimpanzee personality. Journal of Research in Personality, 31, 257–271.
46. Klein, K.J., & Kozlowski, S.W. J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3, 211–236.
47. Law, B.M. F., Shek, D.T. L., & Ma, C.M. S. (2011). Exploration of the factorial structure of the revised personal functions of the volunteerism scale for Chinese adolescents. Social Indicators Research, 100, 517–537.
48. Lüdtke, O., Marsh, H.W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual model: Accuracy-bias trade-offs in full and partial error correction models. Psychological Methods, 13, 203–229.
49. Lüdtke, O., Marsh, H.W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203–229.
50. Marsh, H.W., Lüdtke, O., Nagengast, B., Trautwein, U., Morin, A.J. S., Abduljabbar, A.S., & Köller, O. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47, 106–124.
51. Mathisen, G.E., Einarsen, S., Jørstad, K., & Brønnick, K.S. (2004). Climate for work group creativity and innovation: Norwegian validation of the team climate inventory (TCI). Scadinavian Journal of Psychology, 45, 383–392.
52. Mathisen, G.E., Torsheim, T., & Einarsen, S. (2006). The team-level model of climate for innovation: A two-level confirmatory factor analysis. Journal of Occupational and Organizational Psychology, 79, 23–35.
53. Merrell, K.W., Felver-Gant, J.C., & Tom, K.M. (2011). Development and validation of a parent report measure for assessing socio-emotional competencies of children and adolescents. Journal of Child and Family Studies, 20, 529–540.
54. Moerbeek, M. (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39, 129–149.
55. Muthén, B.O. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557–585.
56. Muthén, B.O. (1990). Mean and covariance structure analysis of hierarchical data. Los Angeles, CA: UCLA Statistics Series, #62.
57. Muthén, B.O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 376–398.
58. Muthén, B.O., & Asparouhov, T. (2009). Growth mixture modeling: Analysis with non-Gaussian random effects. In G. Fitzmaurice, M. Davidian, G. Verbeke & G. Molenberghs (Eds.), Longitudinal data analysis (pp. 143–165). Boca Raton, FL: Chapman & Hall/CRC.
59. Muthén, B.O., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P.V. Marsden (Ed.), Sociological methodology (pp. 267–316). Washington, DC: American Sociological Association.
60. Muthén, L.K., & Muthén, B.O. (1998–2013). Mplus user's guide: Statistical analysis with latent variables (7th ed.). Los Angeles, CA: Author.
61. Nelson, J.M., Canivez, G.L., Lindstrom, W., & Hatt, C.V. (2007). Higher-order exploratory factor analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439–456.
62. Noortgate, W.V., Opdenakker, M.C., & Onghena, P. (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16, 281–303.
63. Oliver, J.E., Jose, P.E., & Brough, P. (2006). Confirmatory factor analysis of the work locus of control scale. Educational and Psychological Measurement, 66, 835–851.
64. Opdenakker, M.C., & Van Damme, J. (2000). Effects of schools, teaching staff and classes on achievement and well-being in secondary education: Similarities and differences between school outcomes. School Effectiveness and School Improvement, 11, 165–196.
65. Organization for Economic Cooperation and Development. (2003). PISA 2003 technical report. Paris, France: Author.
66. Organization for Economic Cooperation and Development. (2005). PISA 2003 data analysis manual. Paris, France: Author.
67. Patterson, M.G., West, M.A., Shackleton, V.J., Dawson, J.F., Lawthom, R., Maitlis, S., … Wallace, A.M. (2005). Validating the organizational climate measure: Links to managerial practices, productivity and innovation. Journal of Organizational Behavior, 26, 379–408.
68. Philips, G.A., Shadish, W.R., Murray, D.M., Kubik, M., Lytle, L.A., & Birnbaum, A.S. (2006). The center for epidemiologic studies depression scale with a young adolescent population: A confirmatory factor analysis. Multivariate Behavioral Research, 41, 147–163.
69. Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). Generalized multilevel structural equation modeling. Psychometrika, 69, 167–190.
70. Rabe-Hesketh, S., Skrondal, A., & Zheng, X. (2007). Multilevel structural equation modeling. In S.Y. Lee (Ed.), Handbook of latent variable and related models (pp. 209–227). Amsterdam, The Netherlands: Elsevier.
71. Raspa, M., Bailey, D.B., Jr., Olmsted, M.G., Nelson, R., Robinson, N., Simpson, M.E., … Houts, R. (2010). Measuring family outcomes in early intervention: Findings from a large-scale assessment. Exceptional Children, 76, 496–510.
72. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
73. Raudenbush, S.W., & Sampson, R. (1999). Assessing direct and indirect effects in multilevel designs with latent variables. Sociological Methods and Research, 28, 123–153.
74. Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173–184.
75. Raykov, T. (2004). Estimation of maximal reliability: A note on a covariance structure modelling approach. British Journal of Mathematical and Statistical Psychology, 57, 21–27.
76. Raykov, T., & Shrout, P.E. (2002). Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195–212.
77. Riordan, C.M., Vandenberg, R.J., & Richardson, H.A. (2005). Employee involvement climate and organizational effectiveness. Human Resource Management, 44, 471–488.
78. Robert, C., & Wasti, S.A. (2002). Organizatinal individualism and collectivism: Theoretical development and an empirical test of a measure. Journal of Management, 28, 544–566.
79. Ryu, E., & West, S.G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16, 583–601.
80. Salanova, M., Agut, S., & Peiró, J.M. (2005). Linking organizational resources and work engagement to employee performance and customer loyalty: The mediation of service climate. Journal of Applied Psychology, 90, 1217–1227.
81. Schaubroeck, J., Lam, S.S. K., & Cha, S.E. (2007). Embracing transformational leadership: Team values and the impact of leader behavior on team performance. Journal of Applied Psychology, 92, 1020–1030.
82. Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal and structural equation models. Boca Raton, FL: Chapman & Hall/CRC.
83. Stapleton, L.M. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling, 9, 475–502.
84. Stapleton, L.M. (2006). An assessment of practical solutions for structural equation modeling with complex sample data. Structural Equation Modeling, 13, 28–58.
85. Stapleton, L.M. (2008). Variance estimation using replication methods in structural equation modeling with complex sample data. Structural Equation Modeling, 15, 183–210.
86. Steiger, J.H., & Lind, J.C. (1980, May). Statistically based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA.
87. Sterba, S.K. (2009). Alternative model-based and design-based frameworks for inference from samples to populations: From polarization to integration. Multivariate Behavioral Research, 44, 711–740.
88. Subramony, M., Krause, N., Norton, J., & Burns, G.N. (2008). The relationship between human resource investments and organizational performance: A firm-level examination of equilibrium theory. Journal of Applied Psychology, 93, 778–788.
89. Takeuchi, R., Lepak, D.P., Wang, H., & Takeuchi, K. (2007). An empirical examination of the mechanisms mediating between high-performance work systems and the performance of Japanese organizations. Journal of Applied Psychology, 92, 1069–1083.
90. Tucker, C.M., Rice, K.G., Hou, W., Kaye, L.B., Nolan, S.E. M., Grandoit, D.J., … Desmond, F.F. (2011). Development of the motivators of and barriers to health-smart behaviors inventory. Psychological Assessment, 23, 487–503.
91. Tucker, L.R, & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.
92. van der Vegt, G.S., & Bunderson, J.S. (2005). Learning and performance in multidisciplinary teams: The importance of collective team identification. Academy of Management Journal, 48, 532–547.
93. van Mierlo, H., Vermunt, J.K., & Rutte, C.G. (2009). Composing group-level constructs from individual-level survey data. Organizational Research Methods, 12, 368–392.
94. Wang, M.- T., Willett, J.B., & Eccles, J.S. (2011). The assessment of school engagement: Examining dimensionality and measurement invariance by gender and race/ethnicity. Journal of School Psychology, 49, 465–480.
95. Widaman, K.F. (1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263–311.
96. Wilkinson, L., & The Task Force of Statistical Inference. (1999). Statistical methods in psychology journals. American Psychologist, 54, 594–604.
97. Yuan, K.-H., & Bentler, P.M. (2007). Multilevel covariance structure analysis by fitting multiple single-level models. Sociological Methodology, 37, 53–82.
98. Zhou, K.Z., Gao, G.Y., Yang, Z., & Zhou, N. (2005). Developing strategic orientation in China: Antecedents and consequences of market and innovation orientations. Journal of Business Research, 58, 1049–1058.
99. Zohar, D., & Tenne-Gazit, O. (2008). Transformational leadership and group interaction as climate antecedents: A social network analysis. Journal of Applied Psychology, 93, 744–757.
100. Zyphur, M.J., Kaplan, S.A., & Christian, M.S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12, 127–140.