Are fit indices really fit to estimate the number of factors with categorical variables? Some cautionary findings via monte carlo simulation

Psychological Methods

Volumn 21, Issue 1, 2016, Pages 93-111

  Garrido, Luis Eduardo ^a   Abad, Francisco José ^b   Ponsoda, Vicente ^b  

^a Universidad Iberoamericana en República Dominicana   (Dominican Republic)

^b UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

Author keywords

Categorical variables; Exploratory structural equation modeling; Fit indices; Number of factors; Parallel analysis

Indexed keywords

HORN; HUMAN; MONTE CARLO METHOD; PSYCINFO; SCIENTIST; THEORETICAL MODEL; STATISTICAL ANALYSIS; STATISTICAL MODEL;

DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; MONTE CARLO METHOD;

EID: 84949845161     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000064     Document Type: Article

References (65)

1. Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397-438. http://dx.doi .org/10.1080/10705510903008204
2. Barendse, M. T., Oort, F. J., & Timmerman, M. E. (2015). Using exploratory factor analysis to determine the dimensionality of discrete responses. Structural Equation Modeling, 22, 87-101. http://dx.doi.org/10.1080/10705511.2014.934850
3. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186-203. http://dx.doi.org/10.1207/s15328007sem1302_2
4. Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258. http://dx.doi.org/ 10.1177/0049124192021002005
5. Browne, M. W., MacCallum, R. C., Kim, C. T., Andersen, B. L., & Glaser, R. (2002). When fit indices and residuals are incompatible. Psychological Methods, 7, 403-421. http://dx.doi.org/10.1037/1082-989X.7.4.403
6. Campbell-Sills, L., Liverant, G. I., & Brown, T. A. (2004). Psychometric evaluation of the behavioral inhibition/behavioral activation scales in a large sample of outpatients with anxiety and mood disorders. Psychological Assessment, 16, 244-254. http://dx.doi.org/10.1037/1040-3590.16.3.244
7. Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36, 462-494. http://dx.doi.org/10.1177/0049124108314720
8. Cho, S.-J., Li, F., & Bandalos, D. (2009). Accuracy of the parallel analysis procedure with polychoric correlations. Educational and Psychological Measurement, 69, 748 -759. http://dx.doi.org/10.1177/ 0013164409332229
9. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
10. Curran, P. J., Bollen, K. A., Paxton, P., Kirby, J., & Chen, F. (2002). The noncentral chi-square distribution in misspecified structural equation models: Finite sample results from a Monte Carlo simulation. Multivariate Behavioral Research, 37, 1-36. http://dx.doi.org/10.1207/S15327906MBR3701_01
11. Ferrando, P. J., & Lorenzo-Seva, U. (2000). Unrestricted versus restricted factor analysis of multidimensional test items: Some aspects of the problem and some suggestions. Psicológica, 21, 301-323.
12. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491. http://dx.doi.org/10.1037/1082-989X.9.4.466
13. Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286-299. http://dx.doi.org/10.1037/1040-3590.7.3.286
14. Forero, C., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16, 625-641. http://dx.doi.org/10.1080/10705510903203573
15. Frazier, T. W., & Youngstrom, E. A. (2007). Historical increase in the number of factors measured by commercial tests of cognitive ability:Are we overfactoring? Intelligence, 35, 169-182. http://dx.doi.org/10.1016/j.intell.2006.07.002
16. Garrido, L. E., Abad, F. J., & Ponsoda, V. (2011). Performance of Velicer's minimum average partial factor retention method with categorical vari-ables. Educational and Psychological Measurement, 71, 551-570. http://dx.doi.org/10.1177/0013164410389489
17. Garrido, L. E., Abad, F. J., & Ponsoda, V. (2013). A new look at Horn's parallel analysis with ordinal variables. Psychological Methods, 18, 454-474. http://dx.doi.org/10.1037/a0030005
18. Geranpayeh, A., & Taylor, L. (Eds.). (2013). Examining listening: Research and practice in assessing second language listening. Studies in Language Testing 35. Cambridge, UK: Cambridge University Press.
19. Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16, 319-336. http://dx.doi.org/10.1037/a0024917
20. Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66, 393-416. http://dx.doi.org/10.1177/0013164405282485
21. Herzog, W., Boomsma, A., & Reinecke, S. (2007). The model-size effect on traditional and modified tests of covariance structures. Structural Equation Modeling, 14, 361-390. http://dx.doi.org/10.1080/ 10705510701301602
22. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179 -185. http://dx.doi.org/10.1007/BF02289447
23. Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453. http://dx.doi.org/10.1037/1082-989X.3.4.424
24. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. http://dx.doi.org/10.1080/ 10705519909540118
25. Jöreskog, K. G., & Moustaki, I. (2001). Factor analysis of ordinal variables:A comparison of three approaches. Multivariate Behavioral Research, 36, 347-387. http://dx.doi.org/10.1207/S15327906347-387
26. Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151. http://dx.doi.org/10.1177/001316446002000116
27. Kenny, D. A., & McCoach, D. B. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10, 333-351. http://dx.doi.org/10.1207/S15328007SEM1003_1
28. Lei, P. W. (2009). Evaluating estimation methods for ordinal data in structural equation modeling. Quality & Quantity: International Journal of Methodology, 43, 495-507. http://dx.doi.org/10.1007/s11135-007-9133-z
29. Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255-268. http://dx.doi.org/10.2307/2532051
30. Lorenzo-Seva, U., Timmerman, M. E., & Kiers, H. A. L. (2011). The Hull method for selecting the number of common factors. Multivariate Behavioral Research, 46, 340-364. http://dx.doi.org/10.1080/00273171.2011.564527
31. Mahler, C. (2011). The effects of misspecification type and nuisance variables on the behaviors of population fit indices used in structural equation modeling (Unpublished doctoral dissertation). University of British Columbia, Vancouver, Canada.
32. Marsh, H. W., Hau, K. T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, 181-220. http://dx.doi .org/10.1207/s15327906mbr3302_1
33. Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules:Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Structural Equation Modeling, 11, 320-341. http://dx.doi.org/ 10.1207/s15328007sem1103_2
34. Marsh, H. W., Muthén, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J. S., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students' evaluations of university teaching. Structural Equation Modeling, 16, 439-476. http://dx.doi.org/10.1080/10705510903008220
35. MathWorks, Inc. (2010). MATLAB (Version R2010a) [Computer software]. Natick, MA: The MathWorks, Inc.
36. Meyers, L. S., Gamst, G., & Guarino, A. (2006). Applied multivariate research: Design and interpretation. Thousand Oaks, CA: Sage.
37. Morata-Ramírez, M. A., & Holgado-Tello, F. P. (2013). Construct validity of Likert scales through confirmatory factor analysis: A simulation study comparing different methods of estimation based on Pearson and polychoric correlations. International Journal of Social Science Studies, 1, 54-61. http://dx.doi.org/10.11114/ijsss.v1i1.27
38. Mulaik, S. A., & Millsap, R. E. (2000). Doing the four-step right. Structural Equation Modeling, 7, 36 -73. http://dx.doi.org/10.1207/S15328007SEM0701_02
39. Muthén, B. (1978). Contributions to factor analysis of dichotomous variables. Psychometrika, 43, 551-560. http://dx.doi.org/10.1007/BF02293813
40. Muthén, B. (1998 -2004). Mplus technical appendices. Los Angeles, CA:Muthén & Muthén.
41. Muthén, B., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38, 171-189. http://dx.doi .org/10.1111/j.2044-8317.1985.tb00832.x
42. Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9, 599-620. http://dx.doi.org/10.1207/S15328007SEM0904_8
43. Muthén, L. K., & Muthén, B. O. (1998 -2010). Mplus user's guide (6th ed.). Los Angeles, CA: Author.
44. Myers, N. D. (2013). Coaching competency and (exploratory) structural equation modeling: A substantive-methodological synergy. Psychology of Sport and Exercise, 14, 709-718. http://dx.doi.org/10.1016/j .psychsport.2013.04.008
45. Nestler, S. (2013). A Monte Carlo study comparing PIV, ULS and DWLS in the estimation of dichotomous confirmatory factor analysis. British Journal of Mathematical and Statistical Psychology, 66, 127-143. http://dx.doi.org/10.1111/j.2044-8317.2012.02044.x
46. Nye, C. D., & Drasgow, F. (2011). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14, 548-570. http://dx.doi.org/10.1177/1094428110368562
47. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460. http://dx.doi.org/ 10.1007/BF02296207
48. Patil, V. H., Singh, S. N., Mishra, S., & Donavan, D. T. (2008). Efficient theory development and factor retention criteria: Abandon the "eigenvalue greater than one" criterion. Journal of Business Research, 61, 162-170. http://dx.doi.org/10.1016/j.jbusres.2007.05.008
49. Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48, 28-56. http://dx.doi.org/10.1080/00273171.2012.710386
50. Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. http://dx.doi.org/10.1037/a0029315
51. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17). Richmond, VA:Psychometric Society. Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf
52. Sanne, B., Torsheim, T., Heiervang, E., & Stormark, K. M. (2009). The Strengths and Difficulties Questionnaire in the Bergen Child Study:A conceptually and methodically motivated structural analysis. Psychological Assessment, 21, 352-364. http://dx.doi.org/10.1037/a0016317
53. Saris, W. E., Satorra, A., & van der Veld, W. M. (2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling, 16, 561-582. http://dx.doi.org/10.1080/ 10705510903203433
54. Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis. Educational and Psychological Measurement, 72, 910-932. http://dx.doi.org/10.1177/0013164412452564
55. Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with categorical data. British Journal of Mathematical and Statistical Psychology, 66, 201-223. http://dx.doi.org/10.1111/j.2044-8317.2012.02049.x
56. Sharma, S., Mukherjee, S., Kumar, A., & Dillon, W. R. (2005). A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. Journal of Business Research, 58, 935-943. http://dx.doi.org/10.1016/j.jbusres.2003.10.007
57. Takane, Y., & de Leeuw, J. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393-408. http://dx.doi.org/10.1007/BF02294363
58. Tepper, K., & Hoyle, R. H. (1996). Latent variable models of need for uniqueness. Multivariate Behavioral Research, 31, 467-494. http://dx .doi.org/10.1207/s15327906mbr3104_4
59. Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209-220. http://dx.doi.org/10.1037/a0023353
60. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41, 321-327. http://dx.doi .org/10.1007/BF02293557
61. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at seventy (pp. 41-71). New York, NY: Kluwer Academic/Plenum Press. http://dx.doi.org/ 10.1007/978-1-4615-4397-8_3
62. Widaman, K. F. (1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263-311. http://dx.doi.org/10.1207/s15327906mbr2803_1
63. Yang, Y., & Xia, Y. (2015). On the number of factors in exploratory factor analysis for ordered categorical data. Behavior Research Methods, 47, 756-772. http://dx.doi.org/10.3758/s13428-014-0499-2
64. Yang-Wallentin, F., Jöreskog, K. G., & Luo, H. (2010). Confirmatory factor analysis of ordinal variables with misspecified models. Structural Equation Modeling, 17, 392-423. http://dx.doi.org/10.1080/10705511.2010.489003
65. Yu, C. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes (Unpublished doctoral dissertation). University of California, Los Angeles, Los Angeles, CA.