Evaluating Model Fit in Bayesian Confirmatory Factor Analysis With Large Samples: Simulation Study Introducing the BRMSEA

Educational and Psychological Measurement

Volumn 78, Issue 4, 2018, Pages 537-568

  Hoofs, Huub ^a,b   van de Schoot, Rens ^c,d   Jansen, Nicole W H ^a   Kant, IJmert ^a  

^a MAASTRICHT UNIVERSITY   (Netherlands)

^b South Limburg Public Health Service   (Netherlands)

^c UTRECHT UNIVERSITY   (Netherlands)

^d NORTH WEST UNIVERSITY   (South Africa)

Author keywords

Bayesian procedures; factor analysis; model fit; simulation; validity

Indexed keywords

EID: 85043456522     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164417709314     Document Type: Article

References (68)

1. Asparouhov T., (2010, October 29). Deviance information criterion [Online forum comment]. Retrieved from http://www.statmodel.com/discussion/messages/9/6184.html?1288395842
2. Asparouhov T., Muthén B. O., (2010). Bayesian analysis of latent variable models using Mplus (Technical report). Los Angeles, CA: Muthén & Muthén
3. Asparouhov T., Muthén B., Morin A. J. S., (2015). Bayesian structural equation modeling with cross-loadings and residual covariances: Comments on Stromeyer et al. Journal of Management, 41, 1561-1577. doi:10.1177/0149206315591075
4. Barrett P., (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42, 815-824. doi:10.1016/j.paid.2006.09.018
5. Bentler P. M., (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. doi:10.1037/0033-2909.107.2.238
6. Bentler P. M., (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42, 825-829. doi:10.1016/j.paid.2006.09.024
7. Brown T. A., (2015). Confirmatory factor analysis for applied research (2nd ed.). New York, NY: Guilford Press
8. Browne M. W., Cudeck R., (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258. doi:10.1177/0049124192021002005
9. Chen F., Curran P. J., Bollen K. A., Kirby J., Paxton P., (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36, 462-494. doi:10.1177/0049124108314720
10. Cheng Y., Luh W. M., Guo Y. L., (2003). Reliability and validity of the Chinese version of the Job Content Questionnaire in Taiwanese workers. International Journal of Behavioral Medicine, 10, 15-30. doi:10.1207/s15327558ijbm1001_02
11. Cieciuch J., Davidov E., Schmidt P., Algesheimer R., Schwartz S. H., (2014). Comparing results of an exact vs. an approximate (Bayesian) measurement invariance test: A cross-country illustration with a scale to measure 19 human values. Frontiers in Psychology, 5, 982. doi:10.3389/fpsyg.2014.00982
12. Curran P. J., Bollen K. A., Chen F., Paxton P., Kirby J. B., (2003). Finite sampling properties of the point estimates and confidence intervals of the RMSEA. Sociological Methods & Research, 32, 208-252. doi:10.1177/0049124103256130
13. De Araújo T. M., Karasek R. (2008). Validity and reliability of the job content questionnaire in formal and informal jobs in Brazil. SJWEH Supplements, 34(6), 52-59
14. Depaoli S., Van de Schoot R., (2017). Improving transparency and replication in Bayesian statistics: The WAMBS-Checklist. Psychological Methods, 22, 240-261
15. DiStefano C., Hess B., (2005). Using confirmatory factor analysis for construct validation: An empirical review. Journal of Psychoeducational Assessment, 23, 225-241. doi:10.1177/073428290502300303
16. Fan X., Sivo S. A., (2007). Sensitivity of fit indices to model misspecification and model type. Multivariate Behavioral Research, 42, 509-529. doi:10.1080/00273170701382864
17. Gelman A., Carlin J. B., Stern H. S., Rubin D. B., (2014). Bayesian data analysis (3rd ed.). Boca Raton, FL: Chapman & Hall/CRC Press
18. Hallquist M., Wiley J., (2013). MplusAutomation: Automating Mplus model estimation and interpretation. Retrieved from http://CRAN.R-project.org/package=MplusAutomation
19. Heene M., Hilbert S., Freudenthaler H. H., Bühner M., (2012). Sensitivity of SEM fit indexes with respect to violations of uncorrelated errors. Structural Equation Modeling: A Multidisciplinary Journal, 19, 36-50. doi:10.1080/10705511.2012.634710
20. Hoijtink H., Van de Schoot R., (2017). Testing small variance priors using prior-posterior predictive P-values. Psychological Methods. Advance online publication. doi:10.1037/met0000131
21. Hu L., Bentler P. M., (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118
22. Kant I., Bultmann U., Schroer K., Beurskens A., van Amelsvoort L. G. P. M., Swaen G., (2003). An epidemiological approach to study fatigue in the working population: The Maastricht Cohort Study. Occupational and Environmental Medicine, 60(Suppl. 1), i32-i39. doi:10.1136/oem.60.suppl_1.i32
23. Kaplan D., Depaoli S., (2012). Bayesian structural equation modeling. In Hoyle R., (Ed.), Handbook of structural equation modeling (pp. 650-673). New York, NY: Guilford Press
24. Karasek R., (1985). Job Content Questionnaire and user’s guide. Lowell: University of Massachusetts, Department of Work Environment
25. Kenny D., (2015, November 24). Measuring model fit. Retrieved from http://davidakenny.net/cm/fit.htm
26. Kline R. B., (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford
27. Kuhn M., (2015). caret: Classification and regression training. Retrieved from http://CRAN.R-project.org/package=caret
28. Lai K., Green S. B., (2016). The problem with having two watches: Assessment of fit when RMSEA and CFI disagree. Multivariate Behavioral Research, 51, 220-39
29. Lee S.-Y., (2007). Structural equation modeling: A Bayesian approach. Chichester, England: Wiley
30. Lee S.-Y., Song X.-Y., (2004). Evaluation of the Bayesian and maximum likelihood approaches in analyzing structural equation models with small sample sizes. Multivariate Behavioral Research, 39, 653-686. doi:10.1207/s15327906mbr3904_4
31. Levy R., (2011). Bayesian data-model fit assessment for structural equation modeling. Structural Equation Modeling, 18, 663-685. doi:10.1080/10705511.2011.607723
32. Lung F.-W., Chiang T.-L., Lin S.-J., Shu B.-C., Lee M.-C., (2011). Developing and refining the Taiwan birth cohort study (TBCS): Five years of experience. Research in Developmental Disabilities, 32, 2697-2703. doi:10.1016/j.ridd.2011.06.002
33. MacCallum R. C., (2003). 2001 presidential address: Working with imperfect models. Multivariate Behavioral Research, 38, 113-139. doi:10.1207/S15327906MBR3801_5
34. MacCallum R. C., Browne M. W., Sugawara H. M., (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130-149. doi:10.1037/1082-989x.1.2.130
35. MacCallum R. C., Edwards M. C., Cai L., (2012). Hopes and cautions in implementing Bayesian structural equation modeling. Psychological Methods, 17, 340-345. doi:10.1037/a0027131
36. Marcoulides K. M., Yuan K.-H., (2017). New ways to evaluate goodness of fit: A note on using equivalence testing to assess structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 24, 148-153. doi:10.1080/10705511.2016.1225260
37. Marsh H. W., Balla J. R., McDonald R. P., (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410. doi:10.1037/0033-2909.103.3.391
38. Marsh H. W., Hau K.-T., Wen Z., (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11, 391-410. doi:10.1207/s15328007sem1103_2
39. McDonald R. P., Ho M.-H. R., (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7, 64-82. doi:10.1037/1082-989X.7.1.64
40. Millsap R. E., (2007). Structural equation modeling made difficult. Personality and Individual Differences, 42, 875-881. doi:10.1016/j.paid.2006.09.021
41. Millsap R. E., (2011). Statistical approaches to measurement invariance. New York, NY: Routledge
42. Milojev P., Osborne D., Greaves L. M., Barlow F. K., Sibley C. G., (2013). The mini-IPIP6: Tiny yet highly stable markers of big six personality. Journal of Research in Personality, 47, 934-946. doi:10.1016/j.jrp.2013.09.004
43. Muthén B. O., Asparouhov T., (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313-335. doi:10.1037/a0026802
44. Muthén L. K., Muthén B. O., (1998). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén
45. Muthén L. K., Muthén B. O., (2013). BSEM measurement invariance analysis (Mplus Web Notes No. 17). Retrieved from http://www.statmodel.com
46. Pianta R. C., Hamre B. K., La Paro K. M., (2008). The Classroom Assessment Scoring System Manual. Baltimore, MD: Brookes
47. Pelfrene E., Clays E., Moreau M., Mak R., Vlerick P., Kornitzer M., De Backer G., (2003). The job content questionnaire: Methodological considerations and challenges for future research. Archives of Public Health, 61, 53-74
48. R Development Core Team. (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
49. Rindskopf D., (2012). Next steps in Bayesian structural equation models: Comments on, variations of, and extensions to Muthén and Asparouhov (2012). Psychological Methods, 17, 336-396. doi:10.1037/a0027130
50. Rupp A. A., Dey D. K., Zumbo B. D., (2004). To Bayes or not to Bayes, from whether to when: Applications of Bayesian methodology to modeling. Structural Equation Modeling: A Multidisciplinary Journal, 11, 424-451. doi:10.1207/s15328007sem1103_7
51. Saris W. E., Satorra A., Van der Veld W. M., (2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling: A Multidisciplinary Journal, 16, 561-582. doi:10.1080/10705510903203433
52. Savalei V., (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72, 910-932. doi:10.1177/0013164412452564
53. Scheines R., Hoijtink H., Boomsma A., (1999). Bayesian estimation and testing of structural equation models. Psychometrika, 64, 37-52. doi:10.1007/bf02294318
54. Shevlin M., Miles J. N. V., (1998). Effects of sample size, model specification and factor loadings on the GFI in confirmatory factor analysis. Personality and Individual Differences, 25, 85-90. doi:10.1016/S0191-8869(98)00055-5
55. Spiegelhalter D. J., Best N. G., Carlin B. P., Van Der Linde A., (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 583-639. doi:10.1111/1467-9868.00353
56. Steiger J. H., (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180. doi:10.1207/s15327906mbr2502_4
57. Steiger J. H., (2000). Point estimation, hypothesis testing, and interval estimation using the RMSEA: Some comments and a reply to Hayduk and Glaser. Structural Equation Modeling: A Multidisciplinary Journal, 7, 149-162. doi:10.1207/S15328007SEM0702_1
58. Steiger J. H., (2007). Understanding the limitations of global fit assessment in structural equation modelling. Personality and Individual Differences, 42, 893-898. doi:10.1016/j.paid.2006.09.017
59. Steiger J. H., (2014, October). Still crazy after all these years: Complexity, principles, and practice in multivariate statistics. Presented at the annual meeting of the Society of Multivariate Experimental Psychology, Nashville, TN
60. Steiger J. H., Lind J. C., (1980, June). Statistically based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA
61. Stromeyer W. R., Miller J. W., Sriramachandramurthy R., DeMartino R., (2015). The prowess and pitfalls of Bayesian structural equation modeling: Important considerations for management research. Journal of Management Research, 41, 491-520. doi:10.1177/0149206314551962
62. Van de Schoot R., Kaplan D., Denissen J., Asendorpf J. B., Neyer F. J., van Aken M. A. G., (2014). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85, 842-860. doi:10.1111/cdev.12169
63. Van de Schoot R., Kluytmans A., Tummers L., Lugtig P., Hox J., Muthén B. O., (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 4, 770. doi:10.3389/fpsyg.2013.00770
64. Van de Schoot R., Schmidt P., De Beuckelaer A., Lek K., Zondervan-Zwijnenburg M., (2015). Editorial: Measurement invariance. Frontiers in Psychology, 6, 1064. doi:10.3389/fpsyg.2015.01064
65. Van de Schoot R., Winter S., Ryan O., Zondervan-Zwijnenburg M., Depaoli S., (2016). A systematic review of Bayesian papers in psychology: The last 25 years. Psychological Methods, 4(21). doi:10.1037/met0000100
66. West G. W., Taylor A. B., Wu W., (2012). Model fit and model selection in structural equation modelling. In Hoyle R., (Ed.), Handbook of structural equation modeling (pp. 209-246). New York, NY: Guilford Press
67. Yuan K.-H., (2005). Fit indices versus test statistics. Multivariate Behavioral Research, 40, 115-148. doi:10.1207/s15327906mbr4001_5
68. Yuan K.-H., Chan W., Marcoulides G. A., Bentler P. M., (2016). Assessing structural equation models by equivalence testing with adjusted fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 23, 319-330. doi:10.1080/10705511.2015.1065414