An examination of g-theory methods for modeling multitrait-multimethod data: Clarifying links to construct validity and confirmatory factor analysis

Organizational Research Methods

Volumn 15, Issue 1, 2012, Pages 134-161

  Woehr, David J ^a   Putka, Dan J ^b   Bowler, Mark C ^c  

^a UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

^b HUMAN RESOURCES RESEARCH ORGANIZATION   (United States)

^c EAST CAROLINA UNIVERSITY   (United States)

Author keywords

confirmatory factor analysis; generalizability theory; multitrait multimethod

Indexed keywords

EID: 84055185276     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428111408616     Document Type: Article

References (54)

1. Arthur W.,Woehr D. J.,Maldegen R.Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox.Journal of Management. 2000;26:813-835.
2. Baltes B. B.,Bauer C. B.,Bajdo L.,Parker C. P.The use of multitrait-multimethod data for detecting nonlinear relationships: The case of psychological climate and job satisfaction.Journal of Business and Psychology. 2002;17:3-17.
3. Bell J. F.Generalizability theory: The software problem.Journal of Educational Behavior and Statistics. 1985;10:19-29.
4. Bost J. E.The effects of correlated errors on generalizability and dependability coefficients.Applied Psychological Measurement. 1995;19:191-203.
5. Bowler M. C.,Woehr D. J.A meta-analytic investigation of assessment center construct validity.Journal of Applied Psychology. 2006;91:1114-11124.
6. Bowler M. C.,Woehr D. J.Assessment center construct-related validity: Stepping beyond the MTMM matrix.Journal of Vocational Behavior. 2009;75:173-182.
7. Box G. E. P.Some theorems on quadratic forms applied in the study of analysis of variance problems II: Effects of inequality of variance and of correlation between errors in the two way classification.Annals of Mathematical Statistics. 1954;29:885-891.
8. Brennan R. L.Generalizability theory. New York: Springer-Verlag; 2001:.
9. Campbell D. T.,Fiske D. W.Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin. 1959;56:81-105.
10. Cresswell S. L.,Eklund R. C.The convergent and discriminant validity of burnout measures in sport: A multi-trait/multi-method analysis.Journal of Sport Sciences. 2006;24:209-220.
11. Cronbach L. J.,Gleser G. C.,Nanda H.,Rajaratnam N.The dependability of behavioral measurements.Theory of generalizability for scores and profiles. New York: John Wiley; 1972:.
12. Cronbach L. J.,Linn R. L.,Brennan R. L,Haertel E. H.Generalizability analyses for performance assessments of student achievement or school effectiveness.Educational and Psychological Measurement. 1997;57:373-399.
13. DeShon R. P.Restricted maximum likelihood estimation of variance components in generalizability theory: Overcoming balanced design requirements. Paper presented at the 10th annual conference of the Society of Industrial and Organizational Psychology; 1995Orlando, FL; 1995.
14. DeShon R. P.A cautionary note on measurement error corrections in structural equation models.Psychological Methods. 1998;3:412-423.
15. DeShon R. P.Measuring and analyzing behavior in organizations. Drasgow F.Schmitt N., ed. San Francisco: Jossey-Bass; 2002:189-220.
16. Dickinson T. L.Brooks Air Force Base, TX: Air Force Systems Command; 1986:.
17. Guilford J. P.Psychometric methods. New York: John Wiley; 1954:.
18. Haertel E. H.Educational measurement. 2006. Reliability. In BrennanR. L.(Ed.),(pp. 65-110). 4th ed.Westport, CT: American Council on Education and Praeger Publishers.,2006:65-110.
19. Halbesleben J. R. B.,Demerouti E.The construct validity of an alternative measure of burnout: Investigating the English translation of the Oldenburg Burnout Inventory.Work and Stress. 2005;19:208-220.
20. Hoffman D. J.,Woehr D. J.Disentangling the meaning of multisource feedback source and dimension factors.Personnel Psychology. 2009;62:735-765.
21. Jackson S.,Brashers D. E.Random factors in ANOVA. Thousand Oaks, CA: Sage; 1994:.
22. Kane M.The precision of measurements.Applied Measurement in Education. 1996;9:355-379.
23. Kane M.Inferences about variance components and reliability-generalizability coefficients in the absence of random sampling.Journal of Educational Measurement. 2002;39:165-181.
24. Kenny D. A.,Kashy D. A.Analysis of multitrait-multimethod matrix by confirmatory factor analysis.Psychological Bulletin. 1992;112:165-172.
25. King L.,Hunter J. E.,Schmidt F. L.Halo in a multidimensional forced-choice performance evaluation scale.Journal of Applied Psychology. 1980;65:507-516.
26. Kraiger K.,Teachout M. S.Generalizability theory as construct-related evidence of the validity of performance ratings.Human Performance. 1990;3:19-35.
27. Lance C. E.,Lambert T. A.,Gewin A. G.,Lievens F.,Conway J. M.Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings.Journal of Applied Psychology. 2004;89:377-385.
28. Lance C. E.,Noble C. L.,Scullen S. E.A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data.Psychological Methods. 2002;7:228-244.
29. Lance C. E.,Woehr D. J.,Meade A. W.Case study: A Monte Carlo investigation of assessment center construct validity models.Organizational Research Methods. 2007;10:430-448.
30. Lane S.,Stone C.Educational measurement. Performance assessment. In BrennanR. L. (Ed.) 4th ed.Westport, CT: American Council on Education and Praeger Publishers. 2006:387-431.
31. Lievens F.Assessors and use of assessment centre dimensions: A fresh look at a troubling issue.Journal of Organizational Behavior. 2001;22:203-221.
32. Lievens F.Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects.Journal of Applied Psychology. 2002;87:675-686.
33. Marcoulides G. A.An alternative method for estimating variance components in generalizability theory.Psychological Reports. 1990;66:102-109.
34. Marcoulides G. A.Estimating variance components in generalizability theory: The covariance structure analysis approach.Structural Equation Modeling. 1996;3:290-299.
35. Marsh H. W.Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions.Applied Psychological Measurement. 1989;13:335-361.
36. Marsh H. W.,Bailey M.Confirmatory factor analyses of multitrait-multimethod data: A comparison of alternate models.Applied Psychological Measurement. 1991;15:47-70.
37. Marsh H. W.,Byrne B. M.,Craven R.Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance.Multivariate Behavioral Research. 1993;27:489-507.
38. Marsh H. W.,Grayson D.Latent variable models of multitrait-multimethod data Structural equation modeling: Concepts, issues, and applications. Hoyle R. H., ed. Thousand Oaks, CA: Sage; 1995:177-198.
39. Mitte K.,Kämpfe N.Personality and the four faces of positive affect: A multitrait-multimethod analysis using self- and peer-report.Journal of Research in Personality. 2008;42:1370-1375.
40. Putka D. J.,Lance C. E.,Le H.,McCloy R. A.A cautionary note on modeling multitrait-multirater data arising from ill-structured measurement designs.Organizational Research Methods. 2011;14:503-529.
41. Putka D. J.,Le H.,McCloy R. A.,Diaz T.Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability.Journal of Applied Psychology. 2008;93:959-981.
42. PutkaD. J.McCloyR. A. (2008, February). Estimating variance components in SPSS and SAS: An annotated reference guide.Alexandria, VA: Human Resources Research Organization. Retrieved fromhttp://www.humrro.org/corpsite/sites/default/files/dputkaFiles/EstimatingVarianceComponentsinSPSSandSAS.pdf.
43. Putka D. J.,Sackett P. R.Reliability and validity.Handbook of employee selection. Farr J.L.Tippins N. T., ed. New York: Routledge; 2010:9-49.
44. Raykov T.,Marcoulides G. A.Estimation of generalizability coefficients via a structural equation modeling approach to scale reliability estimation.International Journal of Testing. 2006;6:81-95.
45. Schmitt N.,Stults D. M.Methodological review: Analysis of multitrait-multimethod matrices.Applied Psychological Measurement. 1986;10:1-22.
46. Scullen S. E.Using confirmatory factor analysis of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices.Organizational Research Methods. 1999;2:275-292.
47. Searle S. R.,Casella G.,McCulloch C. E.Variance components. New York: John Wiley; 1992:.
48. Shavelson R. J.,Baxter G. P.,Gao X.Sampling variability of performance assessments.Journal of Educational Measurement. 1993;30:215-232.
49. Shavelson R. J.,Webb N. M.Generalizability theory: A primer. Newbury Park, CA: Sage; 1991:.
50. Smith P. L.,Luecht R. M.Correlated effects in generalizability studies.Applied Psychological Measurement. 1992;16:229-235.
51. Stanley J. C.Analysis of unreplicated three-way classifications, with applications to rater bias and trait independence.Psychometrika. 1961;26:205-219.
52. Webb N. M,Schlackman J.,Sugrue B.The dependability and interchangeability of assessment methods in science.Applied Measurement in Education. 2000;13:277-301.
53. Webb N. M.,Shavelson R. J.Multivariate generalizability of general educational developmental ratings.Journal of Educational Measurement. 1981;18:13-22.
54. Woehr D. J.,Sheehan M. K.,Bennett W.Assessing measurement equivalence across rating sources: A multitrait-multimethod approach.Journal of Applied Psychology. 2005;90:592-600.