A cautionary note on modeling multitrait-multirater data arising from ill-structured measurement designs

Organizational Research Methods

Volumn 14, Issue 3, 2011, Pages 503-529

  Putka, Dan J ^a   Lance, Charles E ^b   Le, Huy ^c   McCloy, Rodney A ^d  

^a HUMAN RESOURCES RESEARCH ORGANIZATION   (United States)

^b University of Georgia   (United States)

^c UNIVERSITY OF CENTRAL FLORIDA   (United States)

^d Human Resources Research Organization   (United States)

Author keywords

confirmatory factor analysis; job performance ratings; measurement design

Indexed keywords

EID: 79957847421     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428110362107     Document Type: Article

References (42)

1. Balzer, W.K., Greguras, G.J., & Raymark, P.H. (2004). Multisource feedback. In J. C. Thomas (Ed.), Comprehensive handbook of psychological assessment, Vol. 4: Industrial and organizational assessment (pp. 390-411). Hoboken, NJ: John Wiley.
2. Barr, M.A., & Raju, N.S. (2003). IRT-based assessments of rater effects in multiple-source feedback instruments. Organizational Research Methods, 6, 15-43.
3. Bentler, P.M., & Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
4. Borman, W.C., & Motowidlo, S.J. (1997). Task performance and contextual performance: The meaning for personnel selection research. Human Performance, 10, 99-109.
5. Bracken, D. W., Timmreck, C. W., & Church, A. H. (Eds.). (2001). The handbook of multisource feedback: The comprehensive resource for designing and implementing MSF processes. San Francisco: Jossey-Bass.
6. Brannick, M.T., & Spector, P.E. (1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14, 325-339.
7. Burnham, K.P., & Anderson, D.R. (2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York: Springer.
8. Campbell, J. P., & Knapp, D. J. (Eds.). (2001). Exploring the limits in personnel selection and classification. Mahwah, NJ: Lawrence Erlbaum.
9. Conway, J.M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
10. Conway, J.M., & Huffcutt, A.I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
11. Craig, S.B., & Kaiser, R.B. (2003). Applying item response theory to multisource performance ratings: What are the consequences of violating the independent observations assumption? Organizational Research Methods, 6, 44-60.
12. Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.
13. Eid, M., Nussbeck, F.W., Geiser, C., Gollwitzer, M., Cole, D.A., & Lischetzke, T.A. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230-253.
14. Ewen, A.J., & Edwards, M.R. (2001). Readiness for multisource feedback. In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), The handbook of multisource feedback (pp. 33-47). San Francisco: Jossey-Bass.
15. Greguras, G.J., Robie, C., Schleicher, D.J., & Goff, M. III. (2003). A field study of the effects of rating purpose on the quality of multisource ratings. Personnel Psychology, 56, 1-21.
16. Harris, M.M., Becker, A.S., & Smith, D.E. (1993). Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675-678.
17. Hoffman, B.J., Lance, C.E., Bynum, B.H., & Gentry, W.A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119-151.
18. Hoyt, W.T., & Kerns, M. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
19. Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6, 1-55.
20. Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: SAGE.
21. Kenny, D.A., & Kashy, D.A. (1992). Analysis of multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
22. Knapp, D.J., Sager, C.E., & Tremble, T.R. (2005). Development of experimental Army enlisted personnel selection and classification tests and job performance criteria (Technical Report 1168). Arlington, VA: United States Army Research Institute for the Behavioral and Social Sciences.
23. LaHuis, D.M., & Avis, J.M. (2007). Using multilevel random coefficient modeling to investigate rater effects in performance ratings. Organizational Research Methods, 10, 97-107.
24. Lance, C.E., Noble, C.L., & Scullen, S.E. (2002). A critique of the correlated trait-Correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods, 7, 228-244.
25. Luo, W., & Kwok, O. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182-212.
26. Marsh, H.W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling (pp. 177-198). Thousand Oaks, CA: SAGE.
27. Maurer, T.J., Raju, N.S., & Collins, W.C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693-702.
28. Meyers, J.L., & Beretvas, S.N. (2006). The impact of inappropriate modeling of cross-classified data structures. Multivariate Behavioral Research, 41, 473-497.
29. Mount, M.K., Judge, T.A., Scullen, S.E., Sytsma, M.R., & Hezlett, S.A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
30. Muthén, B.O. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338-354.
31. Muthén, B.O. (1994). Multilevel covariance structure analysis. Sociological Research Methods, 22, 376-398.
32. Pulakos, E.D., & Schmitt, N. (1995). Experience-based and situational interview questions: Studies of validity. Personnel Psychology, 48, 289-308.
33. Putka, D.J., Ingerick, M., & McCloy, R.A. (2008). Integrating traditional perspectives on error in ratings: Capitalizing on advances in mixed effects modeling. Industrial and Organizational Psychology, 1, 167-173.
34. Putka, D.J., Le, H., McCloy, R.A., & Diaz, T. (2008). Ill-structured measurement designs in organizational research: Implications for estimating interrater reliability. Journal of Applied Psychology, 93, 959-981.
35. Rasbash, J., & Browne, W.J. (2008). Non-hierarchical multilevel models. In J. de Leeuw & E. Meijer (Eds.), Handbook of multilevel analysis (pp. 301-334). New York: Springer.
36. Rasbash, J., & Goldstein, H. (1994). Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19, 337-350.
37. Scullen, S.E., Mount, M.K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
38. Scullen, S.E., Mount, M.K., & Judge, T.A. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50-66.
39. Van Iddekinge, C.H., Raymark, P.H., Eidson, C.E., & Attenweiler, W.J. (2004). What do structured selection interviews really measure? The construct validity of behavior description interviews. Human Performance, 17, 71-93.
40. Viswesvaran, C., Ones, D.S., & Schmidt, F.L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
41. Viswesvaran, C., Schmidt, F.L., & Ones, D.S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
42. Woehr, D.J., Sheehan, M.K., & Bennett, W. Jr. (2005). Assessing measurement equivalence across rating sources: A multitrait-multimethod approach. Journal of Applied Psychology, 90, 592-600.