A multilevel CFA-MTMM approach for multisource feedback instruments: Presentation and application of a new statistical model

Structural Equation Modeling

Volumn 23, Issue 1, 2015, Pages 91-110

  Mahlke, Jana ^a   Schultze, Martin ^a   Koch, Tobias ^a   Eid, Michael ^a   Eckert, Regina ^b   Brodbeck, Felix C ^c  

^a FREIE UNIVERSITÄT BERLIN   (Germany)

^b Center for Creative Leadership   (Belgium)

^c UNIVERSITY OF MUNICH   (Germany)

Author keywords

Confirmatory factor analysis; Convergent and discriminant validity; Method effects; Multisource feedback

Indexed keywords

HUMAN RESOURCE MANAGEMENT; MULTIVARIANT ANALYSIS; SOFTWARE RELIABILITY;

COMPLEX DATA STRUCTURES; CONFIRMATORY FACTOR ANALYSIS; CONVERGENT AND DISCRIMINANT VALIDITY; METHOD EFFECTS; MULTILEVEL STRUCTURAL EQUATION; MULTISOURCES; STATISTICAL MODELING; VALIDATION STUDY;

FACTOR ANALYSIS;

EID: 84933056370     PISSN: 10705511     EISSN: 15328007     Source Type: Journal    
DOI: 10.1080/10705511.2014.990153     Document Type: Article

References (48)

1. Atwater, L., Ostroff, C., Yammarino, F. J., & Fleenor, J. W. (1998). Self-other agreement: Does it really matter? Personnel Psychology, 51, 577-598. doi:10.1111/j.1744-6570.1998.tb00252.x
2. Atwater, L., Wang, M., Smither, J.W., & Fleenor, J.W. (2009). Are cultural characteristics associated with the relationship between self and others’ ratings of leadership? Journal of Applied Psychology, 94, 876-886. doi:10.1037/a0014561
3. Beehr, T. A., Ivanitskaya, L., Hansen, C. P., Erofeev, D., & Gudanowski, D. M. (2001). Evaluation of 360 degree feedback ratings: Relationships with each other and with performance and selection predictors. Journal of Organizational Behavior, 22, 775-788. doi:10.1002/job.113
4. Borman, W. C. (1974). The rating of individuals in organizations: An alternate approach. Organizational Behavior and Human Performance, 12, 105-124. doi:10.1016/0030-5073(74)90040-3
5. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016
6. Carretero-Dios, H., Eid, M., & Ruch, W. (2011). Analyzing multitrait-multimethod data with multilevel confirmatory factor analysis: An application to the validation of the State-Trait Cheerfulness Inventory. Journal of Research in Personality, 45, 153-164. doi:10.1016/j.jrp.2010.12.007
7. Center for Creative Leadership. (2010). BENCHMARKS® Technical Manual. Greensboro, NC: Center for Creative Leadership.
8. Conway, J. M., & Huffcutt, A. I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360. doi:10.1207/s15327043hup1004_2
9. Eckert, R., Ekelund, B. Z., Gentry, W. A., & Dawson, J. F. (2010). “I don’t see me like you see me, but is that a problem?” Cultural influences on rating discrepancy in 360-degree feedback instruments. European Journal of Work and Organizational Psychology, 19, 259-278. doi:10.1080/13594320802678414
10. Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241-261. doi:10.1007/BF02294377
11. Eid, M., & Diener, E. (2006). Introduction: The need for multimethod measurement in psychology. In M. Eid & E. Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 3-9). Washington, DC: American Psychological Association.
12. Eid, M., Lischetzke, T., & Nussbeck, F. W. (2006). Structural equation models for multitrait-multimethod data. In M. Eid & E. Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 283-299). Washington, DC: American Psychological Association.
13. Eid, M., Lischetzke, T., Nussbeck, F. W., & Trierweiler, L. I. (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38-60. doi:10.1037/1082-989X.8.1.38
14. Eid, M., Nussbeck, F. W., Geiser, C., Cole, D. A., Gollwitzer, M., & Lischetzke, T. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230-253. doi:10.1037/a0013219
15. Fleenor, J. W., Smither, J. W., Atwater, L., Braddy, P. W., & Sturm, R. E. (2010). Self-other rating agreement in leadership: A review. The Leadership Quarterly, 21, 1005-1034. doi:10.1016/j.leaqua.2010.10.006
16. Garnier-Villarreal, M., Rhemtulla, M., & Little, T. D. (2014). Twomethod planned missing designs for longitudinal research. International Journal of Behavioral Development, 38, 411-422. doi:10.1177/ 0165025414542711
17. Geiser, C., Eid, M., & Nussbeck, F.W. (2008). On the meaning of the latent variables in the CT-C(M-1) model: A comment on Maydeu-Olivares & Coffman (2006). Psychological Methods, 13, 49-57. doi:10.1037/ 1082-989X.13.1.49
18. Ghorpade, J. (2000). Managing five paradoxes of 360-degree feedback. Academy of Management Perspectives, 14(1), 140-150. doi:10.5465/AME.2000.2909846
19. Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological Methods, 11, 323-343. doi:10.1037/1082-989x.11.4.323
20. Greguras, G. J., & Robie, C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968. doi:10.1037//0021-9010.83.6.960
21. Harris, M. M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41, 43-62. doi:10.1111/j.1744-6570.1988.tb00631.x
22. Hoffman, B., Lance, C. E., Bynum, B., & Gentry, W. A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119-151. doi:10.1111/j.1744-6570.2009.01164.x
23. House, R. J., Hanges, P. J., Javidan, M., Dorfman, P., & Gupta, V. (Eds.). (2004). Leadership, culture, and organizations: The GLOBE study of 62 societies. Thousand Oaks, CA: Sage.
24. Hox, J. J., & Maas, C. J. M. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8, 157-174. doi:10.1207/S15328007SEM0802_1
25. Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172. doi:10.1037//0033-2909.112.1.165
26. Kets de Vries, M. F. R., Vrignaud, P., & Florent-Treacy, E. (2004). The Global Leadership Life Inventory: Development and psychometric properties of a 360-degree feedback instrument. The International Journal of Human Resource Management, 15, 475-492. doi:10.1080/0958519042000181214
27. Koch, T., Eid, M., & Lochner, K. (in press). Multitrait-multimethodanalysis: The psychometric foundation of multitrait-multimethod (CFAMTMM) models. In P. Irwing, T. Booth, & D. Hughes (Eds.), Handbook of psychometric testing. London: John Wiley & Sons.
28. Kulas, J. T., & Finkelstein, L. M. (2007). Content and reliability of discrepancy-defined self-awareness in multisource feedback. Organizational Research Methods, 10, 502-522. doi:10.1177/ 1094428107301100
29. Lance, C. E. (1994). Test of a latent structure of performance ratings derived from Wherry’s (1952) theory of rating. Journal of Management, 20, 757-771. doi:10.1177/014920639402000404
30. Lebreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K., & James, L. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6, 80-128. doi:10.1177/1094428102239427
31. Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, 151-173. doi:10.1207/S15328007SEM0902_1
32. Lombardo, M. M., McCauley, C. D., McDonald-Mann, D., & Leslie, J. B. (1999). BENCHMARKS® developmental reference points. Greensboro, NC: Center for Creative Leadership.
33. Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361. doi:10.1177/ 014662168901300402
34. McCauley, C. D., Lombardo, M. M., & Usher, C. J. (1989). Diagnosing management development needs: An instrument based on how managers develop. Journal of Management, 15, 389-403. doi:10.1177/014920638901500303
35. Morgeson, F. P., Mumford, T. V., & Campion, M. A. (2005). Coming full circle: Using research and practice to address 27 questions about 360-degree feedback programs. Consulting Psychology Journal: Practice and Research, 57, 196-209. doi:10.1037/1065-9293.57.3.196
36. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-577. doi:10.1111/j.1744-6570.1998.tb00251.x
37. Muthén, L. K., & Muthén, B. O. (1998-2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén.
38. Putka, D. J., Lance, C. E., Le, H., & McCloy, R. A. (2011). A cautionary note on modeling multitrait-multirater data arising from ill-structured measurement designs. Organizational Research Methods, 14, 503-529. doi:10.1177/1094428110362107
39. Ruch, W., Köhler, G., & van Thriel, C. (1996). Assessing the temperamental basis of the sense of humor: Construction of the facet and standard trait forms of the State-Trait Cheerfulness Inventory-STCI. International Journal of Humor Research, 9, 303-339. doi:10.1515/ humr.1996.9.3-4.303
40. Schultze, M., Koch, T., & Eid, M. (2015). The effects of nonindependent rater sets in multilevel-multitrait-multimethod models. Structural Equation Modeling. doi: 10.1080/10705511.2014.937675
41. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970. doi:10.1037/0021-9010.85.6.956
42. Van der Zee, K. I., Zaal, J. N., & Piekstra, J. (2003). Validation of the multicultural personality questionnaire in the context of personnel selection. European Journal of Personality, 17, 77-100. doi:10.1002/per.483
43. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574. doi:10.1037/0021-9010.81.5.557
44. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding constructlevel convergence. Journal of Applied Psychology, 87, 345-354. doi:10.1037//0021-9010.87.2.345
45. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131. doi:10.1037/0021-9010.90.1.108
46. Woehr, D. J., Sheehan, M. K., & Bennett, W., Jr. (2005). Assessing measurement equivalence across rating sources: A multitrait-multirater approach. Journal of Applied Psychology, 90, 592-600. doi:10.1037/0021-9010.90.3.592
47. Yammarino, F. J. (2003). Modern data analytic techniques for multisource feedback. Organizational Research Methods, 6, 6-14. doi:10.1177/1094428102239423
48. Yammarino, F. J., & Atwater, L. (1997). Do managers see themselves as others see them? Implications of self-other rating agreement for human resources management. Organizational Dynamics, 25(4), 35-44. doi:10.1016/S0090-2616(97)90035-8