Analysis and design of multitrait-multirater performance appraisal studies

Journal of Management

Volumn 22, Issue 1, 1996, Pages 139-162

  Conway, James M ^a  

^a SETON HALL UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0030527034     PISSN: 01492063     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0149-2063(96)90015-X     Document Type: Article

References (60)

1. denotes studies providing MTMR matrices Albrecht P.A., Glaser E.M., Marks J. Validation of a multiple-assessment procedure for managerial personnel. Journal of Applied Psychology. 48:1964;351-360.
2. Bagozzi R.P. Assessing construct validity in personality research: Applications to measures of self-esteem. Journal of Research in Personality. 27:1993;49-87.
3. Bagozzi R.P., Yi Y. Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions of work. Journal of Applied Psychology. 75:1990;547-560.
4. Bagozzi R.P., Yi Y. Multitrait-multimethod matrices in consumer research. Journal of Consumer Research. 17:1991;426-439.
5. Becker T.E., Cote J.A. Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management. 20:1994;625-641 denotes studies providing MTMR matrices.
6. Becker T.E., Vance R.J. Construct validity of three types of organizational citizenship behavior: An illustration of the direct product model with refinements. Journal of Management. 19:1993;663-682.
7. Boomsma A. Nonconvergence, improper solutions, and starting values in LISREL maximum likelihood estimation. Psychometrika. 50:1985;229-242.
8. Borman W.C. Rating of individuals in organizations: An alternate approach. Organizational Behavior and Human Performance. 12:1974;105-124 denotes studies providing MTMR matrices.
9. Borman W.C., Motowidlo S.J. Expanding the criterion domain to include elements of contextual performance. Schmitt N., Borman W.C. Personnel selection in organizations. 1993;71-98 Jossey-Bass, San Francisco.
10. Boruch R.F., Larkin J.D., Wolins L., MacKinney A.C. Alternative methods of analysis: Multitrait-multimethod data. Educational and Psychological Measurement. 30:1970;833-853 denotes studies providing MTMR matrices.
11. Brannick M.T., Spector P.E. Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement. 14:1990;325-339.
12. Browne M.W. The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology. 37:1984;1-21.
13. Browne M.W. MUTMUM user's guide. April, 1992;Department of Psychology, Ohio State University.
14. Byrne B.M., Goffin R.D. Modeling MTMM data from additive and multiplicative covariance structures: An audit of construct validity concordance. Multivariate Behavioral Research. 28:1993;67-96.
15. Campbell D.T., Fiske D.W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin. 56:1959;81-105.
16. Campbell D.T., O'Connell E.J. Method factors in multitrait-multimethod matrices: Multiplicative rather than additive? Multivariate Behavioral Research. 2:1967;409-426.
17. Campbell D.T., O'Connell E.J. Methods as diluting trait relationships rather than adding irrelevant systematic variance. Brinberg D., Kidder L.H. New directions for methodology of social and behavioral science: Forms of validity in research. 1982;93-111 Jossey-Bass, San Francisco, CA.
18. Cote J.A., Buckley M.R. Estimating trait, method, and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research. 24:1987;315-318.
19. Cudeck R. Analysis of correlation matrices using covariance structure models. Psychological Bulletin. 105:1989;317-327.
20. Dickinson T.L., Hassett C.E., Tannenbaum S.I. Work performance ratings: A meta-analysis of multitrait-multimethod studies. 1986, December;Air Force Human Resources Laboratory, Brooks Air Force Base, TX. AFHRL-TP-86-32.
21. Dickinson T.L., Tice T.E. A multitrait-multimethod analysis of scales developed by retranslation. Organizational Behavior and Human Performance. 9:1973;421-438 denotes studies providing MTMR matrices.
22. Dunham R.B., Smith F.J., Blackburn R.S. Validation of the index of organizational reactions with the JDI, the MSQ, and faces scales. Academy of Management Journal. 20:1977;420-432.
23. Elbert N.F. Questionnaire validation by confirmatory factor analysis: An improvement over multitrait-multimethod matrices. Decision Sciences. 10:1979;629-644.
24. Forsythe G.B., McGaghie W.C., Friedman C.P. Construct validity of medical clinical competence measures: A multitrait-multimethod matrix study using confirmatory factor analysis. American Educational Research Journal. 23:1986;315-336 denotes studies providing MTMR matrices.
25. Freedman R.D., Stumpf S.A. Student evaluations of courses and faculty based on a perceived learning criterion: Scale construction, validation, and comparison of results. Applied Psychological Measurement. 2:1978;189-202.
26. Goffin R.D., Jackson D.N. Analysis of multitrait-multirater performance appraisal data: Composite direct product method versus confirmatory factor analysis. Multivariate Behavioral Research. 27:1992;363-385.
27. Gunderson E.K.E., Nelson P.D. Criterion measures for extremely isolated groups. Personnel Psychology. 19:1966;67-80 denotes studies providing MTMR matrices.
28. Gunderson E.K.E., Ryman D.H. Convergent and discriminant validities of performance evaluations in extremely isolated groups. Personnel Psychology. 24:1971;715-724 denotes studies providing MTMR matrices.
29. Harris M.M., Schaubroeck J. A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology. 41:1988;43-62.
30. Holzbach R.L. Rater bias in performance ratings: Superior, self-, and peer ratings. Journal of Applied Psychology. 63:1978;579-588 denotes studies providing MTMR matrices.
31. Joreskog K.G., Sorbom D. LISREL 8 user's reference guide. 1993;Scientific Software International, Chicago, IL.
32. Kavanagh M.J., MacKinney A.C., Wolins L. Issues in managerial performance: Multitrait-multimethod analyses of ratings. Psychological Bulletin. 75:1971;34-49 denotes studies providing MTMR matrices.
33. Kenny D.A. An empirical application of confirmatory factor analyses to the multitrait-multimethod matrix. Journal of Experimental Social Psychology. 12:1976;247-252.
34. Kenny D.A. Correlation and causality. 1979;Wiley, New York.
35. Kenny D.A. The multitrait-multimethod matrix: Design, analysis, and conceptual issues. Shrout P.E., Fiske S.T. Personality, research, methods, and theory: A Festschrift honoring Donald W. Fiske. 1994;111-124 Erlbaum, Hillsdale, NJ.
36. Kenny D.A., Kashy D.A. Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin. 112:1992;165-172.
37. King L.M., Hunter J.E., Schmidt F.L. Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology. 65:1980;507-516 denotes studies providing MTMR matrices.
38. Lance C.E., Teachout M.S., Donnelly T.M. Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology. 77:1967;437-452 denotes studies providing MTMR matrices.
39. Lawler E.E. III. The multitrait-multirater approach to measuring managerial job performance. Journal of Applied Psychology. 51:1967;369-381 denotes studies providing MTMR matrices.
40. Marsh H.W. Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement. 13:1989;335-361.
41. Marsh H.W., Bailey M. Confirmatory factor analysis of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement. 15:1991;47-70.
42. Marsh H.W., Balla J.R., McDonald R.P. Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin. 103:1988;391-410.
43. Marsh H.W., Butler S. Evaluating reading diagnostic tests: An application of confirmatory factor analysis to multitrait-multimethod data. Applied Psychological Measurement. 8:1984;307-320.
44. McDonald R.P., Marsh H.W. Choosing a multivariate model: Noncentrality and goodness of fit. Psychological Bulletin. 107:1990;247-255.
45. Meritt-Haston R.E. Explaining interrater reliability. Dissertation Abstracts International. 45:1984;2723 Michigan State University, denotes studies providing MTMR matrices.
46. Mount M.K. Psychometric properties of subordinate ratings of managerial performance. Personnel Psychology. 37:1984;687-702 denotes studies providing MTMR matrices.
47. Murphy K.R., Cleveland J.N. Performance appraisal: An organizational perspective. 1991;Allyn & Bacon, Boston.
48. Nunnally J.C. Psychometric theory. 2nd ed. 1978;McGraw-Hill, New York.
49. Orpen C. An empirical assessment of the job performance of high-level executives by means of a multitrait-multimethod matrix. Psychologia Africana. 15:1973;7-14 denotes studies providing MTMR matrices.
50. Piotrowski M.J. Unpublished raw data. Ratings of human resource workers. 1993; denotes studies providing MTMR matrices.
51. denotes studies providing MTMR matrices Pollak F. Team assessments. 1994;. unpublished raw data.
52. Rothstein H.R. Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology. 75:1990;322-327.
53. Schmitt N., Coyle B.W., Saari B.B. A review and critique of analyses of multitrait-multimethod matrices. Multivariate Behavioral Research. 12:1977;447-478.
54. Schmitt N., Stults D.M. Methodology review: Analysis of multitrait-multimethod matrices. Applied Psychological Measurement. 10:1986;1-22.
55. denotes studies providing MTMR matrices Sypher B.D., Sypher H.E. Seeing ourselves as others see us: Convergence and divergence in assessments of communication behavior. Communication Research. 11:1984;97-115.
56. Tanaka J.S. "How big is big enough?": Sample size and goodness of fit in structural equation models with latent variables. Child Development. 58:1987;134-146.
57. denotes studies providing MTMR matrices Tucker M.F., Cline V.B., Schmitt J.R. Prediction of creativity and other performance measures from biographical information among pharmaceutical scientists. Journal of Applied Psychology. 51:1967;131-138.
58. Widaman K.F. Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement. 9:1985;1-26.
59. Williams L.J., Cote J.A., Buckley M.R. Lack of method variance in self-reported affect and perceptions at work: Reality or artifact? Journal of Applied Psychology. 74:1989;462-468.
60. Wothke W., Browne M.W. The direct product model for the MTMM matrix parameterized as a second order factor analysis model. Psychometrika. 55:1990;255-262.