Construct validity of multi-source performance ratings: An examination of the relationship of self-, supervisor-, and peer-ratings with cognitive and personality measures

International Journal of Selection and Assessment

Volumn 14, Issue 1, 2006, Pages 67-81

  Van Hooft, Edwin A J ^a,b   Van Flier, Henk Der ^a   Minne, Marjolein R ^a  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33845428825     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2006.00334.x     Document Type: Article

References (58)

1. Antonioni, D. (1996) Designing an effective 360-degree appraisal feedback process. Organizational Dynamics, 25(2), 24-38.
2. Arthur, W., Day, E.A., McNelly, T.L. and Edens, P.S. (2003) A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125-154.
3. Atkins, P.W.B. and Wood, R.E. (2002) Self- versus others' ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. Personnel Psychology, 55, 871-904.
4. Atwater, L.E., Waldman, D.A. and Brett, J.F. (2002) Understanding and optimizing multisource feedback. Human Resource Management, 41, 193-208.
5. Atwater, L.E. and Yammarino, F.J. (1992) Does self-other agreement on leadership perceptions moderate the validity of leadership and performance predictions? Personnel Psychology, 45, 141-164.
6. Becker, T.E. and Klimoski, R.J. (1989) A field study of the relationship between the organizational feedback environment and performance. Personnel Psychology, 42, 343-358.
7. Bettenhausen, K.L. and Fedor, D.B. (1997) Peer and upward appraisals: A comparison of their benefits and problems. Group & Organization Management, 22, 236-263.
8. Borman, W.C. (1997) 360° ratings: An analysis of assumptions and a research agenda for evaluating their validity. Human Resource Management Review, 7, 299-315.
9. Bozeman, D.P. (1997) Interrater agreement in multi-source performance appraisal: A commentary. Journal of Organizational Behavior, 18, 313-316.
10. Cascio, W.F. (1991) Applied psychology in personnel management (4th Edn.). Englewood Cliffs, NJ: Prentice-Hall.
11. Church, A.H. and Bracken, D.W. (1997) Advancing the state of the art of 360-degree feedback: Guest editors' comments on the research and practive of multirater assessment methods. Group & Organization Management, 22, 149-191.
12. Conway, J.M. (1996) Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
13. Conway, J.M. and Huffcutt, A.I. (1997) Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
14. De Kok, P. (1996) PC In-Basket onderzoeksverslag [PC In-Basket research report]. Weesp, the Netherlands: Construct.
15. DeNisi, A.S. and Kluger, A.N. (2000) Feedback effectiveness: Can 360-degree appraisals be improved? The Academy of Management Executive, 14, 129-139.
16. Drenth, P.J.D. (1998) Personnel appraisal. In P.J.D. Drenth, H. Thierry and C.J. De Wolff (Eds.), Handbook of work and organizational psychology Vol. 3: Personnel psychology (2nd Edn. pp. 59-87). Hove, UK: Psychology Press.
17. Evers, A., Van Vliet-Mulder, J.C. and Groot, C.J. (2000) Documentatie van tests en testresearch in Nederland [Documentation of tests and test research in The Netherlands]. Assen, the Netherlands: Van Gorcum.
18. Fahr, J.L., Cannella, A.A. and Bedeian, A.G. (1991) Peer ratings: The impact of purpose on rating quality and user acceptance. Group & Organization Studies, 16, 367-385.
19. Fletcher, C. (1998) Feature 360-degree feedback: Circular argument. People Management, 4(19), 46-49.
20. Fletcher, C., Baldry, C. and Cunningham-Snell, N. (1998) The psychometric properties of 360 degree feedback: An empirical study and a cautionary tale. International Journal of Selection and Assessment, 6, 19-34.
21. Greguras, G.J. and Robie, C. (1998) A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
22. Harris, M.M. and Schaubroeck, J. (1988) A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41, 43-62.
23. Houtman, S. (1996) Onderzoeksrapport Quest en Brain [Research report Quest and Brain]. Woerden, the Netherlands: Meurs Personeelsadvies.
24. Houtman, S., Van Leeuwen, L. and Vinke, I. (1999) Aanvulled onderzoek MBS naar aanleiding van de Cotanbeoordeling 1999 [Additional research MBS in respons to the Dutch test committee's evaluation]. Woerden, The Netherlands: Meurs Personeelsadvies.
25. Hu, L. and Bentler, P.M. (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
26. Jansen, P.G.W. (1993) De werking van het assessment center [The operation of the assessment center]. Gedrag & Organisatie, 6, 10-24.
27. Jansen, P.G.W. and Vloeberghs, D. (1999) Multi-rater feedback methods: Personal and organizational implications. Journal of Managerial Psychology, 14, 455-476.
28. Jellema, F. (2000) Toepassing van 360-graden feedback in Nederlandse organisaties [Use of 360-degree feedback in Dutch organizations]. Opleiding & Ontwikkeling, 13(7-8), 21-25.
29. Kane, J.S. and Lawler, E.E. (1979) Performance appraisal effectiveness: Its assessment and determinants. In B.M. Staw (Ed.), Research in organizational behavior. Greenwich, UK: JAI Press.
30. Katz, R.L. (1974) Skills of an effective administrator. Harvard Business Review, 52, 90-102.
31. Kenny, D.A., Albright, L., Malloy, T.E. and Kashy, D.A. (1994) Consensus in interpersonal perception: Acquaintance and the big five. Psychological Bulletin, 116, 245-258.
32. Kolk, N.J., Born, M.Ph. and Akkerman, T.E. (1998) De analyse/presentatie opdracht: Een meta-analyse van de psychometrische kwaliteiten van een veelzijdig assessment center onderdeel [The analysis/presentation exercise: A meta-analysis of the psychometric qualities of a many-sided assessment center assignment]. Gedrag & Organisatie, 11, 148-160.
33. Kolk, N.J., Born, M.Ph. and Van der Flier, H. (2004) A triadic approach to the construct validity of the assessment center: The effect of categorizing dimensions into a Feeling, Thinking, and Power taxonomy. European Journal of Psychological Assessment, 20, 149-156.
34. Lance, C.E., Lambert, T.A., Gewin, A.G., Lievens, F. and Conway, J.M. (2004) Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89, 377-385.
35. Lance, C.E., Teachout, M.S. and Donnelly, T.M. (1992) Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology, 77, 437-452.
36. Lepsinger, R. and Lucia, A.D. (1997) 360° feedback and performance appraisal. Training, 34(9), 62-70.
37. Lievens, F. and Conway, J.M. (2001) Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
38. London, M. and Smither, J.W. (1995) Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. Personnel Psychology, 48, 803-840.
39. Long, P. (1986) Performance appraisal revisited. London: Institute of Personnel Management.
40. Mann, F.C. (1975) Toward an understanding of the leadership role in formal organization. In R. Dubin, G.C. Homans, F.C. Mann and D.C. Miller (Eds.), Leadership and productivity: Some facts of industrial life (pp. 68-103). San Francisco, CA: Chandler.
41. McEvoy, G.M. and Buller, P.F. (1987) User acceptance of peer appraisals in an industrial setting. Personnel Psychology, 40, 785-797.
42. Minne, M.R. (1999). De PC In Basket 'Zeezicht': Een onderzoek naar betrouwbaarheid en validiteit [The In Basket test 'Zeezicht': A study of the reliability and the validity] Unpublished master's thesis, Free University, Amsterdam, the Netherlands.
43. Mount, M.K., Judge, T.A., Scullen, S.E., Sytsma, M.R. and Hezlett, S.A. (1998) Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
44. Murphy, K.R. and Cleveland, J.N. (1991) Performance appraisal: An organizational perspective. Boston: Allyn and Bacon.
45. Nilsen, D. and Campbell, D.P. (1993) Self-observer rating discrepancies: Once an overrater, always an overrater? Human Resource Management, 32, 265-281.
46. Roth, P.L. (1994) Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47, 537-560.
47. Scharley, T. (1994) PC In-Basket for Windows: Investigation at an electrical company. Konstanz, Germany: Scharley & Partner Management Development Consultants.
48. Schmidt, F.L. and Hunter, J.E. (1998) The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
49. Scullen, S.E., Mount, M.K. and Goff, M. (2000) Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
50. Scullen, S.E., Mount, M.K. and Judge, T.A. (2003) Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50-66.
51. Sternberg, R.J. (2000) The concept of intelligence. In R.J. Sternberg (Ed.), Handbook of intelligence. Cambridge, UK: Cambridge University Press.
52. Thornton III, G.C. and Byham, W.C. (1982) Assessment centers and managerial performance. New York: Academic Press.
53. Toegel, G. and Conger, J.A. (2003) 360-degree assessment: Time for reinvention. Academy of Management Learning and Education, 2, 297-311.
54. Vance, R.J., Coovert, M.D., MacCallum, R.C. and Hedge, J.W. (1989) Construct models of taks performance. Journal of Applied Psychology, 74, 447-455.
55. Viswesvaran, C., Ones, D.S. and Schmidt, F.L. (1996) Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
56. Waldman, D.A., Atwater, L.E. and Antonioni, D. (1998) Has 360 degree feedback gone amok? The Academy of Management Executive, 12, 86-94.
57. Wanous, J.P., Reichers, A.E. and Hudy, M.J. (1997) Overall job satisfaction: How good are single-items measures? Journal of Applied Psychology, 82, 247-252.
58. Zedeck, S. and Cascio, W.F. (1982) Performance appraisal decisions as a function of rater training and purpose of the appraisal. Journal of Applied Psychology, 67, 752-758.