Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity

Journal of Applied Psychology

Volumn 86, Issue 2, 2001, Pages 255-264

  Lievens, Filip ^a  

^a GHENT UNIVERSITY   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; CLINICAL TRIAL; COMPARATIVE STUDY; CONTROLLED CLINICAL TRIAL; CONTROLLED STUDY; DISCRIMINANT ANALYSIS; FEMALE; HUMAN; IN SERVICE TRAINING; MALE; MIDDLE AGED; OBSERVER VARIATION; PERSONNEL MANAGEMENT; PSYCHOLOGICAL ASPECT; RANDOMIZED CONTROLLED TRIAL; STATISTICS; STUDENT;

ADULT; DISCRIMINANT ANALYSIS; EMPLOYEE PERFORMANCE APPRAISAL; FEMALE; HUMANS; INSERVICE TRAINING; MALE; MIDDLE AGED; OBSERVER VARIATION; PERSONNEL MANAGEMENT; STUDENTS;

EID: 85047683691     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.86.2.255     Document Type: Article

References (52)

1. Abelson, R. P. (1981). Psychological status of the script concept. American Psychologist, 36, 715-729.
2. Arvey, R. D., & Murphy, K. R. (1998). Performance evaluation in work settings. Annual Review of Psychology, 49, 141-168.
3. Ballantyne, I., & Povah, N. (1995). Assessment and development centres. Aldershot, England: Gower.
4. Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
5. Borman, W. C. (1977). Consistency of rating accuracy and rater errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238-252.
6. Borman, W. C. (1978). Exploring the upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63, 135-144.
7. Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA: American College Testing Program.
8. Byham, W. C. (1977). Assessor selection and training. In J. L. Moses & W. C. Byham (Eds.), Applying the assessment center method (pp. 89-125). New York: Pergamon Press.
9. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
10. Cantor, N., & Mischel, W. (1977). Traits as prototypes: Effects on recognition memory. Journal of Personality and Social Psychology, 35, 38-48.
11. Cardy, R. L., Bernardin, H. J., Abbott, J. G., Senderak, M. P., & Taylor, K. (1987). The effects of individual performance schemata and dimension familiarization on rating accuracy. Journal of Occupational and Organisational Psychology, 60, 197-205.
12. Cardy, R. L., & Keefe, T. J. (1994). Observational purpose and evaluative articulation in frame-of-reference training: The effects of alternative processing modes on rating accuracy. Organizational Behavior and Human Decision Processes, 57, 338-357.
13. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
14. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
15. Cronbach, L. J. (1955). Processes affecting scores on "understanding of others" and "assumed similarity." Psychological Bulletin, 32, 177-193.
16. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
17. Day, D. V., & Sulsky, L. M. (1995). Effects of frame-of-reference training and information configuration on memory organization and rating accuracy. Journal of Applied Psychology, 80, 158-167.
18. Fiske, S. T., & Dyer, L. M. (1985). Structure and development of social schemata: Evidence from positive and negative transfer effects. Journal of Personality and Social Psychology, 48, 839-852.
19. Fiske, S. T., & Taylor, S. E. (1991). Social cognition. Reading, MA: Addison-Wesley.
20. Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493-511.
21. Gaugler, B. B., & Thornton, G. C. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74, 611-618.
22. Heneman, R. L. (1988). Traits, behaviors, and rater training: Some unexpected results. Human Performance, 1, 85-98.
23. Ilgen, D. R., Barnes-Farrell, J. L., & McKellin, D. B. (1993). Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organisational Behavior and Human Decision Processes, 54, 321-368.
24. Jones, A. (1981 ). Inter-rater reliability in the assessment of group exercises at the UK assessment centre. Journal of Occupational and Organisational Psychology, 54, 79-86.
25. Kane, M. T. (1982). A sampling model for validity. Applied Psychological Measurement, 6, 125-160.
26. Klimoski, R. J., & Brickner, M. (1987). Why do assessment centers work? The puzzle of assessment center validity. Personnel Psychology, 40, 243-260.
27. Kozlowski, S. W. J., Kirsch, M. P., & Chao, G. T. (1986). Job knowledge, ratee familiarity, conceptual similarity and halo error: An exploration. Journal of Applied Psychology, 71, 45-49.
28. Kozlowski, S. W. J., & Mongillo, M. (1992). The nature of conceptual similarity schemata: Examination of some basic assumptions. Personality and Social Psychology Bulletin, 18, 88-95.
29. Kraiger, K., & Teachout, M. S. (1990). Generalizability theory as construct-related evidence of the validity of job performance ratings. Human Performance, 3, 19-35.
30. Lichtenstein, M., & Srull, T. K. (1987). Processing objectives as a determinant of the relationship between recall and judgement. Journal of Experimental Social Psychology, 23, 93-118.
31. Lievens, F. (1998a, August). Assessment centers and what they measure: Disentangling assesee, assessor, exercise, and dimension effects. Paper presented at the Annual Meeting of the Academy of Management, San Diego, CA.
32. Lievens, F. (1998b). Factors which improve the construct validity of assessment centers: A review. International Journal of Selection and Assessment, 6, 141-152.
33. Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47, 1231-1243.
34. Lord, R. G. (1985). Accuracy in behavioral measurement: An alternative definition based on raters' cognitive schema and signal detection theory. Journal of Applied Psychology, 70, 66-71.
35. Marcoulides, G. A. (1989). The application of generalizability analysis to observational studies. Quality and Quantity, 23, 115-127.
36. Pulakos, E. D. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes, 38, 76-91.
37. Rumelhart, D. E., & Ortony, A. (1977). The representation of knowledge in memory. In R. C. Anderson, R. J. Spiro, & W. E. Montague (Eds.), Schooling and the acquisition of knowledge (pp. 99-136). Hillsdale, NJ: Erlbaum.
38. Sagie, A., & Magnezy, R. (1997). Assessor type, number of distinguishable dimension categories, and assessment centre construct validity. Journal of Occupational and Organizational Psychology, 70, 103-108.
39. Schleicher, D. J., & Day, D. V. (1998). A cognitive evaluation of frame-of-reference rater training: Content and process issues. Organizational Behavior and Human Decision Processes, 73, 76-101.
40. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
41. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
42. Srull, T. K., & Wyer, R. S. (1989). Person memory and judgment. Psychological Review, 96, 58-83.
43. Stamoulis, D. T., & Hauenstein, N. M. A. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of Applied Psychology, 78, 994-1003.
44. Sulsky, L. M., & Balzer, W. K. (1988). The meaning and measurement of performance rating accuracy: Some methodological concerns. Journal of Applied Psychology, 73, 497-506.
45. Sulsky. L. M., & Day, D. V. (1992). Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues. Journal of Applied Psychology, 77, 501-510.
46. Sulsky, L. M., & Day, D. V. (1994). Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
47. Task Force on Assessment Center Guidelines. (1989). Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 18, 457-470.
48. Taylor, S. E., & Crocker, J. (1981). Schematic bases of social information processing. In E. T. Higgins, S. P. Herman, & M. P. Zanna (Eds.), Social cognition: The Ontario symposium (pp. 89-134). Hillsdale, NJ: Erlbaum.
49. Thornton, G. C., III. (1992). Assessment centers in Human Resource Management. Reading. MA: Addison-Wesley.
50. Woehr, D. J. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79, 525-534.
51. Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organisational Psychology, 67, 189-205.
52. Zedeck, S. (1986). A process analysis of the assessment center method. Research in Organizational Behavior, 8, 259-296.