Interrater correlations do not estimate the reliability of job performance ratings

Personnel Psychology

Volumn 53, Issue 4, 2000, Pages 873-900

  Murphy, Kevin R ^a   Deshon, Richard ^b  

^a Department of Environmental and Radiological Health Sciences   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0345991563     PISSN: 00315826     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2000.tb02421.x     Document Type: Article

References (59)

1. Banks CG, Murphy KR. (1985). Toward narrowing the research practice gap in performance appraisal. PERSONNEL PSYCHOLOGY, 38, 335-345.
2. Barrick MR, Mount MK. (1991). The Big Five personality dimensions and job performance: A meta-analysis. PERSONNEL PSYCHOLOGY, 44, 1-26.
3. Bernardin HJ, Beatty RW. (1984). Performance appraisal: Assessing human behavior at work. Boston: Kent.
4. Bernardin HJ, Buckley MR. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
5. Bobko P, Colella A. (1994). Employee reactions to performance standards: A review of related literatures and an identification of research needs. PERSONNEL PSYCHOLOGY, 47, 1-30.
6. Bjerke DC, Cleveland JN, Morrison RF, Wilson WC. (1987). Officer fitness report evaluation study. Navy Personnel Research and Development Center Report, TR 88-4.
7. Borman WC. (1974). The rating of individuals in organizations: An alternative approach. Organizational Behavior and Human Performance, 12, 105-124.
8. Bretz RD, Milkovich GT, Read W. (1992). The current state of performance research and practice: Concerns, directions, and implications. Journal of Management, 18, 321-352.
9. Cleveland JN, Murphy KR. (1993). Analyzing performance appraisal as goal-directed behavior. In Ferris G, Rowland K (Eds.), Research in personnel and human resources management (Vol. 10, pp. 121-185). Greenwich, CT: JAI Press.
10. Cleveland J, Murphy K, Williams R. (1989). Multiple uses of performance appraisal: Prevalence and correlates. Journal of Applied Psychology, 74, 130-135.
11. Conway JM. (1998). Understanding method variance in multitrait multirater performance appraisal matrices: Examples using general impressions and interpersonal affect as measures method factors. Human Performance, 11, 29-56.
12. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
13. Cronbach LJ, Rajaratnam N, Gleser GC. (1965). Theory of generalizability: A liberalization of reliability theory. The British Journal of Statistical Psychology, 16, 137-163.
14. Dansereau F, Graen G, Haga WJ. (1975). A vertical dyadic linkage approach to leadership within formal organizations: A longitudinal investigation of role-making processes. Organizational Behavior and Human Performance, 12, 46-78.
15. DeShon RP. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
16. Gregarus GJ, Robie C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
17. Harris MH, Schaubroeck J. (1988). A meta-analysis of self-supervisory, self-peer, and peer-supervisory ratings. PERSONNEL PSYCHOLOGY, 41, 43-62.
18. Heneman RL. (1986). The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. PERSONNEL PSYCHOLOGY, 39, 811-826.
19. Heneman RL, Wexley KN. (1983). The effects of delay in rating and amount of information observed on performance rating accuracy. Academy of Management Journal, 26, 677-686.
20. Heneman RL, Wexley KN, Moore ML. (1987). Performance-rating accuracy: A critical review. Journal of Business Research, 15, 431-448.
21. Hough LM, Eaton NK, Dunnette MD, Kamp JD, McCloy RA. (1991). Criterion-related validities of personality constructs and the effects of response distortion on those validities. Journal of Applied Psychology, 75, 581-595.
22. Hunter JE, Hirsh HR. (1987). Applications of meta-analysis. In Cooper CL, Robertson IT (Eds.), International review of industrial and organizational psychology (pp. 321-357). Chichester, U.K.: Wiley.
23. Hunter JE, Hunter RF. (1984). Validity and utility of alternate predictors of job performance. Psychological Bulletin, 96, 72-98.
24. Ilgen DR, Barnes-Farrell JL, McKellin DB. (1993). Performance appraisal process research in the 1980s: What has it contributed to appraisals in use. Organizational Behavior and Human Decision Processes, 54, 321-368.
25. Jawahar IM, Williams CR. (1997). Where all the children are above average: The performance appraisal purpose effect. PERSONNEL PSYCHOLOGY, 50, 905-926.
26. Kingstrom PO, Mainstone LE. (1985). An investigation of rater ratee' acquaintance and rater bias. Academy of Management Journal, 28, 641-653.
27. Kraiger K, Ford JK. (1985). A meta-analysis of race effects in performance appraisal. Journal of Applied Psychology, 70, 56-65.
28. Landy FJ, Farr JL. (1980). Performance rating. Psychological Bulletin, 87, 72-107.
29. Linville PW, Jones EJ. (1980). Polarized appraisals of out-group members. Journal of Personality and Social Psychology, 38, 689-703.
30. Long P. (1986) Performance appraisal revisited. London: Institute of Personnel Management.
31. Longenecker CO, Sims HP, Gioia DA. (1987). Behind the mask: The politics of employee appraisal. Academy of Management Executive, 1, 183-193.
32. Lord FM, Novick MR. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
33. Lumsden J. (1976). Test theory. Annual Review of Psychology, 27, 251-280
34. McEvoy GM, Cascio WF. (1989). Cumulative evidence of the relationship between employee age and job performance. Journal of Applied Psychology, 74, 11-17.
35. McIntyre RM, Smith D, Hassett CE. (1984). Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology, 69, 147-156.
36. Milkovich GT, Wigdor AK. (1991). Pay for performance. Washington, DC: National Academy Press.
37. Mohrman AM, Lawler EE. (1983). Motivation and performance appraisal behavior. In Landy F, Zedeck S, Cleveland J (Eds.), Performance measurement and theory. Hillsdale, NJ: Erlbaum.
38. Murphy KR, Balzer WK, Kellam KL, Armstrong J. (1984). Effect of purpose of rating on accuracy in observing teacher behavior and evaluating teaching performance. Journal of Educational Psychology, 76, 45-54.
39. Murphy KR, Cleveland JN. (1991). Performance appraisal: An organizational perspective. Needham Heights, MA: Allyn and Bacon.
40. Murphy KR, Cleveland JN. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
41. Murphy K, Davidshofer C. (1998). Psychological testing: Principles and applications (4th ed). Englewood Cliffs, NJ: Prentice Hall.
42. Murphy K, Herr B, Lockhart M, Maguire E. (1986). Evaluating the performance of paper people. Journal of Applied Psychology, 71, 654-661.
43. Nunnally JC. (1978). Psychometric theory. New York: McGraw-Hill.
44. Park B, Rothbart M. (1982). Perception of out-group homogeneity and levels of social categorization: Memory for subordinate attributes of in-group members. Journal of Personality and Social Psychology, 42, 1051-1068.
45. Ree MJ, Eartes JA. (1991). Predicting training success: Not much more thang. PERSONNEL PSYCHOLOGY, 44, 321-332.
46. Rothstein HR. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to learn. Journal of Applied Psychology, 75, 322-327.
47. Scullen SE, Mount MK, Goff M. (in press). Understanding the latent structure of job performance ratings. Journal of Applied Psychology.
48. Schmidt FL, Hunter JE. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
49. Shrout PE, Fleiss JL. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
50. Thornton CC III. (1980). Psychometric properties of self-appraisals of job performance. PERSONNEL PSYCHOLOGY, 33, 263-271.
51. Tornow WW. (1993). Perceptions or reality: Is multiple-perspective measurement a means or an end. Human Resource Management, 32, 221-230.
52. Tziner A, Murphy K. (in press). Additional evidence of attitudinal influences in performance appraisal. Journal of Business and Psychology.
53. Tziner A, Murphy KR, Cleveland JN. (1998). Relationships between work attitudes and performance evaluation: A multinational study. Presented at the International Congress of Applied Psychology, San Francisco, CA.
54. Tziner A, Murphy K, Cleveland J, Beaudin G, Marchand S. (1998). Impact of rater beliefs regarding performance appraisal and its organizational contexts on appraisal quality. Journal of Business and Psychology, 12, 457-467.
55. Vecchio RP, Gobdel BC. (1984). The vertical dyadic linkage model of leadership: Problems and prospects. Organizational Behavior and Human Performance, 34, 5-20.
56. Villanova P, Bernardin HJ. (1989). Impression management in the context of performance appraisal. In Giacalone RA, Rosenfeld P (Eds), Impression management in the organization (pp. 299-314). Hillsdale, NJ: Erlbaum.
57. Viswesvaran C, Ones DS, Schmidt FL. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
58. Wherry RJ, Bartlett CJ. (1982). The control of bias in ratings: A theory of rating. PERSONNEL PSYCHOLOGY, 35, 521-555.
59. Williams JR, Levy PE. (1992). The effects of perceived system knowledge on the agreement between self-ratings and supervisor ratings. PERSONNEL PSYCHOLOGY, 45, 835-847.