Placing peer ratings in context: Systematic influences beyond ratee performance

Personnel Psychology

Volumn 60, Issue 1, 2007, Pages 93-126

  Dierdorff, Erich C ^a   Surface, Eric A ^b  

^a DEPAUL UNIVERSITY   (United States)

^b Surface Ward and Associates ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33847166221     PISSN: 00315826     EISSN: 17446570     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2007.00066.x     Document Type: Article

References (92)

1. Arvey RD, Murphy KR. (1998). Performance evaluation in work setting. Annual Review of Psychology, 49, 141-168.
2. Barnes-Farrell JL. (2001). Performance appraisal: Person perception processes. In London M (Ed.), How people evaluate others in organizations (pp. 67-90). Mahwah, NJ: Erlbaum.
3. Beaty JC, Cleveland JN, Murphy KR. (2001). The relation between personality and contextual performance in "strong" versus "weak" situations. Human Performance, 14, 125-148.
4. Bentler PM. (1990). Comparative fix indexes in structural models. Psychological Bulletin, 107, 238-246.
5. Bentler PM, Bonett DG. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
6. Block J, Block JH. (1981). Studying situational dimensions: A grand perspective and some limited empiricism. In Magnusson D (Ed.), Toward a psychology of situations: An interactional perspective. Hillsdale, NJ: Erlbaum.
7. Bollen KA, Paxton P. (1998). Detection and determinants of bias in subjective measures. American Sociological Review, 63, 465-478.
8. Borman WC. (1974). The rating of individuals in organizations: An alternative approach. Organizational Behavior and Human Performance, 12, 105-124.
9. Borman WC. (1979). Format and training effects on rating accuracy and rater errors. Journal of Applied Psychology, 64, 410-421.
10. Bowers KS. (1973). Situationalism in psychology: An analysis and critique. Psychological Review, 80, 307-336.
11. Browne MW, Cudeck R. (1993). Alternate ways of assessing model fit. In Bollen KA, Long JS (Eds.), Testing structural equation models (pp. 136-162). Thousand Oaks, CA: Sage.
12. Byrne BM. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Erlbaum.
13. Campbell JP. (1999). The definition and measurement of performance in the new age. In Ilgen DR, Pulakos ED (Eds.), The changing nature of performance: Implications for staffing, motivation, and development (pp. 399-430). San Francisco: Jossey-Bass.
14. Campbell JP, Gasser MB, Oswald FL. (1996). The substantive nature of job performance variability. In Murphy KR (Ed.), Individual differences and behavior in organizations (pp. 258-299). San Francisco: Jossey-Bass.
15. Campbell JP, McCloy RA, Oppler SH, Sager CE. (1993). A theory of performance. In-Schmitt N, Borman WC (Eds.), Personnel selection in organizations (pp. 35-70). San Francisco: Jossey-Bass.
16. Cappelli P, Sherer PD. (1991). The missing role of context in OB: The need for a mesc-level approach. Research in Organizational Behavior, 13, 55-110.
17. Carson KP, Cardy RL, Dobbins GH. (1991). Performance appraisal as effective management or deadly management disease. Group and Organization Studies, 16, 143-159.
18. Cacsio WF. (1998). Applied psychology in human resource management (5th ed.). Upper Saddle River, NJ: Prentice-Hall.
19. Cleveland JN, Hollmann G. (1991). Context and discrimination in personnel decisions: Direct and mediated approaches. In Meindl JR, Cardy RL, Puffer S (Eds.), Advances in information processing in organizations (vol. 4, pp. 233-238). Greenwich, CT: JAI Press.
20. Cleveland JN, Murphy KR, Williams RE. (1989). Multiple uses of performance appraisal: Prevalences and correlates. Journal of Applied Psychology, 74, 130-135.
21. Conway JM. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
22. Conway JM, Lombardo K, Sanders KC. (2001). A meta-analysis of incremental validity and nomological networks for subordinate and peer ratings. Human Performance, 14, 267-303.
23. Cooper WH. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-224.
24. Coovert MD, Craiger JP, Teachout MS. (1997). Effectiveness of the direct product model versus confirmatory factor model for reflecting the structure of multitrait-multirater job performance data. Journal of Applied Psychology, 82, 271-280.
25. DeNisi AS, Cafferty TP, Meglino BM. (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behavior and Human Performance, 33, 360-396.
26. Drexler JA Jr, Beehr TA, Stetz TA. (2001). Peer appraisals: Differentiation of individual performance on group task. Human Resource Management, 40, 333-345.
27. Dumenci L. (2000). Multitrait-multimethod analysis. InTinsley H, Brown S (Eds.) Handbook of applied multivariate statistics and mathematical modeling. London: Academic Press.
28. Fiske ST. (1995). Social cognition. In Tesser A (Ed.), Advanced social psychology (pp. 149-194), New York: McGraw-Hill.
29. Fornell C, Larcker DF. (1980). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 28, 39-50.
30. Goffin RD, Jackson DN. (1992). Analysis of multitrait-multirater performance appraisal data: Composite direct product method versus confirmatory factor analysis. Multivariate Behavioral Research, 27, 363-385.
31. Greguras GJ, Robie C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
32. Greguras GJ, Robie C, Born MPh. (2001). Applying the social relations model to self and peer evaluations. The Journal of Management Development, 20, 508-525.
33. Grey RJ, Kipnis D. (1976). Untangling the performance appraisal dilemma: The influence of perceived organizational context on evaluative processes. Journal of Applied Psychology, 61, 329-335.
34. Guion RM. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Erlbaum.
35. Hattrup K, Jackson SE. (1996). Learning about individual differences by taking situations seriously. In Murphy KR (Ed.), Individual differences and behavior in organizations (pp. 507-547). San Francisco: Jossey-Bass.
36. Hofmann DA, Jacobs R, Barrata JE. (1993). Dynamic criteria and the measurement of change. Journal of Applied Psychology, 78, 194-204.
37. Hofmann DA, Jacobs R, Gerras SJ. (1992). Mapping individual performance across time. Journal of Applied Psychology, 77, 185-195.
38. Hough LM, Oswald F. (2000). Personnel selection: Looking toward the future-Remembering the past. Annual Review of Psychology, 51, 631-664.
39. Hu L, Bentler PM. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453.
40. Hu L, Bentler PM. (1999). Cutoff criteria for fit indexes in covariance structure modeling: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
41. Ilgen DR, Barnes-Farrell JL, McKellin D. (1993). Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321-368.
42. Ilgen DR, Feldman JM. (1983). Performance appraisal: A process focus. In Staw BM, Cummings LL (Eds.), Research in organizational behavior (pp. 141-197). Greenwich, CT: JAI Press.
43. Imada AS. (1982). Social interaction, observation, and stereotypes as determinants of differentiation in peer ratings. Organizational Behavior and Human Performance, 29, 397-415.
44. Jackson DL. (2001). Sample size and number of parameter estimates in maximum likelihood confirmatory factor analysis: A Monte Carlo investigation. Structural Equation Modeling, 8, 205-223.
45. Johns G. (1991). Substantive and methodological constraints on behavior and attitudes in organizational research. Organizational Behavior and Human Decision Processes, 49, 80-104.
46. Johns G. (2006). The essential impact of context on organizational behavior. Academy of Management Review, 31, 386-408.
47. Kane JS, Lawler EE. (1978). Methods of peer assessment. Psychological Bulletin, 88, 80-81.
48. Katz D, Kahn RL. (1978). The social psychology of organizations (2nd ed.). New York: Wiley.
49. Kenny DA, Berman JS. (1980). Statistical approaches to the correction of correlational bias. Psychological Bulletin, 88, 288-295.
50. Kenny DA, Kashy DA. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
51. Klimoski RJ, Donahue LM. (2001). Person perception in organizations: An overview of the field. In London M (Ed.), How people evaluate others in organizations (pp. 5-44). Mahwah, NJ: Erlbaum.
52. Lance CE, Lambert TA, Gerwin AG, Lievens F, Conway JM. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise ratings. Journal of Applied Psychology, 89, 377-385.
53. Lance CE, Noble CL, Scullen SE. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7, 228-244.
54. London M, Smither JW. (1995). Can multisource feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. PERSONNEL PSYCHOLOGY, 48, 803-839.
55. London M, Tornow WW. (1998). Introduction: 360-degree feedback-More than a tool. In Tornow WW, London M (Eds.), Maximizing the value of 360-degree feedback. San Francisco: Jossey-Bass.
56. MacCallum RC, Austin JT. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51, 201-226.
57. MacCallum RC, Kim C, Malarkey WB, Kiecolt-Glaser JK. (1997). Studying multivariate change using multilevel models and latent curve models. Multivariate Behavioral Research, 32, 215-253.
58. Marsh HW, Balla JR, McDonald RP. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410.
59. Martin SL, Klimoski RJ. (1990). Use of verbal protocols to trace cognitions associated with self- and supervisor evaluations of performance. Organizational Behavior and Human Decision Processes, 46, 135-154.
60. Maurer TJ, Raju NS, Collins WC. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693-702.
61. Mischel W. (1968). Personality and assessment. New York: Wiley.
62. Mischel W. (1973). Toward a cognitive social learning reconceptualization of personality. Psychological Review, 80, 252-283.
63. Mischel W. (1977). The interaction of person and situation. In Magnusson D, Endler N (Eds.), Personality at the crossroads: Current issues in interactional psychology. Hillsdale, NJ: Erlbaum.
64. Mount MK, Judge TA, Scullen SE, Sytsma MR, Hezlett SA. (1998). Trait, rater, and level effects in 360-degree performance ratings. PERSONNEL PSYCHOLOGY, 51, 557-576.
65. Mount MK, Scullen SE. (2001). Multi-source feedback ratings: What do they really measure? In London M (Ed.), How people evaluate others in organizations (pp. 155-180). Mahwah, NJ: Erlbaum.
66. Mulaik SA, Quartetti DA. (1997). First order or higher order general factor? Structural Equation Modeling, 4, 193-211.
67. Murphy KR, Cleveland JN. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
68. Noe R, Gerhart B, Hollenbeck J, Wright P. (2003). Human resource management: Gaining a competitive advantage (4th ed.). New York: McGraw-Hill/Irwin.
69. Nunnally JC. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
70. Peters LH, O'Connor EJ. (1980). Situational constraints and work outcomes: The influence of a frequently overlooked construct. Academy of Management Review, 5, 391-397.
71. Ployhart RE, Hakel MD. (1998). The substantive nature of performance variability: Predicting interindividual differences in intraindividual performance. PERSONNEL PSYCHOLOGY, 51, 859-901.
72. Pulakos ED, Schmitt N, Chan D. (1996). Models of job performance ratings: An examination of ratee race, ratee gender, and rater level effects. Human Performance, 9, 103-120.
73. Reilly RR, McGourty J. (1998). Performance appraisal in team settings. In Smither JW (Ed.), Performance appraisal: State of the art in practice (pp. 244-277). San Francisco: Jossey-Bass.
74. Saavedra R, Kwun SK. (1993). Peer evaluation in self-managing work groups. Journal of Applied Psychology, 78, 450-462.
75. Sackett PR, Tuzinski K. (2001). The role of dimensions and exercises in assessment center ratings. In London M (Ed.), How people evaluate others in organizations (pp. 111-129). Mahwah, NJ: Erlbaum.
76. Scullen SE. (1999). Using confirmatory factor analysis of correlated uniqueness to estimate method variance in multitrait.multimethod matrices. Organizational Research Methods, 2, 275-292.
77. Scullen SE, Mount MK, Goff M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
78. Stamoulis DT, Hauenstein NM. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of Applied Psychology, 78, 994-1003.
79. Steiger JH. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180.
80. st century: The development of O*NET (pp. 127-146). Washington, DC: APA.
81. Taylor SE, Fiske ST. (1978). Salience, attention, and attribution: Top of the head phenomena. In Berkowitz L (Ed.), Advances in experimental social psychology (vol. 11, pp. 249-288). New York: Academic Press.
82. Terborg JR. (1981). Interactional psychology and research on human behavior in organizations. Academy of Management Review, 6, 569-576.
83. Tett RP, Burnett DD. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88, 500-517.
84. Tomas JM, Hontangas PM, Amparo O. (2000). Linear confirmatory factor models to evaluate multitrait-multimethod matrices: The effects of number of indicators and correlation among methods. Multivariate Behavioral Research, 35, 469-499.
85. Tucker LR, Lewis C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10.
86. Viswesvaran C, Ones DS, Schmidt FL. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-560.
87. Viswesvaran C, Schmidt FL, Ones DS. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct level convergence and rating difficulty. Journal of Applied Psychology, 87, 345-354.
88. Wherry RJ Sr, Bartlett CJ. (1982). The control of bias in ratings: A theory of rating. PERSONNEL PSYCHOLOGY, 35, 521-551.
89. Willett JB, Sayer AG. (1994). Using covariance structure analysis to detect correlates and predictors of individual change over time. Psychological Bulletin, 116, 363-381.
90. Woehr DJ, Huffcutt AI. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.
91. Woehr DJ, Sheehan MK, BennettWJr. (2005). Assessing measurement equivalence across rating sources: A multitrait-multitrater approach. Journal of Applied Psychology, 90, 592-600.
92. Wohlers AJ, London M. (1989). Ratings of managerial characteristics: Evaluation difficulty, co-worker agreement, and self-awareness. PERSONNEL PSYCHOLOGY, 42, 235-261.