Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management

Canadian Journal of Behavioural Science

Volumn 41, Issue 4, 2009, Pages 227-240

  Macdonald, Heather A ^a   Sulsky, Lorne M ^a  

^a WILFRID LAURIER UNIVERSITY   (Canada)

Author keywords

Culture; Performance management; Rater training; Rating formats

Indexed keywords

EID: 70449377485     PISSN: 0008400X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0015165     Document Type: Article

References (118)

1. Adler, N. J. (1983). Cross-cultural management research: The ostrich and the trend. Academy of Management Review, 8, 226-232.
2. Alliger, G. M. (2006). Leveraging distributed systems to enable job redesign: Performance criteria in the electronic age. In W. Bennett Jr., C. E. Lance, and D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 273-296). Mahwah, NJ: Erlbaum.
3. Arvey, R. D., & Murphy, K. R. (1998). Performance evaluation in work settings. Annual Review of Psychology, 49, 141-168.
4. Atkins, P. W., Wood, R. E., & Rutgers, P. J. (2002). The effects of feedback format on dynamic decision making. Organizational Behavior and Human Decision Processes, 88, 587-604.
5. Balzer, W. K., & Sulsky, L. M. (1990). Performance appraisal effectiveness. In K. R. Murphy & F. E. Saal (Eds.), Psychology in organizations: Integrating science and practice (pp. 133-156). Hillsdale, NJ: Erlbaum.
6. Balzer, W. K., & Sulsky, L. M. (1992). Halo and performance appraisal research: A critical examination. Journal of Applied Psychology, 77, 975-985.
7. Banks, C. G., & Murphy, K. R. (1985). Toward or narrowing the researchpractice gap in performance appraisal. Personnel Psychology, 38, 335-345.
8. Bemardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Reviewv, 6, 205-212.
9. Bernardin, H. J., Buckley, M. R., Tyler, C. L., & Wiese, D. S. (2000). A reconsideration of strategies in rater training. Research in Personnel and Human Resources Management, 18, 221-274.
10. Bernardin, H. J., & Pence, E. C. (1980). Effects of rater error training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60-66.
11. Bernardin, H. J., & Smith, P. C. (1981). A clarification of some issues regarding the development and use of behaviorally anchored ratings scales. Journal of Applied Psychology, 66, 458-463.
12. Biernat, M., Crandall, C. S., Young, L. V., Kobrynowicz, D., & Halpin, S. M. (1998). All that you can be: Stereotyping of self and others in a military context. Journal of Personality and Social Psychology, 75, 301-317.
13. Bono, J. E., & Colbert, A. E. (2005). Understanding responses to multisource feedback: The role of core self-evaluations. Personnel Psychology, 58, 171-203.
14. Borman, W. C, Buck, D. E., Hanson, M. A., Motowidlo, S. J., Stark, S., and Drasgow, F. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965-973.
15. Bretz, R. D., Jr., Milkovich, G. T., & Read, W. (1992). The current state of performance appraisal research and practice: Concerns, directions, and implication. Journal of Management, 18, 321-352.
16. Brown, T. C, & Latham, G. P. (2002). The effects of behavioural outcome goals, learning goals, and urging people to do their best on an individual's teamwork behaviour in a group problem solving task. Canadian Journal of Behavioural Science, 34, 276-285.
17. Caligiuri, P. M. (2006). Measurement performance in a cross-cultural context. In W. Bennett Jr., C. E. Lance, & D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 227-244). Mahwah, NJ: Erlbaum.
18. Caligiuri, P. M., & Day, D. V. (2000). Effects of self-monitoring on technical, contextual, and assignment-specific performance: A study of cross-national work performance ratings. Group and Organization Management, 25, 154-175.
19. Cawley, B. D., Keeping, L. M., & Levy, L. M. (1998). Participation in the performance appraisal process and employee reactions: A meta analytic review of field investigations. Journal of Applied Psychology, 83, 615-633.
20. Chi-Ching, Y., & Noi, L. S. (2001). Learning styles and their implications for cross cultural management in Singapore. Journal of Social Psychology, 134, 593-600.
21. Chirico, C. E., Buckley, M. R., Wheeler, A. R., Facteau, J. D., Bernardin, H. J., & Beu, D. S. (2004). A note on the need for true scores in frame-of-reference (FOR) training research. Journal of Managerial Issues, 16, 382-395.
22. Church, A. H., Rogelberg, S. G., & Waclawski, J. (2000). Since when is no news good news? The relationship between performance and response rates in multirater feedback. Personnel Psychology, 53, 435-451.
23. Cleveland, J. N., Murphy, K. R., & Williams, R. E. (1989). Multiple uses of performance appraisal: Prevalence and correlates. Journal of Applied Psychology, 74, 130-135.
24. Cox, T. (1991). The multicultural organization. Academy of Management Executive, 5, 34-47.
25. Deming, W. E. (1986). Out of the crisis. Cambridge, MA: MIT Institute for Advanced Engineering Study.
26. DeNisi, A. S., Cafferty, T., & Meglino, B. (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behavior and Human Performance, 33, 360-396.
27. DeNisi, A. S., & Kluger, A. N. (2000). Feedback effectiveness: Can 360-degree appraisals be improved? Academy of Management Executive, 14, 129-139.
28. DeNisi, A. S., & Pritchard, R. D. (2006). Performance appraisal, performance management, and improving individual performance: A motivation framework. Management and Organization Review, 2, 253-277.
29. Dewberry, C. (2001). Performance disparities between whites and ethnic minorities: Real differences or assessment bias? Journal of Occupational and Organizational Psychology, 74, 659-673. (Pubitemid 33396020)
30. Eagly, A. H., & Karau, S. J. (2002). Role congruity theory of prejudice toward female leaders. Psychological Review, 109, 573-598.
31. Eagly, A. H., Karau, S. J., & Makhijani, M. G. (1995). Gender and the effectiveness of leaders: A meta-analysis. Psychological Bulletin, 117, 125-145.
32. Eagly, A. H., Makhijani, M. G., & Klonsky, B. G. (1992). Gender and the evaluation of leaders: A meta-analysis. Psychological Bulletin, 111, 3-22.
33. Farr, J. L., & Jacobs, R. (2006). Trust us: New perspectives on performance appraisal. In W. Bennett, D. Woehr, & C. Lance (Eds)., Performance measurement: Current perspectives and future challenges (pp. 321-337). Mahwah, NJ: Lawrence Erlbaum Associates.
34. Fay, C. H., & Latham, G. P. (1982). Effects of training and rating scale on rating errors. Personnel Psychology, 35, 105-116.
35. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66, 127-148.
36. Ferris, G. R., & Treadway, D. C. (2008). Cultural diversity and performance appraisal systems. In D. L. Stone & E. F. Stone-Romero (Eds.), The influence of culture on human resource management processes and practices (pp. 135-156). New York: Erlbaum.
37. Fletcher, C. (2001). Performance appraisal and management: The developing research agenda. Journal of Occupational and Organizational Psychology, 73, 473-487. (Pubitemid 33387658)
38. Folger, R., Konovsky, M. A., & Cropanzano, R. (1992). A due process metaphor for performance appraisal. In B. M. Staw & L. L. Cummings (Eds.), Research in organizational behavior (pp. 127-148). Greenwich, CT: JAI.
39. Goodenough, D. R. (1976). The role of individual differences in field dependence as a factor in learning and memory. Psychological Bulletin, 83, 675-694.
40. Guion, R. M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Erlbaum.
41. Hartel, C. E. J. (1993). Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics. Journal of Applied Psychology, 78, 212-217.
42. Hedge, J. W., & Kavanaugh, M. J. (1988). Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training. Journal of Applied Psychology, 73, 68-73.
43. Heimbeck, D., Frese, M., & Sonnentag, S. (2003). Integrating errors into the training process: The function of error management instructions and the role of goal orientation. Personnel Psychology, 56, 333-362.
44. Heine, S. J. (2007). Culture and motivation: What motivates people to act in the ways that they do? In S. Kitayama & D. Cohen (Eds.), Handbook of cultural psychology (pp. 714-733). New York: Guilford.
45. Heine, S. J., Kitayama, S., & Lehman, D. R. (2001). Cultural differences in self-evaluation: Japanese readily accept negative self-relevant information. Journal of Cross-Cultural Psychology, 32, 434-443.
46. Heine, S. J., Lehman, D. R., Markus, H. R., & Kitayama, S. (1999). Is there a universal need for positive self-regard? Psychological Review, 106, 766-794.
47. Hofstede, G. (1991). Cultural and organizations: Software of the mind. London: McGraw-Hill.
48. Ilgen, D. R., Barnes-Farrell, J. J., & McKellin, D. B. (1993). Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321-368.
49. Jelley, R. B., & Goffin, R. D. (2001). Can performance-feedback accuracy be improved? Effects of rater priming and rating-scale format on rating accuracy. Journal of Applied Psychology, 86, 134-144.
50. Ji, L., Peng, K., & Nisbett, R. (2000). Culture, control, and perception of relations in environment. Journal of Personality and Social Psychology, 78, 943-955.
51. Ji, L.-J., Zhang, Z., & Nisbett, R. (2004). Is it culture or is it language? Examination of language effects in cross-cultural research on categorization. Journal of Personality and Social Psychology, 87, 57-65.
52. Johnson, T., Kulesa, P., Cho, Y. I., & Shavitt, S. (2005). The relation between culture and response styles: Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264-277.
53. Keeping, L. M., & Levy, P. E. (2000). Performance appraisal reactions: Measurement, modeling, and method bias. Journal of Applied Psychology, 85, 708-723.
54. Keown, J. L., & Sulsky, L. M. (2001). The effects of task information training and frame-of-reference training with situational constraints on rating accuracy. Human Performance, 4, 305-320.
55. Keown, J. L., Sulsky, L. M., & Jones, D. A. (2005, August). Attributional processes in performance appraisal: The salience of situational constraints in supervisory and self ratings. Paper presented at the annual meeting of the Academy of Management, Honolulu, HI.
56. Kinicki, A. J., Prussia, G. E., Wu, W. B., & McKee-Ryan, F. M. (2004). A covariance structure analysis of employees' responses to performance feedback. Journal of Applied Psychology, 89, 1057-1069.
57. Kitayama, S., Duffy, S., Kawamura, T., & Larsen, J. T. (2003). Perceiving an object and its context in different cultures: A cultural look at new look. Psychological Science, 14, 201-206.
58. Klein, H. J., Wesson, M. J., Hollenbeck, J. R., & Alge, B. J. (1999). Goal commitment and the goal-setting process: Conceptual clarification and empirical synthesis. Journal of Applied Psychology, 84, 885-896.
59. Kühnen, U., Hannover, B., & Schubert, B. (2001). The semanticprocedural-interface model of the self: The role of self-knowledge for context-dependent versus context independent modes of thinking. Journal of Personality and Social Psychology, 80, 397-409.
60. Kurman, J. (2001). Self-regulation strategies in achievement settings: Culture and gender differences. Journal of Cross-Cultural Psychology, 32, 491-503.
61. Lalwani, A. K., Shavitt, S., & Johnson, T. (2006). What is the relation between cultural orientation and socially desirable responding? Journal of Personality and Social Psychology, 90, 165-178.
62. Lam, S. S., Yik, M. S., & Schaubroeck, J. (2002). Responses to formal performance appraisal feedback: The role of negative affectivity. Journal of Applied Psychology, 87, 192-201.
63. Lance, C. E. (1994). Test of a latent structure of performance ratings derived from Wherry's (1952). theory of ratings. Journal of Management, 20, 757-771.
64. Landau, J. (1995). The relationship of race and gender to managers' ratings of promotion potential. Journal of Organizational Behavior, 16, 391-400.
65. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.
66. Latham, G. P. (1986). Job performance and appraisal. In C. Cooper & I. Robertson (Eds.), Review of industrial and organizational psychology. Chichester, England: Wiley.
67. Latham, G. P. (2007). Work motivation: History, theory, research and practice. Thousand Oaks, CA: Sage.
68. Latham, G. P., Fay, C. H., & Saari, L. M. (1979). The development of behavioral observation scales for appraising the performance of foremen. Personnel Psychology, 32, 299-311.
69. Latham, G. P., Locke, E. A., & Fassina, N. E. (2002). The high performance cycle: Standing the test of time. In S. Sonnentag (Ed.), The psychological management of individual performance: A handbook in the psychology of management in organizations (pp. 201-228). Chichester, England: Wiley.
70. Latham, G. P., & Mann, S. (2006). Advances in the science of performance appraisal: Implications for practice. International Journal of Organizational and Industrial Psychology, 21, 295-337.
71. Latham, G. P., Mitchell, T. R., & Dossett, D. L. (1978). The importance of participative goal setting and anticipated rewards on goal difficulty and job performance. Journal of Applied Psychology, 63, 163-171.
72. Latham, G. P., & Seijts, G. H. (1997). The effect of appraisal instrument on managerial perceptions of fairness and satisfaction with appraisals from their peers. Canadian Journal of Behavioural Science, 29, 275-282.
73. Latham, G. P., Skarlicki, D., Irvine, D., & Siegel, J. P. (1993). The increasing importance of performance appraisals to employee effectiveness in organizational settings in North America. International Review of Industrial and Organizational Psychology, 8, 87-131.
74. Latham, G. P., & Wexley, K. N. (1977). Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 30, 61255-268.
75. Latham, G. P., & Wexley, K. N. (1994). Increasing productivity through performance appraisal. Reading, MA: Addison Wesley.
76. Latham, G. P., Wexley, K. N., & Pursell, E. D. (1975). Training managers to minimize rating errors in the observation of behavior. Journal of Applied Psychology, 60, 550-555.
77. Lefkowitz, J. (2000). The role of interpersonal affective regard in supervisory performance ratings: A literature review and proposed causal model. Journal of Occupational and Organizational Psychology, 73, 67-85.
78. Levy, P. E., & Williams, J. R. (2004). The social context of performance appraisal: A review and framework for the future. Journal of Management, 30, 881-905.
79. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57, 705-717.
80. Longenecker, C. O., Sims, H., & Gioia, D. (1987). Behind the mask: The politics of employee appraisal. Academy of Management Executive, 1, 183-191.
81. Longnecker, C. O., & Nykodym, N. (1996). Public sector performance appraisal effectiveness: A case study. Public Personnel Management, 25, 151-165.
82. Maier, N. R. (1958). The appraisal interview: Objectives, methods, and skills. New York: Wiley.
83. Markus, H. R., & Kitayama, S. (1991). Cultural and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224-253.
84. Masuda, T., & Nisbett, R. E. (2001). Attending holistically versus analytically: Comparing the context sensitivity of Japanese and Americans. Journal of Personality and Social Psychology, 81, 922-934.
85. McIntyre, T. (1996). Does the way we teach create behavior disorders in culturally different students? Education and Treatment of Children, 19, 354-370.
86. Miller, J. S., & Cardy, R. L. (2000). Self-monitoring and performance appraisal: Rating outcomes in project teams. Journal of Organizational Behavior, 21, 609-626.
87. Mitchell, T. R., & Wood, R. E. (1980). Supervisor's responses to subordinate poor performance: A test of an attributional model. Organizational Behavior and Human Performance, 25, 123-138.
88. Miyamoto, Y., Nisbett, R. E., & Masuda, T. (2006). Culture and the physical environment: Holistic versus analytic perceptual affordances. Psychological Science, 17, 113-119.
89. Murphy, K. R., & Balzer, W. K. (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619-624.
90. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal. Thousand Oaks, CA: Sage.
91. Nease, A. A., Mudgett, B. O., & Quinones, M. A. (1999). Relationships among feedback sign, self-efficacy, and acceptance of performance feedback. Journal of Applied Psychology, 84, 806-814.
92. Noonan, L., & Sulsky, L. M. (2001). Examination of frame-of-reference and behavioral observation training on alternative training effectiveness criteria in a Canadian military sample. Human Performance, 14, 3-26.
93. Pazy, A., & Oron, I. (2001). Sex proportion and performance evaluation among high-ranking military officers. Journal of Organizational Behavior, 22, 689-702.
94. Peters, L. H., & O'Connor, E. J. (1980). Situational constraints and work outcomes: The influences of a frequently overlooked construct. Academy of Management Review, 5, 391-397.
95. Pritchard, R. D., & Youngcourt, S. S. (2008). Culture, feedback, and motivation. In D. L. Stone & E. F. Stone-Romero (Eds.), The influence of culture on human resource management processes and practices (pp. 157-180). New York: Erlbaum.
96. Pulakos, E. D. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes, 38, 78-91.
97. Pursell, E. D., Dossett, D. L., & Latham, G. P. (1980). Obtaining validated predictors by minimizing rating errors in the criterion. Personnel Psychology, 9, 1-96.
98. Renn, R. W. (2003). Moderation by goal commitment of the feedback-performance relationship: Theoretical explanation and preliminary study. Human Resource Management Review, 13, 561-580.
99. Saks, A. M., & Waldman, D. A. (1998). The relationship between age and job performance evaluations for entry-level professionals. Journal of Organizational Behavior, 19, 409-419.
100. Salas, E., Wilson, K. A., & Lyons, R. (2008). Designing and delivering training for multicultural interactions in organizations. In D. L. Stone and E. F. Stone-Romero (Eds.), The influence of culture on human resource management processes and practices (pp. 115-134). New York: Erlbaum.
101. Sanchez-Burks, J. (2002). Protestant relational ideology and (in) attention to relational cues in work settings. Journal of Personality and Social Psychology, 83, 919-929.
102. Sarkar-Barney, S. (2004). The role of national culture in enhancing training effectiveness. In E. Salas (Ed.), Advances in human performance and cognitive engineering research (pp. 183-213). Amsterdam: Elsevier.
103. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
104. Smith, P. B. (2004). Acquiescent response bias as an aspect of cultural communication style. Journal of Cross-Cultural Psychology, 35, 50-61.
105. Smith, P. C, & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
106. Smither, J. W., London, M., Flautt, R., Vargas, Y., & Kucine, I. (2003). Can working with an executive coach improve multisource feedback ratings over time? A quasi-experimental field study. Personnel Psychology, 56, 23-45.
107. Statistics Canada. (2006). The Chinese community in Canada: A growing community. Retrieved March 15, 2007, from http://www.statcan. ca/english/freepub/89-621-XIE/89-621-XIE2006001.htm
108. Stone, D. L., Stone-Romero, E. F., & Lukaszewski, K. M. (2007). The impact of cultural values on the acceptance and effectiveness of human resource management policies and practices. Human Resource Management Review, 17, 152-165.
109. Strauss, J. P., Barrick, M. R., & Connerley, M. L. (2001). An investigation of personality similarity effects (relational and perceived) on peer and supervisor ratings and the role of familiarity and liking. Journal of Occupational and Organizational Psychology, 74, 637-657.
110. Sue-Chan, C, & Ong, M. (2002). Goal assignment and performance: Assessing the mediating roles of goal commitment and self-efficacy and the moderating role of power distance. Organizational Behavior and Human Decision Processes, 89, 1140-1161.
111. Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497-506.
112. Sulsky, L. M., & Day, D. V. (1992). Frame of reference training and cognitive categorization: An empirical investigation of rater memory issues. Journal of Applied Psychology, 77, 501-510.
113. Sulsky, L. M., & Day, D. V. (1994). An examination of the effects of frame-of-reference training on rating accuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
114. Sulsky, L. M., & Keown, J. L. (1998). Performance appraisal in the changing world of work: Implications for the meaning and measurement of work performance. Canadian Psychology, 39, 52-59.
115. Sulsky, L. M., & Kline, T. (2007). Understanding frame-of-reference training success: A social learning theory perspective. International Journal of Training and Development, 11, 121-131.
116. Tannenbaum, S. I. (2006). Applied measurement: Practical issues and challenges. In W. Bennett Jr., C. E. Lance, & D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 297-320). Mahwah, NJ: Erlbaum.
117. Triandis, H. C. (1994). Theoretical and methodological approaches to the study of collectivism and individualism. In U. Kim, H. C. Triandis, Ç;. Kagitçibasi, S. Choi, & G. Yoon (Eds.), Individualism and collectivism (pp. 41-51). Thousand Oaks, CA: Sage.
118. Tziner, A., Joanis, C, & Murphy, K. R. (2000). A comparison of three models of performance appraisal with regard to goal properties, goal