Evidence for the effectiveness of an alternative multisource performance rating methodology

Personnel Psychology

Volumn 65, Issue 3, 2012, Pages 531-563

  Hoffman, Brian J ^a   Gorman, C Allen ^b   Blair, Carrie A ^c   Meriac, John P ^d   Overstreet, Benjamin ^a   Atchley, E Kate ^e  

^a UNIVERSITY OF GEORGIA   (United States)

^b RADFORD UNIVERSITY   (United States)

^c School of Business   (United States)

^d UNIVERSITY OF MISSOURI ST LOUIS   (United States)

^e UNIVERSITY OF TENNESSEE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84864780560     PISSN: 00315826     EISSN: 17446570     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2012.01252.x     Document Type: Article

References (85)

1. Atwater LE, Waldman DA, Brett JF. (2002). Understanding and optimizing multisource feedback. Human Resource Management, 41, 193-208.
2. Austin JT, Crespin TR. (2006). Problems of criteria in industrial and organizational psychology: Progress, pitfalls, and prospects. In Bennett W, Jr., Lance CE, Woehr DJ (Eds.), Performance measurement: Current perspectives and future challenges (pp. 9-48). Mahwah, NJ : Erlbaum.
3. Austin JT, Villanova P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77, 836-874.
4. Bartram D. (2007). Increasing validity with forced-choice criterion measure formats. International Journal of Selection and Assessment, 15, 263-272.
5. Becker BE, Cardy RL. (1986). Influence of halo error on appraisal effectiveness: A conceptual and empirical reconsideration. Journal of Applied Psychology, 71, 662-671.
6. Benson PG, Buckley MR, Hall S. (1988). The impact of rating scale format on rater accuracy: An evaluation of the mixed standard scale. Journal of Management, 14, 415-423.
7. Bernardin HJ, Smith PC. (1981). A clarification of some issues regarding the development and use of behaviorally anchored rating scales (BARS). Journal of Applied Psychology, 66, 458-463.
8. Bernardin HJ, Tyler CL, Wiese DS. (2001). A reconsideration of strategies for rater training. In Ferris GR (Ed.), Research in personnel and human resources management (pp. 221-274). Stamford, CT : JAI Press.
9. Borman WC. (1974). The rating of individuals in organizations: An alternate approach. Organizational Behavior & Human Performance, 12, 105-124.
10. Borman WC. (1977). Consistency of rating accuracy and rating errors in the judgment of human performance. Organizational Behavior & Human Performance, 20, 238-252.
11. Borman WC. (1979). Format and training effects on rating accuracy and rater errors. Journal of Applied Psychology, 64, 410-421.
12. Borman WC, Brush DH (1993). More progress toward a taxonomy of managerial performance requirements. Human Performance, 6, 1-21.
13. Borman WC, Buck DE, Hanson MA, Motowidlo SJ, Stark S, Drasgow F. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965-973.
14. Campbell DJ, Fiske DW. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
15. Campbell DJ, Lee C. (1988). Self-appraisal in performance evaluation: Development versus evaluation. The Academy of Management Review, 13, 302-314.
16. Cardy R, Dobbins GH. (1994). Performance appraisal: A consideration of alternative perspectives. Cincinnati, OH : South-Western.
17. Church AH, Bracken, DW (Eds.). (1997). Special issue: 360-degree feedback systems. Group and Organization Management, 22, 149-161.
18. Conway JM. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
19. Conway JM. (1999). Distinguishing contextual performance from task performance for managerial jobs. Journal of Applied Psychology, 84, 3-13.
20. Cooper WJ. (1981). Ubiquitous halo: Sources, solutions, and a paradox. Psychological Bulletin, 59, 177-193.
21. Cronbach L. (1955). Processes affecting scores on "understanding of others" and "assumed similarity." Psychological Bulletin, 52, 177-193.
22. Davis BL, Skube CL, Hellervik LW, Gebelein SH, Sheard JL. (1996). Successful manager's handbook. Minneapolis, MN : Personnel Decisions International.
23. Deming WE. (1986). Out of the crisis. Cambridge : MIT Institute for Advanced Engineering Study.
24. DeNisi AS. (1996). Cognitive approach to performance appraisal: A program of research. New York, NY : Taylor & Francis.
25. DeNisi AS, Sonesh S. (2011). The appraisal and management of performance at work. In Zedeck S (Ed.), APA handbook of industrial and organizational psychology (Vol. 1, pp. 255-281). Washington DC : APA Press.
26. DeRue DS, Nahrgang JD, Wellman N, Humphrey SE. (2011). Trait and behavioral theories of leadership: An integration and meta-analytic test of their relative validity. Personnel Psychology. 64, 7-52.
27. Fisicaro SA. (1988). A reexamination of the relation between halo error and accuracy. Journal of Applied Psychology, 73, 239-244.
28. Ghorpade J. (2000). Managing five paradoxes of 360-degree feedback. Academy of Management Executive, 14, 140-150.
29. Goffin RD, Gellatly IR, Paunonen SV, Jackson DN, Meyer JP. (1996). Criterion validation of two approaches to performance appraisal: The behavioral observation scale and the relative percentile method. Journal of Business and Psychology, 11, 23-33.
30. Gorman CA, Rentsch JR. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology, 94, 1336-1344.
31. Harris MM, Schaubroeck J. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41, 43-62.
32. Hauenstein NMA. (1998). Training raters to increase accuracy of appraisals and the usefulness of feedback. In Smither JW (Ed.), Performance appraisal: State of the art in practice. (pp. 404-411). San Francisco, CA : Jossey-Bass.
33. Hauenstein NMA, Foti RJ. (1989). From laboratory to practice: Neglected issues in implementing frame-of-reference rater training. Personnel Psychology, 42, 359-379.
34. Hoffman BJ, Baldwin SP. (2012). Modern managerial assessment: A comparison of assessment centers and multisource feedback (143-162). In G Thornton & N Povah (Eds.), Assessment centers and global talent management. Burlington, VT : Gower.
35. Hoffman BJ, Lance C, Bynum B, Gentry B. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119-151.
36. Hoffman BJ, Melchers K, Blair CA, Kleinmann M, Ladd, R. (2011). Exercises and dimensions are the currency of assessment centers. Personnel Psychology, 64, 351-395.
37. Hoffman BJ, Woehr DJ. (2009). Disentangling the meaning of multisource performance rating source and dimension factors. Personnel Psychology, 62, 735-765.
38. Hoffman BJ, Woehr DJ, Maldegan R, Lyons B. (2011). Great man or great myth? A meta-analysis of the relationship between individual difference and effective leadership. Journal of Occupational and Organisational Psychology, 84, 347-381.
39. Hooijberg R, Choi J. (2000). Which leadership roles matter to whom? An examination of rater effects on perceptions of effectiveness. The Leadership Quarterly, 11, 341-364.
40. James LR. (1973). Criterion models and construct validity for criteria. Psychological Bulletin, 80, 75-83.
41. Jelley RB, Goffin RD. (2001). Can performance-feedback accuracy be improved? Effects of rater priming and rating-scale format on rating accuracy. Journal of Applied Psychology, 86, 134-144.
42. Kingstrom PO, Bass AR. (1981). A critical analysis of studies comparing behaviorally anchored ratings scales (BARS) and other rating formats. Personnel Psychology, 34, 263-289.
43. Lance CE, Hoffman BJ, Gentry B, Baranik LE. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223 - 232
44. Lance CE, Newbolt WH, Gatewood RD, Foster MR, French NR, Smith DE. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323-353.
45. Lance CE, Woehr DJ. (1989). Statistical control of halo: Clarification from two cognitive models of the performance appraisal process. Journal of Applied Psychology, 71, 679-685.
46. Landy FJ, Farr JL. (1980). Performance rating. Psychological Bulletin, 87, 72-107.
47. Lombardo MM, McCauley CD. (1994). BENCHMARKS®: A manual and trainer's guide. Greensboro, NC : Center for Creative Leadership.
48. London M. (2001). How people evaluate others in organizations. Mahwah, NJ : Erlbaum.
49. London M, Smither JW. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. Personnel Psychology, 48, 803-839.
50. MacDonald HA, Sulsky LM. (2009). Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management. Canadian Journal of Behavioral Science, 41, 227-240.
51. Mount MK, Judge TA, Scullen SE, Sytsma MR, Hezlett SA. (1998). Trait, rater, and level effects in 360-degree performance settings. Personnel Psychology, 51, 557-576.
52. Murphy KR. (2008). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 148-160.
53. Murphy KR, Balzer WK. (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 746, 19-624.
54. Murphy KR, Constans JL. (1987). Behavioral anchors as a source of bias in rating. Journal of Applied Psychology, 72, 573-577.
55. Murphy KR, Garcia M, Kerkar S, Martin C, Balzer WK. (1982). Relationship between observational accuracy and accuracy in evaluating performance. Journal of Applied Psychology, 67, 320-325.
56. Nathan BR, Tippins N. (1990). The consequences of halo error in performance ratings: A field study of the moderating effect of halo on test validation results. Journal of Applied Psychology, 75, 290-296.
57. Noonan LE, Sulsky LM. (2001). Impact of frame-of-reference and behavioral observation training on alternative training effectiveness criteria in a Canadian military sample. Human Performance, 14, 3-26.
58. Nunnally JC, Bernstein IH. (1994). Psychometric theory (3rd ed.). New York, NY : McGraw-Hill.
59. Paterson DG. (1922). The Scott Company graphic rating scale. Journal of Personnel Research, 1, 361-376.
60. Paterson DG. (1923). Methods of rating human qualities. The Annals of the American Academy of Political and Social Science, 110, 81-93.
61. Pulakos ED. (1984). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69, 581-588.
62. Pulakos ED. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes, 38, 78-91.
63. Roch SG, Mishra V, Kieszczynka U, Woehr DJ. (2010, April). Frame of reference training: An updated meta-analysis. Paper presented at the 25th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA
64. Rogelberg SG, Waclawski J. (2001). Instrumentation design. In Bracken DW, Timmreck CW, Church AH (Eds.), The handbook of multisource feedback: the comprehensive resource for designing and implementing MSF processes. San Francisco, CA : Jossey-Bass.
65. Saal FE, Downey RG, Lahey MA. (1980).Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
66. Schleicher DJ, Day DV. (1998). A cognitive evaluation of frame-of-referencetraining: Content and process issues. Organizational Behavior and Human Decision Processes, 38, 78-91.
67. Schleicher DJ, Day DV, Mayes BT, Riggio RE. (2002). A new frame for frame of reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735-746.
68. Scullen SE, Mount MK, Judge TA. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50-66.
69. Smith PC, Kendall LM. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
70. Smither JW, London M, Reilly RR. (2005). Does performance improve following multisource feedback? A theoretical model, meta-analysis and review of empirical findings. Personnel Psychology, 58, 33-66.
71. Stamoulis DT, Hauenstein NMA. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of AppliedPsychology, 78, 994-1003.
72. Sulsky WK, Balzer WK. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497-506.
73. Sulsky LM, Day DV. (1992). Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues. Journal of Applied Psychology, 77, 501-510.
74. Sulsky, LM, Day DV. (1994). Effect of frame-of-reference training on rateraccuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
75. Timmreck CW, Bracken DW. (1995, May). Upward feedback in the trenches: Challenges and realities. In Tornow WW (Chair), Upward feedback: The ups and downs of it. Symposium conducted at the 10th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL
76. Tziner A. (1984). A fairer examination of rating scales when used for performance appraisal in a real organizational setting. Journal of Organizational Behavior, 5, 103-112.
77. Tziner A, Joanis C, Murphy KR. (2000). A comparison of three methods of performance appraisal with regard to goal properties, goal perception, and ratee satisfaction. Group and Organization Management, 25, 175-190.
78. Viswesvaran C, Schmidt FL, Ones DS. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345-354.
79. Viswesvaran C, Schmidt, FL, Ones DS. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
80. Wagner SH, Goffin RD. (1997). Differences in accuracy of absolute and comparative performance appraisal methods. Organizational Behavior and Human Decision Processes, 70, 95-103.
81. Werner JM. (1994). Dimensions that make a difference: Examining the impact of in-role and extrarole behaviors on supervisory ratings. Journal of Applied Psychology, 79, 98-107.
82. Widaman KF. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
83. Woehr DJ. (2008). On the relationship between job performance ratings and ratings of job performance: What do we really know? Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 161-166.
84. Woehr DJ, Huffcutt AI. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.
85. Woehr, DJ, Miller MJ. (1997). Distributional ratings of performance: More evidence for a new rating format. Journal of Management, 23, 5, 705-721.