Rater training revisited: An updated meta-analytic review of frame-of-reference training

Journal of Occupational and Organizational Psychology

Volumn 85, Issue 2, 2012, Pages 370-395

  Roch, Sylvia G ^a   Woehr, David J ^b   Mishra, Vipanchi ^a   Kieszczynska, Urszula ^a  

^a STATE UNIVERSITY OF NEW YORK   (United States)

^b UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84860657064     PISSN: 09631798     EISSN: 20448325     Source Type: Journal    
DOI: 10.1111/j.2044-8325.2011.02045.x     Document Type: Article

References (109)

1. Angkaw, A. C., Tran, G. Q., & Haaga, D. A. F.(2006). Effects of training intensity on observers' ratings of anxiety, social skills, and alcohol-specific coping skills. Behaviour Research and Therapy, 44, 533-544.
2. Aguinis, H., Mazurkiewicz, M. D., & Heggestad, E. D.(2009). Using web-based frame-of-reference training to decrease biases in personality-based job analysis: An experimental field study. Personnel Psychology, 62, 405-438.
3. Allen, L. B.(1999). Assessing the effects of rater training on types and frequency of rater errors in performance appraisals. (Unpublished doctoral dissertation). Stephen F. Austin State University, Nacogdoches, TX
4. Athey, T. R., & McIntyre, R. M. (1987). Effect of rater training on rater accuracy: Levels-of- processing theory and social facilitation theory perspectives. Journal of Applied Psychology, 72, 567-572.
5. Bernardin, H. J., & Buckley, R. M. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
6. Bernardin, H. J., Buckley, M. R., Tyler, C. L., & Wiese, D. S. (2000). A reconsideration of strategies in rater training. Research in Personnel and Human Resources Management, 18, 221-274.
7. Bernardin, H. J., Tyler, C. L., & Villanova, P. (2009). Rating level and accuracy as a function of rater personality. International Journal of Selection and Assessment, 17, 300-310.
8. Borman, W. C. (1977). Consistency of rating accuracy and rating errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238-252.
9. Borman, W. C. (1979). Format and training effects on rating accuracy and rater errors. Journal of Applied Psychology, 64(4), 410-421.
10. Cardy, R. L., & Keefe, T. J.(1994). Observational purpose and evaluation articulation in frame of reference training: The effects of alternative processing modes on rating accuracy. Organizational Behavior and Human Decision Processes, 57, 338-357.
11. Cellar, D. F., Curtis J. R., Kohlepp, K., Poczapski, P., & Mohiuddin, S. (1989). The effect of rater training, job analysis format and congruence of training on job evaluation ratings. Journal of Business and Psychology, 3, 387-401.
12. Cesare, S. J.(1989). Accuracy of performance measurement: An investigation of training method and amount of practice. (Unpublished doctoral dissertation). Old Dominion University, Norfolk, VA
13. Chirico, K. E., Buckley, M. R., Wheeler, A. R., Facteau, J. D., Bernardin, H. J., & Beu, D. S. (2004). A note on the need for true scores in frame-of-reference (FOR) training research. Journal of Managerial Issues, 16, 382-395.
14. Cleveland, J. N., Murphy, K. R., & Williams, R. E. (1989). Multiple uses of performance appraisal: Prevalence and correlates. Journal of Applied Psychology, 74, 130-135.
15. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
16. Cook, T. D., Campbell, D. T., & Peracchio, L. (1990). Quasi experimentation. In M. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp.491-576). Palo Alto, CA: Consulting Psychologists Press, Inc.
17. Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.
18. Cronbach, L. J. (1955). Processes affecting scores on "understanding of others" and "assumed similarity". Psychological Bulletin, 52, 177-193.
19. Day, D. V., & Sulsky, L. M.(1995). Effects of frame-of-reference training and information configuration on memory organization and rating accuracy. Journal of Applied Psychology, 80, 158-167.
20. DeNisi, A. S., & Williams, K. J. (1988). Cognitive approaches to performance appraisal. Research in Personnel and Human Resources Management, 6, 109-155.
21. Dierdorff, E. C., Surface, E. A., & Brown, K. G. (2010). Frame-of-reference training effectiveness: Effects of goal orientation and self-efficacy on affective, cognitive, skill-based, and transfer outcomes. Journal of Applied Psychology, 95, 1181-1191.
22. Fehrmann, M. L., Woehr, D. J., & Arthur, W. (1991).The Angoff cutoff score method: The impact of frame-of-reference rater training. Educational and Psychological Measurement, 51, 857-872.
23. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66, 127-148.
24. Fiske, S. T., & Neuberg, S. L.(1990). A continuum of impression formation, from category-based to individuating process: Influences of information and motivation on attention and interpretation. Advances in Experimental Social Psychology, 23, 1-73.
25. Fiske, S. T., & Taylor, S. E. (1984). Social cognition. Reading, MA: Addison- Wesley.
26. Gaugler, B. B., & Thornton, G. C. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74, 611-618.
27. Gorman, C. A., Moore, L. E., Collins, J. A., Godbey, J. N., Overstreet, B. L., & Stoner, J. A. (2009). Evaluating the effectiveness of frame-of-reference scales for improving rating accuracy. Poster presented at the 24th annual meeting of the Society of Industrial and Organizational Psychology, New Orleans, LA
28. Gorman C. A., & Rentsch, J. R. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology, 94, 1336-1344.
29. Gorman, C. A., Rentsch, J. R., & Gaddis, K. A.(2010). Frame-of-reference training and performance schema accuracy: A replication and extension. Unpublished manuscript. Radford, VA: Radford University.
30. Goodstone, M. S., & Lopez, F. E. (2001). The frame of reference approach as a solution to an assessment center dilemma. Consulting Psychology Journal: Practice and Research, 53, 96-107.
31. Hamilton, D. L., & Trolier, T. K. (1986). Stereotypes and stereotyping: An overview of the cognitive approach. In J. F. Dovidio & S. L. Gaertner (Eds.). Prejudice, discrimination and racism, (pp. 127-163). Orlando, FL: Academic Press.
32. Hartog, S. B. (1991). A systematic evaluation of the components of frame-of-reference training and their effects on rating error, accuracy, and individual cognitive processes. (Unpublished doctoral dissertation). City University of New York, New York, NY
33. Hastie, R., & Park, B. (1986). The relationship between memory and judgment depends on whether the judgment task is memory-based or on-line. Psychological Review, 93, 258-268.
34. Hauenstein, N. M. A., Facteau, J., & Schmidt, J. (1999). Rater variability training: An alternative to rater error training and frame of reference training. Paper presented at the 14th Annual Society of Industrial/Organizational Psychology, Atlanta, GA
35. Hauenstein, N. M. A., & Foti, R. J.(1989). From laboratory to practice: Neglected issues in implementing frame-of-reference training. Personnel Psychology, 42, 359-378.
36. Hedge, J. W., & Teachout, M. S. (2000). Exploring the concept of acceptability as a criterion for evaluation performance measures. Group and Organization Management, 25, 22-44.
37. Hessler, R.(1992). The effect of frame-of reference training on evaluation accuracy in a simulated selection interview. (Unpublished doctoral dissertation). California State University, Long Beach, CA
38. Hoffman, J., & Sulsky, L. M.(1998). Cognitive processing biases and frame-of-reference training effectiveness. Paper presented at the 13th Annual Society of Industrial/Organizational Psychology, Dallas, TX
39. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage Publishers.
40. Ilgen, D. R., & Feldman, J. M.(1983). Performance appraisal: A process focus. Research in Organizational Behavior, 5, 141-197.
41. Jackson, D. J. R., Atkins, S. G., & Fletcher, R. B.(2005). Frame of reference training for assessment centers: Effects on interrater reliability when rating behaviors and ability traits. Public Personnel Management, 34, 17-30.
42. Jen, M. M.(1998). Using training to increase accuracy of performance appraisals. (Unpublished doctoral dissertation). Georgia Institute of Technology, Atlanta, GA
43. Johnson, Jr., R. L. (1987). The influence of direct versus indirect observation, candidate report format, and assessor training on the accuracy of assessor ratings. (Unpublished doctoral dissertation). Old Dominion University, Norfolk, VA
44. Keown-Gerrard, J. L., & Sulsky, L. M. (2001). The effects of task information training and frame-of-reference training with situational constraints on rating accuracy. Human Performance, 14, 305-320.
45. Keefe, T. J. (1988). Frame of reference training: A cognitive approach. (Unpublished doctoral dissertation). State University of New York at Buffalo, New York, NY
46. Klehe, U.-C., König, C. J., Richter, G. M., Kleinmann, M., & Melchers, K. G.(2008). Transparency in structured interviews: Consequences for construct and criterion-related validity. Human Performance, 21, 107-137.
47. Kraiger, K. (2002). Decision-based evaluation. In K. Kraiger (Ed.), Creating, implementing, and managing effective training and development: State of the art lessons for practice (pp. 331-375). San Francisco, CA: Jossey-Bass.
48. Kirkpartrick, D. L.(1994). Evaluating training programs: The four levels. San Franscisco, CA: Berrett-Koehler.
49. Lee, J. A. (1994). The effects of cognitive style and training on performance ratings' validity. Journal of Business and Psychology, 8, 297-308.
50. Landy, F. J., & Farr, J. L.(1980). Performance rating. Psychological Bulletin, 87, 72-107.
51. Levy, P. E., & Williams, J. R. (2004). The social context of performance appraisal: A review and framework for the future. Journal of Management, 30, 881-905.
52. Lewis, C. F. (2002). The effects of consensus process expectations and rater training strategies on rater accuracy, interrater agreement, and behavior recall in an assessment center simulation. (Unpublished doctoral dissertation). University of Missouri, St. Louis, MO
53. Lichtenstein, M., & Srull, T. K.(1987). Processing objective as a determinant of the relationship between recall and judgment. Journal of Experimental Social Psychology, 23, 93-118.
54. Lievens, F. (2001). Assessor training strategies and their effect on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86, 255-264.
55. Lievens, F., & Sanchez, J. I. (2007). Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment. Journal of Applied Psychology, 92, 812-819.
56. London, M., Mone, E. M., & Scott, J. C. (2004). Performance management and assessment: Methods for improved rater accuracy and employee goal setting. Human Resource Management, 43, 319-336.
57. Lord, R. G. (1985). Accuracy in behavioral measurement: An alternative definition based on rater's cognitive schema and signal detection theory. Journal of Applied Psychology, 70, 66-71.
58. Lundstrom, J. T.(2008). A new use of frame-of-reference training: Improving reviewers' inferences from biodata information. (Unpublished doctoral dissertation). Kansas State University, Manhattan, KS
59. McCauley, D. P., Jago, I. A., Gore, B., Lance, C. E., Quarles, F. K., Sledge, L., ... Guest, F. D. (1990, April). The development of more ecologically valid videotapes for use in performance appraisal research. Paper presented at the annual meeting of the Southeastern Psychological Association, Atlanta, GA
60. McIntyre, R. M., Smith, D. E., & Hassett, C. E. (1984). Accuracy of performance ratings as affected by rater training and perceived purpose of ratings. Journal of Applied Psychology, 69, 147-156.
61. Melchers, K. G., Kleinmann, M., & Prinz, M. A. (2010). Do assessors have too much on their plates? The effects of simultaneously rating multiple assessment center candidates on rating quality. International Journal of Selection and Assessment, 18, 329-341.
62. Melchers, K. G., Lienhardt, N., von Aarburg, M., & Kleinmann, M. (2011). Is more structure always better? An evaluation of the effects of rater training and descriptively anchored rating scales on rating accuracy in a structured interview. Personnel Psychology, 64, 53-87.
63. Mills, N. E. (1999). A cognitive approach to rater training. (Unpublished doctoral dissertation). Kansas State University, Manhattan, KS
64. Murphy, K. R. (1991). Criterion issues in performance appraisal research: Behavioral accuracy versus classification accuracy. Organizational Behavior and Human Decision Processes, 50, 45-50.
65. Murphy, K. R., & Balzer, W. K.(1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619-624.
66. Murphy, K. R., & Cleveland, J. N.(1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: SAGE Publications, Inc.
67. Murphy, K. R., Garcia, M., Kerkar, S., Martin, C., & Balzer, W. K. (1982). Relationship between observational accuracy and accuracy in evaluating performance. Journal of Applied Psychology, 67, 320-325.
68. Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218-225.
69. Noonan, L. E., & Sulsky, L. M. (2001). Impact of frame of reference and behavioral observation training on alternative training effectiveness criteria in a Canadian military sample. Human Performance, 14, 3-26.
70. Padgett, M. Y., & Ilgen, D. R.(1989). The impact of ratee performance characteristics on rater cognitive processes and alternative measures of rater accuracy. Organizational Behavior and Human Decision Process, 44, 232-260.
71. Pelly, D. A., & Dossetr, D. L. (1991). The effects of rater training strategies on rating accuracy. Paper presented at the 6th annual meeting of the Society for Industrial and Organizational Psychology, St. Louis, MO
72. Powell, D. M., & Goffin, R. D.(2009). Assessing personality in the employment interview: The impact of training on rater accuracy. Human Performance, 22, 450-465.
73. Pulakos, E. D.(1984a). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69, 581-588.
74. Pulakos, E. D. (1984b). The interactive effects of format and training on performance appraisal accuracy. (Unpublished doctoral dissertation). Michigan State University, East Lansing, MI
75. Pulakos, E. D. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes, 38, 76-91.
76. Roach, D. W., & Gupta, N.(1992). A realistic simulation for assessing the relationship among components of rating accuracy. Journal of Applied Psychology, 77, 196-200.
77. Roberson, L., & Banks, C. G. (1986). Beyond job knowledge: Assessment skill training to increase rating accuracy. Paper presented at the American Psychological Association, Washington, DC
78. Roch, S. G., & O'Sullivan, B. J. (2003). Frame-of-reference rater training issues: Recall, time and behavior observation training. International Journal of Training and Development, 7, 93-107.
79. Rosch, E. (1978). Principles of categorization. In E. Rosch & B. B. Lloyd (Eds.), Cognition and categorization (pp. 27-48). Hillsdale, NJ: Erlbaurm.
80. Roth, L.(1987). Rating errors and accuracy as a function of rater training and rating use. (Unpublished doctoral dissertation). Tulane University, New Orleans, LA
81. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the rating: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
82. Sanchez, J. I., & De La Torre, P. (1996). A second look at the relationship between rating and behavioral accuracy in performance appraisal. Journal of Applied Psychology, 81, 3-10.
83. Sauser, W. I. (1979). A comparative evaluation of the effects of rater participation and rater training on characteristics of employee performance appraisal ratings and related mediating variables. In Dissertation abstracts international (Vol. 39, p. 5116). Georgia Institute of Technology, Atlanta, GA
84. Schleicher, D. J.(1998). A new "frame" for frame of reference training: Enhancing the construct validity of assessment centers. (Unpublished doctoral dissertation). Pennsylvania State University, University Park, PA
85. Schleicher, D. J., & Day, D. V. (1998). A cognitive evaluation of frame-of-reference rater training: Content and process issues. Organizational Behavior and Human Decision Processes, 73, 76-101.
86. Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame of reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735-746.
87. Schlientz, M. D., Briesch, A. M., Walcott, C. M., & Chafouleas, S. M. (2009). The impact of training on the accuracy of direct behavior rating (DBR). School Psychology Quarterly, 24, 73-83.
88. Schmidt, F. L., & Le, H. (2004). Software for the Hunter-Schmidt meta-analysis methods. Iowa City, IA: University of Iowa.
89. Senderak, M. P. (1983). The effects of rater schemata training and rating task simplification on rating effectiveness. (Unpublished doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg, VA
90. Sinclair, A. L., & Hauenstein, N. M. A. (2001). Differentiating rater accuracy training programs. Paper presented at the 16th Annual Society of Industrial/Organizational Psychology, San Diego, CA
91. Snodgrass, J. G., & Corwin, J.(1988). Pragmatics of measuring recognition memory: Applications to dementia and amnesia. Journal of Experimental Psychology: General, 117, 34-50.
92. Stamoulis, D. T., & Hauenstein, N. M. A. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of Applied Psychology, 78, 994-1003.
93. Stevens, S. E. (1995). The effect of frame-of-reference rater training on idiosyncratic and normative raters. (Unpublished doctoral dissertation). University of Akron, Akron, OH
94. Sulsky, L. M. (1988). An empirical investigation of two rater training programs. (Unpublished doctoral dissertation). Bowling Green State University, Bowling Green, OH
95. Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497-506.
96. Sulsky, L. M., & Day, D. V. (1992). Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues. Journal of Applied Psychology, 77, 501-510.
97. Sulsky, L. M., & Day, D. V. (1994). Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
98. Sulsky, L. M., & Kline, T. J. B. (2007). Understanding frame-of-reference training success: A social learning theory perspective. International Journal of Training and Development, 11, 121-131.
99. Sulsky, L. M., Skarlicki, D. P., & Keown, J. L. (2002). Frame-of -reference training: Overcoming the effects of organizational citizenship behavior of performance rating accuracy. Journal of Applied Social Psychology, 32, 1224-1240.
100. Sulsky, L. M., & Uggerslev, K. (2010). What is taught affects how it should be taught: Implications of frame-of-reference training protocols and performance theory characteristics on training effectiveness. Unpublished manuscript.
101. Sulsky, L. M., Uggerslev, K. L., Day, D. V., Keown, J. L., & Law, J. F. (2002). The frame-of-reference training protocol: All feedback is not created equal. Poster presentation at the 17th annual conference for the Society for Industrial and Organizational Psychology, Toronto, Ontario.
102. Svoboda, M. D. (1986). The measurement of change in performance appraisal rater training. (Unpublished doctoral dissertation). Louisiana State University, Baton Rouge, LA
103. Uggerslev, K. L. (2004). Rater ability and motivation in frame-of-reference training implications of rater idiosyncrasy and theory agreement for accuracy and rater reactions. (Unpublished doctoral dissertation). University of Calgary, Alberta, Canada
104. Uggerslev, K. L., & Sulsky, L. M. (2007). Idiosyncratic performance theories and frame-of-reference training: The effects on accuracy and rater reactions. Presented at the 22nd annual conference for the Society for Industrial and Organizational Psychology, New York, NY
105. Uggerslev, K.L., & Sulsky, L.M.(2008). Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy. Journal of Applied Psychology, 93, 711-719.
106. Uggerslev, K. L., Sulsky, L. M., & Day, D. V. (2003). Effects of performance theory characteristics and training protocols on frame-of-reference training. Presented in the interactive poster session, 'The Effects of Democracy and Training Protocols on Student Groups, ' at the 63rd annual meeting for the Academy of Management, Seattle, WA
107. Woehr, D. J. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79, 525-534.
108. Woehr, D. J., & Feldman, J. M.(1993). Processing objective and question order effects on the causal relation between memory and judgment: The tip of the iceberg. Journal of Applied Psychology, 78, 232-241.
109. Woehr, D. J., & Huffcutt, A. I.(1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.