Is more structure really better? A comparison of frame-of-reference training and descriptively anchored rating scales to improve interviewers' rating quality

Personnel Psychology

Volumn 64, Issue 1, 2011, Pages 53-87

  Melchers, Klaus G ^a   Lienhardt, Nadja ^a   Von Aarburg, Miriam ^a   Kleinmann, Martin ^a  

^a UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 79951612365     PISSN: 00315826     EISSN: 17446570     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2010.01202.x     Document Type: Article

References (69)

1. Athey TR, McIntyre RM. (1987). Effect of rater training on rater accuracy: Levels-of-processing theory and social facilitation theory perspectives. Journal of Applied Psychology, 72, 567-572.
2. Bernardin HJ, Buckley MR, Tyler CL, Wiese DS. (2000). A reconsideration of strategies for rater training. In Ferris GL (Ed.), Research in personnel and human resources management (Vol. 18, pp. 221-274). Greenwich, CT JAI Press.
3. Bernardin HJ, Pence EC. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60-66.
4. Buckley MR, Norris AC, Wiese DS. (2000). A brief history of the selection interview: May the next 100 years be more fruitful. Journal of Management History, 6, 113-126.
5. Burke MJ, Dunlap WP. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5, 159-172.
6. Burke MJ, Finkelstein LM, Dusig MS. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
7. Campion MA, Palmer DK, Campion JE. (1997). A review of structure in the selection interview. Personnel Psychology, 50, 655-702.
8. Chapman DS, Zweig DI. (2005). Developing a nomological network for interview structure: Antecedents and consequences of the structured selection interview. Personnel Psychology, 58, 673-702.
9. Cohen J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
10. Conway JM, Jako RA, Goodman DF. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80, 565-579.
11. Craik FI. (2002). Levels of processing: Past, present... and future Memory, 10, 305-318.
12. Craik FI, Lockhart RS. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671-684.
13. Day DV, Sulsky LM. (1995). Effects of frame-of-reference training and information configuration on memory organization and rating accuracy. Journal of Applied Psychology, 80, 158-167.
14. Deller J, Kleinmann M. (1993). Das situative interview [The situational interview]. In Gebert A, Hacker W (Eds.), Arbeits- und Organisationspsychologie 1991 in Dresden (pp. 336-343). Bonn, Germany Deutscher Psychologen Verlag.
15. Dipboye RL. (1997). Structured selection interviews: Why do they work? Why are they underutilized? In Anderson N, Herriot P (Eds.), International handbook of selection and assessment (pp. 455-473). Chichester, UK Wiley.
16. Dipboye RL, Gaugler BB. (1993). Cognitive and behavioral processes in the selection interview. In Schmitt N, Borman WC (Eds.), Personal selection in organizations (pp. 135-170). San Francisco, CA Jossey-Bass.
17. Dreher GF, Ash RA, Hancock P. (1988). The role of the traditional research design in underestimating the validity of the employment interview. Personnel Psychology, 41, 315-327.
18. WG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362.
19. Goodman JS, Wood RE. (2004). Feedback specificity, learning opportunities, and learning. Journal of Applied Psychology, 89, 809-821.
20. Goodman JS, Wood RE. (2009). Faded versus increasing feedback, task variability trajectories, and transfer of training. Human Performance, 22, 64-85.
21. Goodman JS, Wood RE, Hendrickx M. (2004). Feedback specificity, exploration, and learning. Journal of Applied Psychology, 89, 248-262.
22. Highhouse S. (2008). Stubborn reliance on intuition and subjectivity in employee selection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 333-342.
23. Huffcutt AI, Arthur W. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79, 184-190.
24. Huffcutt AI, Roth PL. (1998). Racial group differences in employment interview evaluations. Journal of Applied Psychology, 83, 179-189.
25. Huffcutt AI, Weekley JA, Wiesner WH, DeGroot TG, Jones C. (2001). Comparison of situational and behavior description interview questions for higher-level positions. Personnel Psychology, 54, 619-644.
26. Huffcutt AI, Woehr DJ. (1999). Further analysis of employment interview validity: A quantitative evaluation of interviewer-related structuring methods. Journal of Organizational Behavior, 20, 549-560.
27. James LR, Demaree RG, Wolf G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
28. Klehe U-C, König CJ, Richter GM, Kleinmann M, Melchers KG. (2008). Transparency in structured interviews: Consequences for construct and criterion-related validity. Human Performance, 21, 107-137.
29. Latham GP, Saari LM. (1984). Do people do what they say? Further studies on the situational interview. Journal of Applied Psychology, 69, 569-573.
30. Latham GP, Saari LM, Pursell ED, Campion MA. (1980). The situational interview. Journal of Applied Psychology, 65, 422-427.
31. LeBreton JM, Senter JL. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815-852.
32. Lievens F. (2001). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86, 255-264.
33. Lievens F, De Paepe A. (2004). An empirical investigation of interviewer-related factors that discourage the use of high structure interviews. Journal of Organizational Behavior, 25, 29-46.
34. Maas JB. (1965). Patterned scaled expectation interview: Reliability studies on a new technique. Journal of Applied Psychology, 49, 431-433.
35. Macan T. (2009). The employment interview: A review of current studies and directions for future research. Human Resource Management Review, 19, 203-218.
36. Maurer SD. (2002). A practitioner-based analysis of interviewer job expertise and scale format as contextual factors in situational interviews. Personnel Psychology, 55, 307-327.
37. Maurer SD, Fay C. (1988). Effect of situational interviews, conventional structured interviews, and training on interview rating agreement: An experimental analysis. Personnel Psychology, 41, 329-344.
38. McGraw KO, Wong SP. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
39. Melchers KG, Klehe U-C, Richter GM, Kleinmann M, König CJ, Lievens F. (2009). "I know what you want to know": The impact of interviewees' ability to identify criteria on interview performance and construct-related validity. Human Performance, 22, 355-374.
40. Murphy KR, Cleveland JN. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA Sage.
41. Noonan LE, Sulsky LM. (2001). Impact of frame-of-reference and behavioral observation training on alternative training effectiveness criteria in a Canadian military sample. Human Performance, 14, 3-26.
42. Palmer DK, Campion MA, Green PC. (1999). Interviewing training for both applicant and interviewer. In Eder RW, Harris MM (Eds.), The employment interview handbook (pp. 337-351). Thousand Oaks, CA Sage.
43. Pulakos ED. (1984). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69, 581-588.
44. Roch SG, O'Sullivan BJ. (2003). Frame of reference rater training issues: Recall, time and behavior observation training. International Journal of Training and Development, 7, 93-107.
45. Ryan AM, Daum D, Bauman T, Grisez M, Mattimore K, Nalodka T, McCormick S. (1995). Direct, indirect, and controlled observation and rating accuracy. Journal of Applied Psychology, 80, 664-670.
46. Ryan AM, McFarland L, Baron H, Page R. (1999). An international look at selection practices: Nation and culture as explanations for variability in practice. Personnel Psychology, 52, 359-391.
47. Sackett PR, Lievens F. (2008). Personnel selection. Annual Review of Psychology, 59, 419-450.
48. Salimäki A, Greenberg J. (2007, April). Attracting applicants and retaining employees by training employment interviewers in interactional justice. Paper presented at the 22nd Annual Conference of the Society for Industrial and Organizational Psychology, New York.
49. Salvemini NJ, Reilly RR, Smither JW. (1993). The influence of rater motivation on assimilation effects and accuracy in performance ratings. Organizational Behavior and Human Decision Processes, 55, 41-60.
50. Schleicher DJ, Day DV. (1998). A cognitive evaluation of frame-of-reference rater training: Content and process issues. Organizational Behavior and Human Decision Processes, 73, 76-101.
51. Schleicher DJ, Day DV, Mayes BT, Riggio RE. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735-746.
52. Schmidt FL, Zimmerman RD. (2004). A counterintuitive hypothesis about employment interview validity and some supporting evidence. Journal of Applied Psychology, 89, 553-561.
53. Schmidt RA, Bjork RA. (1992). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science, 3, 207-217.
54. Schuler H, Funke U. (1989). The interview as a multimodal procedure. In Eder RW, Ferris GR (Eds.), The employment interview: Theory, research, and practice (pp. 183-192). Newbury Park, CA Sage.
55. Schuler H, Moser K. (1995). Die validität des multimodalen interviews [Validity of the multimodal interview]. Zeitschrift für Arbeits- und Organisationspsychologie, 39, 2-12.
56. Shrout PE, Fleiss JL. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
57. Stamoulis DT, Hauenstein NM. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of Applied Psychology, 78, 994-1003.
58. Sulsky LM, Balzer WK. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497-506.
59. Sulsky LM, Day DV. (1994). Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
60. Sulsky LM, Kline TJB. (2007). Understanding frame-of-reference training success: A social learning theory perspective. International Journal of Training and Development, 11, 121-131.
61. Taylor PJ, Small B. (2002). Asking applicants what they would do versus what they did do: A meta-analytic comparison of situational and past behaviour employment interview questions. Journal of Occupational and Organizational Psychology, 75, 277-294.
62. Uggerslev KL, Sulsky LM. (2008). Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy. Journal of Applied Psychology, 93, 711-719.
63. van der Zee KI, Bakker AB, Bakker P. (2002). Why are structured interviews so rarely used in personnel selection Journal of Applied Psychology, 87, 176-184.
64. Van Iddekinge CH, Raymark PH, Roth PL, Payne HS. (2006). Comparing the psychometric characteristics of ratings of face-to-face and videotaped structured interviews. International Journal of Selection and Assessment, 14, 347-359.
65. Van Iddekinge CH, Sager CE, Burnfield JL, Heffner TS. (2006). The variability of criterion-related validity estimates among interviewers and interview panels. International Journal of Selection and Assessment, 14, 193-205.
66. Vance RJ, Kuhnert KW, Farr JL. (1978). Interview judgments: Using external criteria to compare behavioral and graphic scale ratings. Organizational Behavior and Human Decision Processes, 22, 279-294.
67. Weekley JA, Gier JA. (1987). Reliability and validity of the situational interview for a sales position. Journal of Applied Psychology, 72, 484-487.
68. Woehr DJ. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79, 525-534.
69. Woehr DJ, Huffcutt AI. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.