Exploring the role of first impressions in rater-based assessments

Advances in Health Sciences Education

Volumn 19, Issue 3, 2014, Pages 409-427

  Wood, Timothy J ^a  

^a UNIVERSITY OF OTTAWA   (Canada)

Author keywords

First impressions; Rater based assessment; Rater cognition

Indexed keywords

COGNITION; DECISION MAKING; EDUCATION; HUMAN; MEDICAL EDUCATION; OBSERVER VARIATION; STANDARDS;

COGNITION; DECISION MAKING; EDUCATION, MEDICAL; EDUCATIONAL MEASUREMENT; HUMANS; JUDGMENT; OBSERVER VARIATION;

EID: 84903887567     PISSN: 13824996     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10459-013-9453-9     Document Type: Review

References (97)

1. AERA, APA, & NCME. (1999). Standards for educational and psychological testing (pp. 9-24). Washington, DC: American Educational Research Association.
2. Ambady, N. (2010). The perils of pondering: Intuition and thin slice judgments. Psychological Inquiry, 21(4), 271-278.
3. Ambady, N., Bernieri, F., & Richeson, J. (2000). Toward a histology of social behavior: Judgmental accuracy from thin slices of the behavioral stream. Advances in Experimental Social Psychology, 32, 201-271.
4. Ambady, N., & Gray, H. M. (2002). On being sad and mistaken: Mood effects on the accuracy of thin-slice judgments. Journal of Personality and Social Psychology, 83(4), 947-961.
5. Ambady, N., Hallahan, M., & Conner, B. (1999). Accuracy of judgments of sexual orientation from thin slices of behavior. Journal of Personality and Social Psychology, 77(3), 538-547.
6. Ambady, N., Hallahan, M., & Rosenthal, R. (1995). On judging and being judged accurately in zero-acquaintance situations. Journal of Personality and Social Psychology, 69(3), 518-529.
7. Ambady, N., & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin, 111(2), 256-274.
8. Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64(3), 431-441.
9. Ames, D. R., Kammrath, L. K., Suppes, A., & Bolger, N. (2010). Not so fast: The (not-quite-complete) dissociation between accuracy and confidence in thin-slice impressions. Personality and Social Psychology Bulletin, 36(2), 264-277.
10. Babad, E., Avni-Babad, D., & Rosenthal, R. (2004). Prediction of students' evaluations from brief instances of professors' nonverbal behavior in defined instructional situations. Social Psychology of Education, 7(1), 3-33.
11. Balzer, W. K., & Sulsky, L. M. (1992). Halo and performance appraisal research: A critical examination. Journal of Applied Psychology, 77(6), 975-985.
12. Bargh, J. A. (1992). The ecology of automaticity: Toward establishing the conditions needed to produce automatic processing effects. The American Journal of Psychology, 105(2), 181-199.
13. Barrick, M. R., Shaffer, J. A., & DeGrassi, S. W. (2009). What you see may not be what you get: Relationships among self-presentation tactics and ratings of interview and job performance. The Journal of Applied Psychology, 94(6), 1394-1411.
14. Barrick, M. R., Swider, B. W., & Stewart, G. L. (2010). Initial evaluations in the interview: Relationships with subsequent interviewer evaluations and employment offers. The Journal of Applied Psychology, 95(6), 1163-1172.
15. Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors perspectives. Advances in Health Sciences Education: Theory and Practice. doi: 10. 1007/s10459-012-9392-x.
16. Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65(1), 60-66.
17. Biesanz, J. C., Human, L. J., Paquin, A. C., Chan, M., Parisotto, K. L., Sarracino, J., et al. (2011). Do we know when our impressions of others are valid? Evidence for realistic accuracy awareness in first impressions of personality. Social Psychological and Personality Science, 2(5), 452-459.
18. Borkenau, P., & Liebler, A. (1992). Trait inferences: Sources of validity at zero acquaintance. Journal of Personality and Social Psychology, 62(4), 645-657.
19. Brooks, L. R. (2005). The blossoms and the weeds. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Expérimentale, 59(1), 62-74.
20. Carney, D., Colvin, C., & Hall, J. (2007). A thin slice perspective on the accuracy of first impressions. Journal of Research in Personality, 41(5), 1054-1072.
21. Chan, M., Rogers, K. H., Parisotto, K. L., & Biesanz, J. C. (2011). Forming first impressions: The role of gender and normative accuracy in personality perception. Journal of Research in Personality, 45(1), 117-120.
22. Clauser, B. E., Margolis, M. J., & Swanson, D. B. (2008). Issues of validity and reliability for assessments in Medical Education. In E. S. Holmboe & R. E. Hawkins (Eds.), Practical guide to the evaluation of clinical competence (pp. 10-23). Philadelphia: Mosby Elsevier.
23. Colvin, C. R., & Funder, D. C. (1991). Predicting personality and behavior: A boundary on the acquaintanceship effect. Journal of Personality and Social Psychology, 60(6), 884-894.
24. Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166. e7-166. e16.
25. Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. A. (2008). Effect of rater training on reliability and accuracy of mini-cex scores: A randomized, controlled trial. Journal of General Internal Medicine, 24(1), 74-79.
26. Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218-244.
27. Croskerry, P. (2009). Clinical cognition and diagnostic error: Applications of a dual process model of reasoning. Advances in Health Sciences Education, 14, 27-35.
28. DeNisi, A. S., Cafferty, T. P., & Meglino, B. M. (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behaviour and Human Performance, 33, 360-396.
29. Dijksterhuis, A., Bos, M. W., Nordgren, L. F., & van Baaren, R. B. (2006). On making the right choice: The deliberation-without-attention effect. Science, 311(5763), 1005-1007.
30. Dipboye, R. L. (1982). Self-fulfilling prophecies in the selection-recruitment interview. The Academy of Management Review, 7(4), 579.
31. Dodson, M., Crotty, B., Prideaux, D., Carne, R., Ward, A., & De Leeuw, E. (2009). The multiple mini-interview: How long is long enough? Medical Education, 43(2), 168-174.
32. Dougherty, T. W., Turban, D. B., & Callender, J. C. (1994). Confirming first impressions in the employment interview: A field study of interview behaviour. Journal of Applied Psychology, 5(5), 659-665.
33. Downing, S. M., & Haladyna, T. M. (2009). Validity and its threats. In S. M. Downing & R. Yudkowsky (Eds.), Assessment in health professions education (pp. 21-56). New York: Routledge.
34. Eva, K. W., & Norman, G. R. (2005). Heuristics and biases-A biased perspective on clinical reasoning. Medical Education, 39(9), 870-872.
35. Eva, K. W., & Regehr, G. (2011). Exploring the divergence between self-assessment and self-monitoring. Advances in Health Sciences Education: Theory and Practice, 16(3), 311-329.
36. Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255-278.
37. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Applied Psychology, 66(2), 127-148.
38. Fisicaro, S. A., & Lance, C. E. (1990). Implications of three causal models for the measurement of halo error. Applied Psychological Measurement, 14(4), 419-429.
39. Fiske, S., & Neuberg, S. (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation. In M. Zanna (Ed.), Advances in experimental social psychology (23rd ed., pp. 1-75). San Diego: Academic Press Inc.
40. Funder, D. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgment. Psychological Bulletin, 101(1), 75-90.
41. Funder, D. C., & West, S. G. (1993). Consensus, self-other agreement, and accuracy in personality judgment: A introduction. Journal of Personality, 61(4), 457-476.
42. Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision making. Annual Review of Psychology, 62, 451-482.
43. Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine, 86(10), S1-S7.
44. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780-786.
45. Goffin, R. D., Jelley, R. B., & Wagner, S. H. (2003). Is halo helpful? Effects of inducing halo on performance rating accuracy. Social Behaviour and Personality, 31(6), 625-636.
46. Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education: Theory and Practice, 16(2), 151-165.
47. Govaerts, M. J. B., van de Wiel, M. W. J., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters' performance theories and constructs. Advances in Health Sciences Education: Theory and Practice. doi: 10. 1007/s10459-012-9376-x.
48. Harris, M., & Garris, C. (2008). You never get a second chance to make a first impression. In N. Ambady & J. Skowronski (Eds.), First impressions (pp. 147-168). New York, NY: Guilford Press.
49. Hasher, L., & Zacks, R. T. (1979). Automatic and effortful processes in memory. Journal of Experimental Psychology: General, 108(3), 356-388.
50. Hawkins, R. E., & Boulet, J. R. (2008). Direct observation: Standardized patients. In E. S. Holmboe & R. E. Hawkins (Eds.), Evaluation of clinical competence (pp. 102-118). Philadelphia, PA: Mosby Elsevier.
51. Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676-682.
52. Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5(1), 64-65.
53. Iramaneerat, C., & Yudkowsky, R. (2007). Rater errors in a clinical skills assessment of medical students. Evaluation and the Health Professions, 30(3), 266-283.
54. Jacoby, L. L. (1991). A process dissociation framework: Separating automatic from intentional uses of memory. Journal of Memory and Language, 30, 513-541.
55. Jacoby, L., & Kelley, C. (1990). An episodic view of motivation: Unconscious influences of memory. In E. T. Higgins & R. M. Sorrentino (Eds.), Handbook of motivation and cognition: Foundations of social behavior (Vol. 2, pp. 451-480). New York, NY: Guilford Press.
56. Johnston, J. H., Driskell, J. E., & Salas, E. (1997). Vigilant and hypervigilant decision making. The Journal of Applied Psychology, 82(4), 614-622.
57. Kahneman, D. (2011). Thinking, fast and slow. Canada: Doubleday.
58. Kenny, D. A. (1993). A coming-of-age for research on interpersonal perception. Journal of Personality, 61(4), 789-807.
59. Kenny, D. A., & Albright, L. (1987). Accuracy in interpersonal perception: A social relations analysis. Psychological Bulletin, 102(3), 390-402.
60. Klein, G. (2009). Streetlights and shadows: Searching for the keys to adaptive decision making. Cambridge, MA: MIT Press.
61. Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048-1060.
62. Lance, C. E., LaPointe, J. A., & Stewart, A. M. (1994). A test of the context dependency of three causal models of halo rater error. Journal of Applied Psychology, 79(3), 332-340.
63. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72-107.
64. Lippa, R. A., & Dietz, J. K. (2000). The relation of gender, personality and intelligence to judges' accuracy in judging strangers' personality from brief video segments. Journal of Nonverbal Behavior, 24(1), 25-43.
65. Logan, G. D. (1992). Attention and preattention in theories of automaticity. The American Journal of Psychology, 105(2), 317-339.
66. Macan, T. H., & Dipboye, R. L. (1990). The relationship of interviewrs' preinterview impressions to selection and recruitment outcomes. Personnel Psychology, 43(4), 745-768.
67. Monteiro, S. D., Sherbino J. D., Ilgen, J. S., Dore, K. L. Gaissmaier, W., Wood, T. J., et al. (unpublished manuscript). Diagnosing Fast and Slow: The Effect of Interruptions on Speeded and Reflective Clinical Reasoning.
68. Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78(2), 218-225.
69. Nathan, B. R., & Tippins, N. (1990). The consequences of halo "error" in performance ratings: A field study of the moderating effect of halo on test validation results. Journal of Applied Psychology, 75(3), 290-296.
70. Norman, G. (2009). Dual processing and diagnostic errors. Advances in Health Sciences Education, 14, 37-49.
71. Norman, G. R., & Eva, K. W. (2010). Diagnostic error and clinical reasoning. Medical Education, 44, 94-100.
72. Norman, G. R., Sherbino, J., Dore, K. L., Wood, T. J. Ph. Young, M. E., Gaissmaier, W., et al. (in press). The etiology of diagnostic errors: A controlled trial of System 1 vs. System 2 reasoning. Academic Medicine.
73. Norman, G., Young, M., & Brooks, L. (2007). Non-analytical models of clinical reasoning: The role of experience. Medical Education, 41, 1140-1145.
74. Patterson, M. L., & Stockbridge, E. (1998). Effects of cognitive demand and judgment strategy on person perception accuracy. Journal of Nonverbal Behavior, 22(4), 253-263.
75. Pelaccia, T., Tardif, J., Triby, E., & Charlin, B. (2011). An analysis of clinical reasoning through a recent and comprehensive approach: The dual-process theory. Medical Education Online, 16, 5890.
76. Rosenthal, R. (1994). Interpersonal expectancy effects: A 30-year perspective. Current Directions in Psychological Science, 3(6), 176-179.
77. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413-428.
78. Schneider, W., & Chein, J. M. (2003). Controlled & automatic processing: Behavior, theory, and biological mechanisms. Cognitive Science, 27(3), 525-559.
79. Sherbino, J., Dore, K. L., Wood, T. J., Young, M. E., Gaissmaier, W., Krueger, S., et al. (2012). On the relation between processing speed and diagnostic error. Academic Medicine, 87(6), 785-791.
80. Smith, H. J., Archer, D., & Costanzo, M. (1991). "Just a hunch": Accuracy and awareness in person perception. Journal of Nonverbal Behavior, 15(1), 3-18.
81. Snyder, M., Tanke, E., & Berscheid, E. (1977). Social perception and interpersonal behavior: On the self-fulfilling nature of social stereotypes. Journal of Personality and Social Psychology, 35(9), 656-666.
82. Stroud, L., Herold, J., Tomlinson, G., & Cavalcanti, R. B. (2011). Who you know or what you know? Effect of examiner familiarity with residents on OSCE scores. Academic Medicine, 86(10), S8-S11.
83. Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education: Theory and Practice. doi: 10. 1007/s10459-012-9370-3.
84. Tom, G., Tong, S. T., & Hesse, C. (2009). Thick slice and thin slice teaching evaluations. Social Psychology of Education, 13(1), 129-136.
85. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.
86. Uleman, J. S., Saribay, S. A., & Gonzalez, C. M. (2008). Spontaneous inferences, implicit impressions, and implicit theories. Annual Review of Psychology, 59, 329-360.
87. van der Vleuten, C. P. M., & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine, 2(2), 58-76.
88. van Merriënboer, J. J. G., & Sweller, J. (2010). Cognitive load theory in health professional education: Design principles and strategies. Medical Education, 44(1), 85-93.
89. Wigton, R. (1980). The effects of student personal characteristics on the evaluation of clinical performance. Journal of Medical Education, 55, 423-427.
90. Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270-292.
91. Willis, J., & Todorov, A. (2006). Making up your mind after a 100-ms exposure to a face. Psychological Science, 17(7), 592-598.
92. Wilson, T. D., & Schooler, J. W. (1991). Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60(2), 181-192.
93. Woehr, D. J., Day, D. V., Winfred, A., & Bedeian, A. G. (1998). The systematic distortion hypothesis: A confirmatory test of the implicit covariance and general impression models. Basic and Applied Social Psychology, 16(4), 417-434.
94. Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.
95. Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education: Theory and Practice. doi: 10. 1007/s10459-012-9396-6.
96. Yaphe, J., & Street, S. (2003). How do examiners decide?: A qualitative study of the process of decision making in the oral examination component of the MRCGP examination. Medical Education, 37(9), 764-771.
97. Yeates, P., O'Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently. Advances in Health Sciences Education: Theory and Practice. doi: 10. 1007/s10459-012-9372-1.