Workplace-based assessment: Effects of rater expertise

Advances in Health Sciences Education

Volumn 16, Issue 2, 2011, Pages 151-165

  Govaerts, M J B ^a   Schuwirth, L W T ^a   van der Vleuten, C P M ^a   Muijtjens, A M M ^a  

^a MAASTRICHT UNIVERSITY   (Netherlands)

Author keywords

Clinical education; Cognition based assessment models; Competence assessment; Performance assessment; Professional education; Professional judgment; Rater expertise; Rating process; Workplace based assessment

Indexed keywords

ARTICLE; ATTITUDE TO HEALTH; CLINICAL COMPETENCE; COGNITION; DECISION MAKING; EDUCATION; EDUCATIONAL STATUS; GENERAL PRACTITIONER; HUMAN; LEARNING; METHODOLOGY; NONPARAMETRIC TEST; TASK PERFORMANCE; WORKPLACE;

CLINICAL COMPETENCE; COGNITION; DECISION MAKING; EDUCATIONAL MEASUREMENT; EDUCATIONAL STATUS; GENERAL PRACTITIONERS; HEALTH KNOWLEDGE, ATTITUDES, PRACTICE; HUMANS; JUDGMENT; STATISTICS, NONPARAMETRIC; TASK PERFORMANCE AND ANALYSIS; VERBAL LEARNING; WORKPLACE;

EID: 79953240210     PISSN: 13824996     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10459-010-9250-7     Document Type: Article

References (46)

1. Arts, J. A. R. M., Gijselaers, W. H., & Boshuizen, H. P. A. (2006). Understanding managerial problem-solving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology, 31(4), 387-410.
2. Boshuizen, H. P. A. (1989). The development of medical expertise; a cognitive-psychological approach. Doctoral dissertation. Maastricht: Rijksuniversiteit Limburg.
3. Cardy, R. L., Bernardin, H. J., Abbott, J. G., Senderak, M. P., & Taylor, K. (1987). The effects of individual performance schemata and dimension familiarization on rating accuracy. Journal of Occupational Psychology, 60, 197-205.
4. Chi, M. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6(3), 271-315.
5. Chi, M. T. H. (2006). Two approaches to the study of experts' characteristics. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 21-30). Cambridge: Cambridge University Press.
6. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science: A Multidisciplinary Journal, 5(2), 121-152.
7. Chi, M. T. H., de Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 5, 121-152.
8. Coderre, S., Mandin, H., Harasym, P. H., & Fick, G. H. (2003). Diagnostic reasoning strategies and diagnostic success. Medical Education, 37, 695-703.
9. Cunnington, J., & Southgate, L. (2002). Relicensure, recertification and practice-based assessment. In G. R. Norman, C. P. M. van der Vleuten, & D. I. Newble DI (Eds.), International handbook of research in medical education (pp. 883-912). Dordrecht: Kluwer Academic Publishers.
10. DeNisi, A. S. (1996). Cognitive approach to performance appraisal: A program of research. New York: Routledge.
11. Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10), S70-S81.
12. Ericsson, K. A. (2006). The Influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 683-704). Cambridge: Cambridge University Press.
13. Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.
14. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
15. Eva, K. W. (2004). What every teacher needs to know about clinical reasoning. Medical Education, 39, 98-106.
16. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66(2), 127-148.
17. Feltovich, P. J., Johnson, P. E., Moller, J. H., & Swanson, D. B. (1984). LCS: The role and development of medical knowledge in diagnostic expertise. In W. J. Clancey & E. H. Shortliffe (Eds.), Readings in medical artificial intelligence: The first decade (pp. 275-319). New York: Addison Wesley.
18. Feltovich, P. J., Prietula, M. J., & Ericsson, K. A. (2006). Studies of expertise from psychological perspectives. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 41-68). Cambridge: Cambridge University Press.
19. Field, A. (2009). Discovering statistics using SPSS. London: Sage Publications Ltd.
20. Gray, J. D. (1996). Global rating scales in residency education. Academic Medicine, 71, S55-S61.
21. Gruppen, L. D., & Frohna, A. Z. (2002). Clinical Reasoning. In G. R. Norman, C. P. M. van der Vleuten, & D. I. Newble DI (Eds.), International handbook of research in medical education (pp. 205-230). Dordrecht: Kluwer Academic Publishers.
22. Hawe, E. (2003). It's pretty difficult to fail: the reluctance of lecturers to award a failing grade. Assessment and Evaluation in Higher Education, 28(4), 371-382.
23. Hobus, P. P., Schmidt, H. G., Boshuizen, H. P., & Patel, V. L. (1987). Contextual factors in the activation of first diagnosis hypotheses: Expert-novice differences. Medical Education, 21, 471-476.
24. Kerrins, J. A., & Cushing, K. S. (2000). Taking a second look: Expert and novice differences when observing the same classroom teaching segment a second time. Journal of Personnel Evaluation in Education, 14(1), 5-24.
25. Klein, G. A. (1993). A recognition primed decision (RPD) model of rapid decision making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision-making in action: Models and methods (pp. 138-147). Norwood, NJ: Ablex.
26. Kreiter, C. D., & Ferguson, K. J. (2001). Examining the generalizability of ratings across clerkships using a clinical evaluation form. Evaluation & The Health Professions, 24, 36-46.
27. Levy, P. E., & Williams, J. R. (2004). The social context of performance appraisal: A review and framework for the future. Journal of Management, 30, 881-905.
28. Lievens, F. (2001). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86(2), 255-264.
29. Mero, N. P., Motowidlo, S. J., & Anna, A. L. (2003). Effects of accountability on rating behavior and rater accuracy. Journal of Applied Social Psychology, 33(12), 2493-2514.
30. Murphy, K. R., & Balzer, W. K. (1986). Systematic distortions in memory-based behavior ratings and performance evaluation: Consequences for rating accuracy. Journal of Applied Psychology, 71, 39-44.
31. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational and goal-based perspectives. Thousand Oaks, CA: Sage Publications.
32. Norcini, J. J. (2005). Current perspectives in assessment: The assessment of performance at work. Medical Education, 39, 880-889.
33. Norman, G. R., Brooks, L. R., & Allen, S. W. (1989). Recall by expert medical practitioners and novices as a record of processing attention. Journal of Experimental Psychology. Learning, Memory, and Cognition, 15, 1166-1174.
34. Norman, G., Eva, K., Brooks, L., & Hamstra, S. (2006). Expertise in medicine and surgery. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 339-354). Cambridge: Cambridge University Press.
35. Ross, K. G., Shafer, J. L., & Klein, G. (2006). Professional judgments and "naturalistic decision making". In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 403-420). Cambridge: Cambridge University Press.
36. Sabers, D. S., Cushing, K. S., & Berliner, D. C. (1991). Differences among teachers in a task characterized by simultaneity, multidimensionality and immediacy. American Educational Research Journal, 28, 63-88.
37. Schmidt, H. G., & Boshuizen, H. P. A. (1993). On the origin of intermediate effects in clinical case recall. Memory and Cognition, 21, 338-351.
38. Silber, C. G., Nasca, T. J., Paskin, D. L., Eiger, G., Robeson, M., & Veloski, J. J. (2004). Do global rating forms enable program directors to assess the ACGME competencies? Academic Medicine, 79, 549-556.
39. van Barneveld, C. (2005). The dependability of medical students' performance ratings as documented on in-training evaluations. Academic Medicine, 80(3), 309-312.
40. van De Wiel, M. W. J., Boshuizen, H. P. A., & Schmidt, H. G. (2000). Knowledge restructuring in expertise development: Evidence from pathophysiological representations of clinical cases by students and physicians. European Journal of Cognitive Psychology, 12(3), 323-356.
41. van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39, 309-317.
42. Voss, J. F., Tyler, S. W., & Yengo, L. A. (1983). Individual differences in the solving of social science problems. In R. Dillon & R. Schmeck (Eds.), Individual differences in cognition (pp. 205-232). New York: Academic Press.
43. Walsh, J. P. (1995). Managerial and organizational cognition: Notes from a trip down to memory lane. Organizational Science, 6(3), 280-321.
44. Williams, R. G., & Dunnington, G. L. (2004). Prognostic value of resident clinical performance ratings. Journal of the American College of Surgeons, 199, 620-627.
45. Williams, R. G., Klamen, D. A., & McCaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270-292.
46. Wimmers, P. F., Schmidt, H. G., Verkoeijen, P. P. J. L., & van de Wiel, M. W. J. (2005). Inducing expertise effects in clinical case recall. Medical Education, 39(9), 949-957.