Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial

Journal of General Internal Medicine

Volumn 24, Issue 1, 2009, Pages 74-79

  Cook, David A ^a   Dupras, Denise M ^a   Beckman, Thomas J ^a   Thomas, Kris G ^a   Pankratz, V Shane ^a  

^a MAYO CLINIC   (United States)

Author keywords

Assessment; Clinical competence; Faculty development; Medical education; Randomized trial; Rater training

Indexed keywords

ARTICLE; CLINICAL EVALUATION; CONTROLLED STUDY; FOLLOW UP; HALO EFFECT; INTERNAL MEDICINE; INTERRATER RELIABILITY; OUTPATIENT DEPARTMENT; RESIDENT; TRAINING; VIDEORECORDING;

ADULT; CLINICAL COMPETENCE; EDUCATION; EDUCATIONAL MEASUREMENT; FEMALE; FOLLOW-UP STUDIES; HUMANS; INTERNAL MEDICINE; INTERNSHIP AND RESIDENCY; MALE; MIDDLE AGED; PRECEPTORSHIP;

EID: 58149156408     PISSN: 08848734     EISSN: 15251497     Source Type: Journal    
DOI: 10.1007/s11606-008-0842-3     Document Type: Article

References (38)

1. ES Holmboe RE Hawkins SJ Huot 2004 Effects of training in direct observation of medical residents' clinical competence: a randomized trial Ann Intern Med 140 874 881
2. JJ Norcini LL Blank FD Duffy GS Fortna 2003 The mini-CEX: a method for assessing clinical skills Ann Intern Med 138 476 481
3. JR Kogan LM Bellini JA Shea 2003 Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship Acad Med 78 10 Suppl S33 S35
4. ES Holmboe RE Hawkins 1998 Methods for evaluating the clinical competence of residents in internal medicine: a review Ann Intern Med 129 42 48
5. JO Woolliscroft JK Stross J Silva Jr 1984 Clinical competence certification: a critical appraisal J Med Educ 59 799 805
6. FJ Kroboth W Kapoor FH Brown M Karpf GS Levey 1985 A comparative trial of the clinical evaluation exercise Arch Intern Med 145 1121 1123
7. JE Herbers Jr. GL Noel GS Cooper J Harvey LN Pangaro MJ Weaver 1989 How accurate are faculty evaluations of clinical competence J Gen Intern Med. 4 202 208
8. FJ Kroboth BH Hanusa S Parker 1992 The inter-rater reliability and internal consistency of a clinical evaluation exercise J Gen Intern Med 7 174 179
9. GL Noel JE Herbers Jr. MP Caplow GS Cooper LN Pangaro J Harvey 1992 How well do internal medicine faculty members evaluate the clinical skills of residents Ann Intern Med. 117 757 765
10. JJ Norcini LL Blank GK Arnold HR Kimball 1995 The mini-CEX (clinical evaluation exercise): a preliminary investigation Ann Intern Med 123 795 799
11. Schroter S, Plowman R, Hutchings A, Gonzalez A. Reporting of Ethical Committee Approval and Patient Consent by Study Design in 5 General Medical Journals. Paper presented at the Fifth International Congress on Peer Review and Biomedical Publication, Chicago, Illinois, September, 2005.
12. MJ Margolis BE Clauser MM Cuddy 2006 Use of the Mini-Clinical Evaluation Exercise to Rate Examinee Performance on a Multiple-Station Clinical Skills Examination: A Validity Study Acad Med 81 10 Suppl S56 S60
13. R Hatala M Ainslie BO Kassen I Mackie JM Roberts 2006 Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination Med Educ 40 950 956
14. RL Brennan 2001 An essay on the history and future of reliability from the perspective of replications J Educ Meas 38 295 317
15. RG Williams DA Klamen WC McGaghie 2003 Cognitive, Social, and Environmental Sources of Bias in Clinical Performance Ratings Teach Learn Med 15 270 292
16. DI Newble J Hoare PF Sheldrake 1980 The selection and training of examiners for clinical examinations Med Educ 14 345 349
17. ES Holmboe S Huot J Chung J Norcini RE Hawkins 2003 Construct validity of the miniclinical evaluation exercise (miniCEX) Acad Med 78 826 830
18. MJ Müller W Rossbach P Dannigkeit F Müller-Siecheneder A Szegedi H Wetzel 1998 Evaluation of standardized rater training for the Positive and Negative Syndrome Scale (PANSS) Schizophr Res 32 151 160
19. MJ Müller A Dragicevic 2003 Standardized rater training for the Hamilton Depression Rating Scale (HAMD-17) in psychiatric novices J Affect Disord 77 65 69
20. AC Angkaw GQ Tran DAF Haaga 2006 Effects of training intensity on observers' rating of anxiety, social skills, and alcohol-specific coping skills Behav Res Ther 44 533 544
21. DJ Woehr AI Huffcutt 1994 Rater training for performance appraisal: A quantitative review J Occup Organ Psychol 67 3 189 205
22. Cook DA, Beckman TJ. Psychometric properties of mini-clinical evaluation exercise (mini-CEX) scores: Accuracy, reliability, and effect of scale length. Paper presented at the 2008 meeting of the American Educational Research Association, New York, March, 2008.
23. Casella G, Berger RL. Statistical Inference. 2New York: Duxbury Press; 2001.
24. SD Walter M Eliasziw A Donner 1998 Sample Size and Optimal Study Designs for Reliability Studies Stat Med 17 101 110
25. SL Zeger K-Y Liang 1986 Longitudinal Data Analysis for Discrete and Continuous Outcomes Biometrics 42 121 130
26. JL Fleiss J Cohen 1973 The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability Educ Psychol Meas 33 613 619
27. R Jacobs SW Kozlowski 1985 A closer look at halo error in performance ratings Acad Manage J 28 201 212
28. LM Harvill 1991 NCME Instructional Module: Standard Error of Measurement Educ Meas: Issues Pract 10 2 33 41
29. Brennan RL. Generalizability Theory. New York: Springer-Verlag; 2001.
30. G Norman 2005 Research in clinical reasoning: past history and current trends Med Educ 39 418 427
31. KR Murphy JN Cleveland AL Skattebo TB Kinney 2004 Raters who pursue different goals give different ratings J Appl Psychol 89 158 164
32. FJ Kroboth BH Hanusa SC Parker 1996 Didactic value of the clinical evaluation exercise. Missed opportunities J Gen Intern Med. 11 551 553
33. M Srinivasan KE Hauer C Der-Martirosian M Wilkes N Gesundheit 2007 Does feedback matter? Practice-based learning for medical students after a multi-institutional clinical performance examination Med Educ. 41 857 865
34. N Fernando J Cleland H McKenzie K Cassar 2008 Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments Med Educ. 42 89 95
35. E Holmboe N Fiebach L Galaty S Huot 2001 Effectiveness of a focused educational intervention on resident evaluations from faculty J Gen Intern Med 16 427 434
36. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2Hillsdale, NJ: Lawrence Erlbaum; 1988.
37. JR Landis GG Koch 1977 The measurement of observer agreement for categorical data Biometrics 33 159 174
38. KA Kobak N Engelhardt JD Lipsitz 2006 Enriched rater training using Internet based technologies: a comparison to traditional rater training in a multi-site depression trial J Psychiatr Res 40 192 199