Consistency, inter-rater reliability, and validity of 441 consecutive mock oral examinations in anesthesiology: Implications for use as a tool for assessment of residents

Anesthesiology

Volumn 91, Issue 1, 1999, Pages 288-298

  Schubert, Armin ^a,c   Tetzlaff, John E ^a   Tan, Ming ^b   Ryckman, J Victor ^d   Mascha, Edward ^d  

^a OHIO STATE UNIVERSITY   (United States)

^b ST JUDE CHILDREN S RESEARCH HOSPITAL   (United States)

^c LERNER RESEARCH INSTITUTE   (United States)

^d NONE

Author keywords

Clinical judgment; Competence; Education; Residency; Testing

Indexed keywords

ANESTHESIST; ARTICLE; CLINICAL EXAMINATION; CLINICAL PRACTICE; COMPETENCE; EDUCATION PROGRAM; HUMAN; MEDICAL DECISION MAKING; ORAL EXAMINATION; PRIORITY JOURNAL; RELIABILITY; RESIDENCY EDUCATION;

ANESTHESIOLOGY; CERTIFICATION; CLINICAL COMPETENCE; HUMANS; INTERNSHIP AND RESIDENCY; PROSPECTIVE STUDIES;

EID: 0033060441     PISSN: 00033022     EISSN: None     Source Type: Journal    
DOI: 10.1097/00000542-199907000-00037     Document Type: Article

References (43)

1. Eagle CJ, Martineau R, Hamilton K: The oral examination in anesthetic resident evaluation. Can J Anaesth 1993; 40:947-53
2. Siker ES: A measure of competence. Anaesthesia 1976; 31:732-42
3. Colliver JA, Verhulst SJ, Williams RG, Norcini JJ: Reliability of performance on standardized patient cases: A comparison of consistency measures based on generalizability theory. Teaching Learning Med 1989; 1:31-7
4. Pope WD: Anaesthesia oral examination (editorial). Can J Anaesth 1993; 40:907-10
5. Carter HD: How reliable are good oral examinations? California J Educ Res 1962; 13:147-53
6. Kelley PR, Matthews KH, Schumacher CF: Analysis of the oral examination of the American Board of Anesthesiology. J Med Educ 1971; 46:982-8
7. Slogoff S, Hughes FP, Hug CC, Longnecker DE, Saidman LJ: A demonstration of validity for certification by the American Board of Anesthesiology. Acad Med 1994; 69:740-6
8. Schubert A, Tetzfall J, Licina M, Mascha E, Smith MP: Organization of a comprehensive anesthesiology oral practice examination program. http://www.jepm.org/electroniclibrary/pre-acceptancepeerreview
9. Hubbard JP: The oral exam, Measuring Medical Education. Philadelphia, Lea & Febiger, 1971, pp 93-9
10. Nunnally JC: Assessment of reliability, Psychometric Theory, 2nd Edition. New York, McGraw-Hill, 1978, pp 245-6
11. Liang KY, Zeger SL: Longitudinal data analysis using generalized liner models. Biometrika 1986; 73:13-22
12. Bland JM, Altman DG: Statistical methods for assessing agreemnent between two methods of clinical measurement. Lancet 1986; 2:307-10
13. Tan M, Qu Y, Kutner MH: Model diagnostics for marginal regression analysis of correlated binary data. Communications Stat B 1997; 26:539-58
14. Jennrich RI, Schluchtar MD: Unbalanced repeated-measures models with structured covariance matrices. Biometrics 1986; 42: 805-20
15. Stillman P, Swanson D, Regan MB, Philbin MM, Newlson V, Ebert T, Ley B, Parrino T, Shorey J, Stillman A: Assessment of clinical skills of residents utilizing standardized patients. Ann Intern Med 1991; 114:393-401
16. Anastakis DJ, Cohen R, Reznick RK: The structured oral examination as a method for assessing surgical residents. Am J Surg 1991; 162:67-70
17. Muzzin LJ, Hart L: Oral examinations, Assessing Clinical Competence. Edited by Neufeld VR, Norman GR. New York, Springer Publishing, 1985, pp 71-93
18. Burchard KW, Rowland-Morin PA, Coe NP, Garb JL: A surgery oral examination: Interrater agreement and the influence of rater characteristics. Acad Med 1995; 70:1044-6
19. McGuire CH: Studies of the oral examination: Experiences with orthopedic surgery, Evaluating the Skills of Medical Specialists. Edited by Lloyd JS, Langley DG. Chicago, American Board of Medical Specialists, 1983, pp 105-9
20. Yang JC, Laube DW: Improvement of reliability of an oral examination by a structured evaluation instrument. J Med Educ 1983; 58:864 -72
21. Evans LR, Ingersoll RW, Smith EJ: The reliability, validity and taxonomic structure of the oral examination. J Med Educ 1966; 41: 651-7
22. Maatsch JL, Huang R: An evaluation of construct validity of four alternative theories of clinical competence. Proc Annu Conf Res Med Educ 1986; 25:69-74
23. McDermott JF Jr, Tanguay PE, Scheiber SC, Juul D, Shore JH, Tucker GJ, McCurdy L, Terr LL: Reliability of the Part II Board Certification Examination in Psychiatry: Interexaminer consistency. Am J Psychiatr 1991; 148:1672-4
24. Colliver JA: Validation of standardized patient assessment: A meaning for clinical competence. Acad Med 1995; 70:1062-4
25. Quattlebaum TG, Darden PM, Sperry JB: In-training examinations as predictors of resident clinical performance. Pediatrics 1989; 84:165-72
26. Keynan A, Friedman M, Benbassat J: Reliability of global rating scales in the assessment of clinical competence of medical students. Med Educ 1987; 21:477-81
27. Zelenock GB, Calhoun JG, Hockman EM, Youmans LC, Erlandson EE, Davis WK, Turcotte JG: Oral examinations: Actual and perceived contributions to surgery clerkship performance. Surgery 1985; 97:737-44
28. Pokorny AD, Frazier SH: An evaluation of oral examinations. J Med Educ 1966; 41:28-40
29. Cohen R, Rothman AI, Poldre P, Ross J: Validity and generalizability of global ratings in an objective structured clinical examination. Acad Med 1991; 66:545-8
30. Wade TP, Andrus CH, Kaminski DL: Evaluations of surgery resident performance correlate with success in board examinations. Surgery 1993; 113:644-8
31. Schwartz RW, Donnelly MB, Sloan DA, Johnson SB, Strodel WE: Assessing senior residents' knowledge and performance: An integrated evaluation program. Surgery 1994; 116:634-7
32. Rhoton MF: A new method to evaluate clinical performance and critical incidents in anesthesia: Quantification of daily comments by teachers. Med Educ 1990: 24:280-9
33. Anderson DC, Harris IB, Allen S, Satran L, Bland CJ, Davis-Feickert IA, Poland GA, Miller WJ: Comparing students' feedback about clinical instruction with their performances. Acad Med 1991; 66:29-34
34. Rowland-Morin PA, Burchard KW, Garb JL, Coe NP: Influence of effective communication by surgery students on their oral examination scores. Acad Med 1991; 66:169-71
35. Evans LR, Ingersoll RW, Smith EJ: The reliability, validity, and taxonomic structure of the oral examination. J Med Educ 1966; 41: 651-7
36. Rutala PJ, Witzke DB, Leko EO, Fulginiti JV: The influences of student and standardized patient genders on scoring in an objective structured clinical examination. Acad Med 1991; 66(suppl 9):S28-30
37. Stillman RM: Effect of prior clinical experience on students' knowledge and performance in surgery. Surgery 1986; 100:77-82
38. Hoff JT, Eisenberg HM: Assessment of training progress and examinations. Acta Neurochir Suppl (Wien) 1997; 69:83-8
39. Friedenberg RM: Qualifying examinations: Are they a measure of competence? Radiology 1995; 194:45A-7A
40. Seidel HM: The role of National Board examinations in medical education. Pharos 1992; 55:12-4
41. Elstein AS: Beyond multiple-choice questions and essays: The need for a new way to assess clinical competence. Acad Med 1993; 68:244-9
42. Stawski WS: Evolution of a mock oral examination program in surgery. Am Surg 1994; 60:603-5
43. Maatsch JL, Huang RR, Barker D, Munger B: The predictive validity of test formats and a psychometric theory of clinical competence. Proc Annu Conf Res Med Educ 1984; 23:76-83