Diagnosing Technical Competence in Six Bedside Procedures: Comparing Checklists and a Global Rating Scale in the Assessment of Resident Performance

Academic Medicine

Volumn 90, Issue 8, 2015, Pages 1100-1108

  Walzak, Alison ^a   Bacchus, Maria ^b   Schaefer, Jeffrey P ^b   Zarnke, Kelly ^b   Glow, Jennifer ^c   Brass, Charlene ^c   McLaughlin, Kevin ^b   Ma, Irene W Y ^b  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF CALGARY   (Canada)

^c UNIVERSITY OF CALGARY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; CANADA; CHECKLIST; CLINICAL COMPETENCE; EDUCATION; FEMALE; HOSPITAL INFORMATION SYSTEM; HUMAN; INTERNAL MEDICINE; MALE; MEDICAL EDUCATION; MEDICAL ERROR; PERSONNEL MANAGEMENT; PROCEDURES; REPRODUCIBILITY; STANDARDS; STATISTICS AND NUMERICAL DATA; VIDEORECORDING;

ADULT; ALBERTA; CHECKLIST; CLINICAL COMPETENCE; EDUCATION, MEDICAL, GRADUATE; EDUCATIONAL MEASUREMENT; EMPLOYEE PERFORMANCE APPRAISAL; FEMALE; HUMANS; INTERNAL MEDICINE; INTERNSHIP AND RESIDENCY; MALE; MEDICAL ERRORS; POINT-OF-CARE SYSTEMS; REPRODUCIBILITY OF RESULTS; VIDEOTAPE RECORDING;

EID: 84938223323     PISSN: 10402446     EISSN: 1938808X     Source Type: Journal    
DOI: 10.1097/ACM.0000000000000704     Document Type: Article

References (36)

1. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Internal Medicine. Effective 2009. http://www.acgme. org/acgmeweb/Portals/0/PFAssets/2013-PRFAQ-PIF/140-internal-medicine-07012013. pdf. Accessed January 20, 2015
2. Royal College of Physicians and Surgeons of Canada. Objectives of Training in the Specialty of Internal Medicine. Version 1.0. 2011. http://www.royalcollege.ca/cs/ groups/public/documents/document/y2vk/ mdaw/~edisp/tztest3rcpsced000910.pdf. Accessed January 20, 2015
3. Surgical Council on Resident Education. Curriculum Outline for General Surgery Residency, 2013-2014. http://www.absurgery. org/xfer/curriculumoutline2013-14.pdf. Accessed January 20, 2015
4. Royal College of Anaesthetists. The CCT in anaesthetics. II: Competency based basic level (specialty training (ST) years 1 and 2). In: Training and Assessment: A Manual for Trainees and Trainers. 1st ed. 2007. http:// www.rcoa.ac.uk/system/files/TRG-CU-2007PartII.pdf. Accessed January 20, 2015
5. Evans LV, Dodge KL. Simulation and patient safety: Evaluative checklists for central venous catheter insertion. Qual Saf Health Care. 2010;19(suppl 3):i42-i46
6. McKinley RK, Strand J, Ward L, Gray T, Alun-Jones T, Miller H. Checklists for assessment and certification of clinical procedural skills omit essential competencies: A systematic review. Med Educ. 2008;42:338-349
7. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative bench station examination. Am J Surg. 1997;173:226-230
8. Ma IW, Sharma N, Brindle ME, Caird J, McLaughlin K. Measuring competence in central venous catheterization: A systematicreview. Springerplus. 2014;3:33
9. Glass GV. Standards and criteria. J Educ Meas. 1978;15:237-261
10. Shepard L. Standard setting issues and methods. Appl Psych Meas. 1980;4:447-467
11. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420-1423
12. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 2003;37:1012-1016
13. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73:993-997
14. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999;74:1129-1134
15. Ma IW, Zalunardo N, Pachev G, et al. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract. 2012;17:457-470
16. Ma IW, Chapelsky S, Bhavsar S, et al. Procedural certification program: Enhancing resident procedural teaching skills. Med Teach. 2013;35:524
17. Foundation Programme. FP Curriculum 2012. Direct Observation of Procedural Skills (DOPS). http://www.foundationprogramme. nhs.uk/pages/home/curriculum-andassessment/ curriculum2012. Accessed February 10, 2015
18. Clayton MJ. Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educ Psychol (UK). 1997;17:373-386
19. Angoff W, ed. Scales, Norms, and Equivalent Scores. 2nd ed. Washington, DC: American Council on Education; 1971
20. Ebel RL. Essentials of Educational Measurement. Oxford, England: Prentice-Hall; 1972
21. Faraggi D, Reiser B. Estimation of the area under the ROC curve. Stat Med. 2002;21:3093-3106
22. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New York, NY: Academic Press; 1988
23. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174
24. Cohen DS, Colliver JA, Robbs RS, Swartz MH. A large-scale study of the reliabilities of checklist scores and ratings of interpersonal and communication skills evaluated on a standardized-patient examination. Adv Health Sci Educ Theory Pract. 1996;1:209-213
25. Van der Vleuten CP, Norman GR, De Graaff E. Pitfalls in the pursuit of objectivity: Issues of reliability. Med Educ. 1991;25:110-118
26. Norman GR, Van der Vleuten CP, De Graaff E. Pitfalls in the pursuit of objectivity: Issues of validity, efficiency and acceptability. Med Educ. 1991;25:119-126
27. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273-278
28. Gerard JM, Kessler DO, Braun C, Mehta R, Scalzo AJ, Auerbach M. Validation of global rating scale and checklist instruments for the infant lumbar puncture procedure. Simul Healthc. 2013;8:148-154
29. Regehr G, Freeman R, Hodges B, Russell L. Assessing the generalizability of OSCE measures across content domains. Acad Med. 1999;74:1320-1322
30. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Mastery learning for health professionals using technology-enhanced simulation: A systematic review and metaanalysis. Acad Med. 2013;88:1178-1186
31. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations. Acad Med. 2011;86:e8-e9
32. Cohen ER, Barsuk JH, McGaghie WC, Wayne DB. Raising the bar: Reassessing standards for procedural competence. Teach Learn Med. 2013;25:6-9
33. American Educational Research Association; American Psychological Association; National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association; 2014
34. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993;100:363-406
35. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 suppl):S70-S81
36. Sznajder JI, Zveibil FR, Bitterman H, Weiner P, Bursztein S. Central vein catheterization. Failure and complication rates by three percutaneous approaches. Arch Intern Med. 1986;146:259-261