Endobronchial ultrasound skills and tasks assessment tool: Assessing the validity evidence for a test of endobronchial ultrasound-guided transbronchial needle aspiration operator skill

American Journal of Respiratory and Critical Care Medicine

Volumn 186, Issue 8, 2012, Pages 773-779

  Davoudi, Mohsen ^a   Colt, Henri G ^a   Osann, Kathryn E ^a   Lamb, Carla R ^b   Mullon, John J ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b LAHEY HOSPITAL AND MEDICAL CENTER   (United States)

^c MAYO CLINIC   (United States)

Author keywords

Assessment; Bronchoscopy; Competency; Education; Endobronchial ultrasound guided transbronchial needle aspiration

Indexed keywords

ARTICLE; CLINICAL ASSESSMENT TOOL; CLINICAL COMPETENCE; COMPUTER ASSISTED TOMOGRAPHY; ENDOBRONCHIAL ULTRASONOGRAPHY; ENDOBRONCHIAL ULTRASOUND SKILLS AND TASKS ASSESSMENT TOOL; HUMAN; IMAGE ANALYSIS; IMAGE DISPLAY; INTERRATER RELIABILITY; MEDIASTINUM; NAMED INVENTORIES, QUESTIONNAIRES AND RATING SCALES; PRIORITY JOURNAL; SKILL; STAFF TRAINING; TRANSBRONCHIAL ASPIRATION; VALIDITY;

BENCHMARKING; BRONCHOSCOPY; CLINICAL COMPETENCE; EDUCATIONAL MEASUREMENT; ENDOSCOPIC ULTRASOUND-GUIDED FINE NEEDLE ASPIRATION; HUMANS; MULTIVARIATE ANALYSIS; OBSERVER VARIATION; REGRESSION ANALYSIS; REPRODUCIBILITY OF RESULTS; SELF-ASSESSMENT; UNITED STATES;

EID: 84867591659     PISSN: 1073449X     EISSN: 15354970     Source Type: Journal    
DOI: 10.1164/rccm.201111-1968OC     Document Type: Article

References (49)

1. Colt HG, Davoudi M, Murgu SD. Scientific evidence and principles for the use of endobronchial ultrasound and transbronchial needle aspiration. Expert Rev Med Devices 2011;8:493-513.
2. Pastis NJ, Nietert PJ, Silvestri GA. Variation in training for interventional pulmonary procedures among us pulmonary/critical care fellowships: A survey of fellowship directors. Chest 2005;127:1614-1621. (Pubitemid 46218785)
3. Unroe MA, Shofer SL, Wahidi MM. Training for endobronchial ultrasound: methods for proper training in new bronchoscopic techniques. Curr Opin Pulm Med 2010;16:295-300.
4. Davoudi M, Colt HG. Bronchoscopy simulation: a brief review. Adv Health Sci Educ Theory Pract 2009;14:287-296.
5. Stather DR, Lamb CR, Tremblay A. Simulation in flexible bronchoscopy and endobronchial ultrasound: a review. J Bronchology Interv Pulmonol 2011;18:247-256.
6. Lamb CR, Feller-Kopman D, Ernst A, Simoff MJ, Sterman DH, Wahidi MM, Kovitz KL. An approach to interventional pulmonary fellowship training. Chest 2010;137:195-199.
7. McGaghie WC, Siddall VJ, Mazmanian PE, Myers J. Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians evidence-based educational guidelines. Chest 2009;135:62-68.
8. Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003;37:830-837. (Pubitemid 37083007)
9. Messick S. Validity. In: Linn RL, editor. Educational measurement, 3rd ed. New York, NY: American Council on Education & Macmillan; 1989. pp. 13-104.
10. Downing SM, Haladyna TM. Validity and its threats. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York, NY: Routledge; 2009. pp. 21-55.
11. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 1999.
12. Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004;38:1006-1012. (Pubitemid 39223016)
13. Bronchoscopy International. The Bronchoscopy Education Project (Part II): Endobronchial Ultrasound Bronchoscopy Competency Program; 2011 [accessed 2011 Sept 25]. Available from: http://www.aabronchology.org/PDF/bronch-course/ BEP%20part%20II%20EBUS%20Training%20Manual.pdf
14. Goldberg R, Colt HG, Davoudi M, Cherrison L. Realistic and affordable lo-fidelity model for learning bronchoscopic transbronchial needle aspiration. Surg Endosc 2009;23:2047-2052.
15. Wahidi MM, Silvestri GA, Coakley RD, Ferguson JS, Shepherd RW, Moses L, Conforti J, Que LG, Anstrom KJ, McGuire F, et al. A prospective multi-center study of competency metrics and educational interventions in the learning of bronchoscopy among starting pulmonary fellows. Chest 2010;137:1040-1049.
16. Harris IB, Simpson D. Christine McGuire: at the heart of the maverick measurement maven. Adv Health Sci Educ Theory Pract 2005;10:65-80. (Pubitemid 40875245)
17. Brasel KJ, Bragg D, Simpson DE, Weigelt JA. Meeting the Accreditation Council for Graduate Medical Education competencies using established residency training program assessment tools. Am J Surg 2004;188:9-12. (Pubitemid 38823047)
18. Downing SM. Twelve steps for effective test development. In: Downing SM, Haladyna TM, editors. Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers; 2006. pp. 3-25.
19. Downing SM, Yudkowsky R. Introduction to assessment in the health professions. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York, NY: Routledge; 2009. pp. 1-20.
20. Downing SM. Face validity of assessments: faith-based interpretations or evidence-based science? Med Educ 2006;40:7-8.
21. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006;119:e7-e16.
22. Downing SM, Haladyna TM. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ 2004;38:327-333. (Pubitemid 38375761)
23. Roid GH. Designing ability tests. In: Downing SM, Haladyna TM, editors. Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers; 2006. pp. 527-542.
24. Kane M. Content related validity evidence in test development. In: Downing SM, Haladyna TM, editors. Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers; 2006. pp. 3-25.
25. Yudkowsky R. Performance tests. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York, NY: Routledge; 2009. pp. 217-243.
26. Axelson RD, Kreiter CD. Reliability. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York, NY: Routledge; 2009. pp. 57-73.
27. Stemler SE. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation 9(4). 2004 [accessed 2011 Sept 25]. Available from: http://PAREonline.net/getvn.asp?v=9&n=4
28. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174. (Pubitemid 8054849)
29. Davis DA, Mazmanian PE, Fordis M, Harrison VR, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094-1102. (Pubitemid 44343529)
30. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol 1999;77:1121-1134.
31. Ehrlinger J, Johnson K, Banner M, Dunning D, Kruger J. Why the unskilled are unaware: further explorations of (absent) self-insight among the incompetent. Organ Behav Hum Decis Process 2008;105:98-121.
32. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991;66:762-769.
33. Regehr G, Hoges B, Tiberius R, Lofchy J. Measuring self-assessment skills: an innovative relative ranking model. Acad Med 1996;71:S52-S54. (Pubitemid 26346660)
34. Yudkowsky R, Downing SM, Tekian A. Standard setting. In: Downing SM, Yudkowsky R, editors. Assessment in health professions education. New York, NY: Routledge; 2009. pp. 119-148.
35. Cizek CJ. Standard setting. In: Downing SM, Haladyna TM, editors. Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers; 2006. pp. 225-258.
36. Norcini JJ. Setting standards on educational tests. Med Educ 2003;37:464-469. (Pubitemid 36566846)
37. Kane M. Validating the performance standards associated with passing scores. Rev Educ Res 1994;64:425-461.
38. Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med 2003;15:270-292. (Pubitemid 37414480)
39. Wayne DB, Butter J, Siddall VJ, Fudala MJ, Wade LD, Feinglass J, McGaghie WC. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med 2006;21:251-256.
40. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med 2006;18:50-57. (Pubitemid 43440082)
41. Luecht RM. Designing tests for pass-fail decisions using item response theory. In: Downing SM, Haladyna TM, editors. Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers; 2006. pp. 575-596.
42. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004;79:S70-S81. (Pubitemid 39281222)
43. Zendejas B, Cook DA, Bingener J, Huebner M, Dunn WF, Sarr MG, Farley DR. Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Ann Surg 2011;254:502-511.
44. Kane MT. The role of reliability in criterion-referenced tests. J Educ Meas 1986;23:221-224.
45. Popham WJ, Husek TR. Implications of criterion-referenced measurement. J Educ Meas 1969;6:1-9.
46. Wolcox RR. A note on the length and passing score of a mastery test. J Educ Stat 1976;1:359-364.
47. Grigorenko EL, Sternberg RJ. Dynamic testing. Psychol Bull 1998;124:75-111.
48. Hodges B, Regehr G, McNaugton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med 1999;74:1129-1134. (Pubitemid 29493389)
49. Newble D, Dawson B, Dauphinee D, Page G, Macdonald M, Swanson D, Mulholland H, Thomson A, van der Vleuten C. Guidelines for assessing clinical competence. Teach Learn Med 1994;6:213-220.