A contemporary approach to validity arguments: A practical guide to Kane's framework

Medical Education

Volumn 49, Issue 6, 2015, Pages 560-575

  Cook, David A ^a   Brydges, Ryan ^b,c   Ginsburg, Shiphra ^b,c   Hatala, Rose ^d  

^a MAYO CLINIC   (United States)

^b UNIVERSITY OF TORONTO   (Canada)

^c UNIVERSITY HEALTH NETWORK   (Canada)

^d UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL COMPETENCE; EDUCATION; HUMAN; MEDICAL EDUCATION; PSYCHOMETRY; REPRODUCIBILITY; STANDARDS; VALIDATION STUDY;

CLINICAL COMPETENCE; EDUCATION, MEDICAL; EDUCATIONAL MEASUREMENT; HUMANS; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; VALIDATION STUDIES AS TOPIC;

EID: 84929398562     PISSN: 03080110     EISSN: 13652923     Source Type: Journal    
DOI: 10.1111/medu.12678     Document Type: Article

References (80)

1. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach 2010;32:676-82.
2. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Acad Med 2013;88:1178-86.
3. Caverzagie KJ, Iobst WF, Aagaard EM et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med 2013;158:557-9.
4. Green ML, Aagaard EM, Caverzagie KJ, Chick DA, Holmboe E, Kane G, Smith CD, Iobst W. Charting the road to competence: developmental milestones for internal medicine residency training. J Grad Med Educ 2009;1:5-20.
5. ten Cate O. Trust, competence, and the supervisor's role in postgraduate training. BMJ 2006;333:748-51.
6. Messick S. Validity. In: Linn RL, ed. Educational Measurement, 3rd edn. New York, NY: American Council on Education and Macmillan 1989;13-103.
7. Kane MT. Validation. In: Brennan RL, ed. Educational Measurement, 4th edn. Westport, CT: Praeger 2006;17-64.
8. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychol Bull 1955;52:281-302.
9. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: AERA 2014.
10. Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003;37:830-7.
11. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006;19:166.e7-16.
12. Kane MT. Validating the interpretations and uses of test scores. J Educ Meas 2013;50:1-73.
13. Hodges B. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach 2013;35:564-8.
14. Schuwirth LWT, van der Vleuten CPM. A plea for new psychometric models in educational assessment. Med Educ 2006;40:296-300.
15. Schuwirth LWT, van der Vleuten CPM. Programmatic assessment and Kane's validity perspective. Med Educ 2012;46:38-48.
16. Chan DW, Bruzek DJ, Oesterling JE, Rock RC, Walsh PC. Prostate-specific antigen as a marker for prostatic cancer: a monoclonal and a polyclonal immunoassay compared. Clin Chem 1987;33:1916-20.
17. Carter HB, Pearson JD, Metter EJ, Brant LJ, Chan DW, Andres R, Fozard JL, Walsh PC. Longitudinal evaluation of prostate-specific antigen levels in men with and without prostate disease. JAMA 1992;267:2215-20.
18. Stamey TA, Yang N, Hay AR, McNeal JE, Freiha FS, Redwine E. Prostate-specific antigen as a serum marker for adenocarcinoma of the prostate. N Engl J Med 1987;317:909-16.
19. Schroder FH, Hugosson J, Roobol MJ et al. Prostate-cancer mortality at 11 years of follow-up. N Engl J Med 2012;366:981-90.
20. Andriole GL, Crawford ED, Grubb RL III et al. Mortality results from a randomised prostate-cancer screening trial. N Engl J Med 2009;360:1310-9.
21. Moyer VA. Screening for prostate cancer: US Preventive Services Task Force recommendation statement. Ann Intern Med 2012;157:120-34.
22. Qaseem A, Barry MJ, Denberg TD, Owens DK, Shekelle P. Screening for prostate cancer: a guidance statement from the Clinical Guidelines Committee of the American College of Physicians. Ann Intern Med 2013;158:761-9.
23. Carter HB, Albertsen PC, Barry MJ et al. Early detection of prostate cancer: AUA guideline. J Urol 2013;190:419-26.
24. Rosario DJ, Lane JA, Metcalfe C et al. Short term outcomes of prostate biopsy in men tested for cancer by prostate specific antigen: prospective evaluation within ProtecT study. BMJ 2012;344:d7894.
25. Schroder FH, Hugosson J, Roobol MJ et al. Screening and prostate-cancer mortality in a randomised European study. N Engl J Med 2009;360:1320-8.
26. Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VE. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomised, controlled trial. J Gen Intern Med 2009;24:74-9.
27. Roediger HL, Karpicke JD. The power of testing memory: basic research and implications for educational practice. Perspect Psychol Sci 2006;1:181-210.
28. Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz VS. Impact of self-assessment questions and learning styles in web-based learning: a randomised, controlled, crossover trial. Acad Med 2006;81:231-8.
29. Cook DA, Thompson WG, Thomas KG. Test-enhanced web-based learning: optimising the number of questions (a randomised crossover trial). Acad Med 2014;89:169-75.
30. Driessen EW, Overeem K, van Tartwijk J, van der Vleuten CPM, Muijtjens AMM. Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006;40:862-6.
31. Kuper A, Reeves S, Albert M, Hodges BD. Assessment: do we need to broaden our methodological horizons? Med Educ 2007;41:1121-3.
32. Larsen DP, Butler AC, Roediger HL III. Test-enhanced learning in medical education. Med Educ 2008;42:959-66.
33. Cook DA. When I say... validity. Med Educ 2014;48:948-9.
34. Brennan RL. Commentary on 'Validating the interpretations and uses of test scores'. J Educ Meas 2013;50:74-83.
35. Ilgen JS, Ma IW, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ 2015;49:161-73.
36. Cook DA, Beckman TJ. Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract 2009;14:655-64.
37. Sandilands DD, Gotzmann A, Roy M, Zumbo BD, De Champlain A. Weighting checklist items and station components on a large-scale OSCE: is it worth the effort? Med Teach 2014;36:585-90.
38. Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgements: rethinking the aetiology of rater errors. Acad Med 2011;86 (10 Suppl):1-7.
39. Yeates P, O'Neill P, Mann K, Eva K. Seeing the same thing differently: mechanisms that contribute to assessor differences in directly observed performance assessments. Adv Health Sci Educ Theory Pract 2013;18:325-41.
40. Cook DA. Much ado about differences: why expert-novice comparisons add little to the validity argument. Adv Health Sci Educ Theory Pract DOI: 10.1007/s10459-014-9551-3 [Epub ahead of print 2014 Sep 27].
41. Cronbach LJ. Five perspectives on validity argument. In: Wainer H, Braun HI, eds. Test Validity. Hillsdale, NJ: Routledge 1988;3-17.
42. Haertel E. Getting the help we need. J Educ Meas 2013;50:84-90.
43. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med 2013;88:872-83.
44. Clauser BE, Margolis MJ, Holtman MC, Katsufrakis PJ, Hawkins RE. Validity considerations in the assessment of professionalism. Adv Health Sci Educ Theory Pract 2012;17:165-81.
45. Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ. Constructing a validity argument for the mini-clinical evaluation exercise: a review of the research. Acad Med 2010;85:1453-61.
46. Oesterling JE. Prostate specific antigen: a critical assessment of the most useful tumour marker for adenocarcinoma of the prostate. J Urol 1991;145:907-23.
47. Oesterling JE, Jacobsen SJ, Chute CG, Guess HA, Girman CJ, Panser LA, Lieber MM. Serum prostate-specific antigen in a community-based population of healthy men. Establishment of age-specific reference ranges. JAMA 1993;270:860-4.
48. Vashi AR, Oesterling JE. Percent free prostate-specific antigen: entering a new era in the detection of prostate cancer. Mayo Clin Proc 1997;72:337-44.
49. Schroder FH, Roobol MJ. Defining the optimal prostate-specific antigen threshold for the diagnosis of prostate cancer. Curr Opin Urol 2009;19:227-31.
50. Ross KS, Carter HB, Pearson JD, Guess HA. Comparative efficiency of prostate-specific antigen screening strategies for prostate cancer detection. JAMA 2000;284:1399-405.
51. Barry MJ. Screening for prostate cancer - the controversy that refuses to die. N Engl J Med 2009;360:1351-4.
52. Wilt TJ, Brawer MK, Jones KM et al. Radical prostatectomy versus observation for localised prostate cancer. N Engl J Med 2012;367:203-13.
53. Hayes JH, Barry MJ. Screening for prostate cancer with the prostate-specific antigen test: a review of current evidence. JAMA 2014;311:1143-9.
54. Hamstra SJ, Brydges R, Hatala R, Zendejas B, Cook DA. Reconsidering fidelity in simulation-based training. Acad Med 2014;89:387-92.
55. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273-8.
56. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 1998;73:993-7.
57. Friedlich M, MacRae H, Oandasan I, Tannenbaum D, Batty H, Reznick R, Regehr G. Structured assessment of minor surgical skills (SAMSS) for family medicine residents. Acad Med 2001;76:1241-6.
58. Hance J, Aggarwal R, Stanbridge R, Blauth C, Munz Y, Darzi A, Pepper J. Objective assessment of technical skills in cardiac surgery. Eur J Cardiothorac Surg 2005;28:157-62.
59. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative 'bench station' examination. Am J Surg 1997;173:226-30.
60. Goff BA, Lentz GM, Lee D, Fenner D, Morris J, Mandel LS. Development of a bench station objective structured assessment of technical skills. Obstet Gynecol 2001;8:412-6.
61. Datta V, Bann S, Beard J, Mandalia M, Darzi A. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 2004;199:603-6.
62. Bann S, Davis IM, Moorthy K, Munz Y, Hernandez J, Khan M, Datta V, Darzi A. The reliability of multiple objective measures of surgery and the role of human performance. Am J Surg 2005;189:747-52.
63. Hatala R, Cook DA, Brydges R, Hawkins R. Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence. Adv Health Sci Educ Theory Pract. [Epub ahead of print 2015 Feb 22].
64. Ginsburg S, Gold W, Cavalcanti RB, Kurabi B, McDonald-Blumer H. Competencies 'plus': the nature of written comments on internal medicine residents' evaluation forms. Acad Med 2011;86 (10 Suppl):30-4.
65. Ginsburg S, McIlroy J, Oulanova O, Eva K, Regehr G. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med 2010;85:780-6.
66. Watling CJ, Kenyon CF, Schulz V, Goldszmidt MA, Zibrowski E, Lingard L. An exploration of faculty perspectives on the in-training evaluation of residents. Acad Med 2010;85:1157-62.
67. Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med 2005;80 (Suppl):84-7.
68. Dudek NL, Marks MB, Wood TJ, Dojeiji S, Bandiera G, Hatala R, Cooke L, Sadownik L. Quality evaluation reports: can a faculty development programme make a difference? Med Teach 2012;34:e725-31.
69. Vivekananda-Schmidt P, MacKillop L, Crossley J, Wade W. Do assessor comments on a multi-source feedback instrument provide learner-centred feedback? Med Educ 2013;47:1080-8.
70. Watling CJ, Kenyon CF, Zibrowski EM, Schulz V, Goldszmidt MA, Singh I, Maddocks HL, Lingard L. Rules of engagement: residents' perceptions of the in-training evaluation process. Acad Med 2008;83 (Suppl):97-100.
71. Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med 2013;88:1539-44.
72. Regehr G, Ginsburg S, Herold J, Hatala R, Eva K, Oulanova O. Using 'standardised narratives' to explore new ways to represent faculty opinions of resident performance. Acad Med 2012;87:419-27.
73. Guerrasio J, Cumbler E, Trosterman A, Wald H, Brandenburg S, Aagaard E. Determining need for remediation through postrotation evaluations. J Grad Med Educ 2012;4:47-51.
74. Cohen GS, Blumberg P, Ryan NC, Sullivan PL. Do final grades reflect written qualitative evaluations of student performance? Teach Learn Med 1993;5:10-5.
75. Richards SH, Campbell JL, Walshaw E, Dickens A, Greco M. A multi-method analysis of free-text comments from the UK General Medical Council Colleague Questionnaires. Med Educ 2009;43:757-66.
76. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings' post-rotation performance ratings detect residents' clinical performance deficiencies? Acad Med 2004;79:453-7.
77. Guerrasio J, Garrity MJ, Aagaard EM. Learner deficits and academic outcomes of medical students, residents, fellows, and attending physicians referred to a remediation programme, 2006-2012. Acad Med 2014;89:352-8.
78. Newton PE. Two kinds of argument? J Educ Meas 2013;50:105-9.
79. Kane MT. Validation as a pragmatic, scientific activity. J Educ Meas 2013;50:115-22.
80. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 1959;56:81-105.