Determining reliability of clinical assessment scores in real time

Teaching and Learning in Medicine

Volumn 21, Issue 3, 2009, Pages 188-194

  Beckman, Thomas J ^a   Mandrekar, Jayawant N ^a   Engstler, Gregory J ^a   Ficalora, Robert D ^a  

^a Mayo Clinic   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; CLINICAL COMPETENCE; COMPUTER PROGRAM; EDUCATION; FACTORIAL ANALYSIS; FEMALE; HUMAN; INTERNAL MEDICINE; MALE; MEDICAL EDUCATION; REPRODUCIBILITY; STANDARD;

ADULT; CLINICAL COMPETENCE; EDUCATION, MEDICAL, GRADUATE; EDUCATIONAL MEASUREMENT; FACTOR ANALYSIS, STATISTICAL; FEMALE; HUMANS; INTERNAL MEDICINE; INTERNSHIP AND RESIDENCY; MALE; REPRODUCIBILITY OF RESULTS; SOFTWARE;

EID: 70449591712     PISSN: 10401334     EISSN: None     Source Type: Journal    
DOI: 10.1080/10401330903014137     Document Type: Article

References (29)

1. Downing SM. Validity: On the meaningful interpretation of assessment data. Medical Education 2003;37:830-7.
2. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: Theory and application. American Journal of Medicine 2006;119:7-16.
3. Downing SM, Haladyna TM. Validity threats: Overcoming interference with proposed interpretations of assessment data. Medical Education 2004;38:327-33.
4. Beckman TJ, Cook DA, Mandrekar JN. Factor instability of clinical teaching assessment scores among general internists and cardiologists. Medical Education 2006;40:1209-16.
5. Messick S. Validity. In RL Linn (Ed.), Educational measurement (3rd ed.) (pp. 13-103). Phoenix, AZ: Oryx Press, 1993.
6. American Education Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Education Research Association, 1999.
7. Smith GT. On construct validity: Issues of method and measurement. Psychological Assessment 2005;17:396-408.
8. Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychological Assessment 1995;7:309-19.
9. Haynes SN, Richard DC, Kubany ES. Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment 1995;7:238-47.
10. Beckman TJ, Ghosh AK, Cook DA, Erwin PJ, Mandrekar JN. How reliable are assessments of clinical teaching? A review of the published instruments. Journal of General Internal Medicine 2004;19:971-7.
11. Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching? Journal of General Internal Medicine 2005;20:1159-64.
12. Beckman TJ, Lee MC, Rohren CH. Evaluating an instrument for the peer review of inpatient teaching. Medical Teacher 2003;25:131-5.
13. Beckman TJ, Mandrekar JN. The interpersonal, cognitive and efficiency domains of clinical teaching: Construct validity of a multi dimensional scale. Medical Education 2005;39:1221-9.
14. Beckman TJ, Lee MC, Mandrekar JN. A comparison of clinical teaching evaluations by resident and peer physicians. Medical Teacher 2004;26:321-5.
15. Beckman TJ. Lessons learned from a peer review of bedside teaching. Academic Medicine 2004;79:343-6.
16. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;20:37-46.
17. Cicchetti DV, Allison T. A new procedure for assessing reliability of scoring EEG sleep recordings. American Journal of EEG Technology 1971;11:101-9.
18. Fleiss JL, Leven B, Paik MC. Statistical methods for rates and proportions (3rd ed.). Hoboken, NJ: Wiley, 2003.
19. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 1973;33:613-9.
20. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 1969;72:323-7.
21. Landis JR, Koch GG. The measure of observer agreement for categorical data. Biometrics 1977;33:159-74.
22. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297-333.
23. Nunnally JC, Berstein IH. Psychometric theory (3rd ed.). New York: McGraw-Hill, 1994.
24. Kundel HL, Polansky M. Measurement of observer agreement. Radiology 2003;228:303-8.
25. DeVellis RF. Scale development: Theory and applications. London: Sage, 1991.
26. Gorsuch RL. Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum, 1983.
27. Messick S. Validity of psychological assessment. Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 1995;50:741-9.
28. Downing SM. Reliability: On the reproducibility of assessment data. Medical Education 2004;38:1006-12.
29. Copeland HL, Hewson MG. Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Academic Medicine 2000;75:161-6.