Can student self-ratings be compared with peer ratings? A study of measurement invariance of multisource feedback

Advances in Health Sciences Education

Volumn 21, Issue 2, 2016, Pages 401-413

  Lee, Keng Lin ^a   Tsai, Shih Li ^a   Chiu, Yu Ting ^a   Ho, Ming Jung ^a  

^a NATIONAL TAIWAN UNIVERSITY   (Taiwan)

Author keywords

Measurement invariance; Multisource feedback; Peer assessment; Professionalism; Self assessment

Indexed keywords

CONSTRUCTIVE FEEDBACK; CONTROLLED STUDY; EDUCATION; FEMALE; HUMAN; MALE; MEDICAL STUDENT; PEER GROUP; PROCEDURES; PSYCHOMETRY; QUESTIONNAIRE; RANDOMIZED CONTROLLED TRIAL; REPRODUCIBILITY; SELF EVALUATION; STANDARDS; TAIWAN;

EDUCATIONAL MEASUREMENT; FEMALE; FORMATIVE FEEDBACK; HUMANS; MALE; PEER GROUP; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; SELF-ASSESSMENT; STUDENTS, MEDICAL; SURVEYS AND QUESTIONNAIRES; TAIWAN;

EID: 84942068042     PISSN: 13824996     EISSN: 15731677     Source Type: Journal    
DOI: 10.1007/s10459-015-9638-5     Document Type: Article

References (63)

1. Abbasi, K. (2011). A way forward for whistleblowing. Journal of the Royal Society of Medicine, 104, 275.
2. Al Ansari, A., Donnon, T., Al Khalifa, K., Darwish, A., & Violato, C. (2014). The construct and criterion validity of the multi-source feedback process to assess physician performance: A meta-analysis. Advances in Medical Education and Practice, 5, 39–51.
3. Al Khalifa, K., Al Ansari, A., Violato, C., & Donnon, T. (2013). Multisource feedback to assess surgical practice: A systematic review. Journal of Surgical Education, 70(4), 475–486.
4. Allerup, P., Aspegren, K., Ejlersen, E., Jørgensen, G., Malchow-Møller, A., Møller, et al. (2007). Use of 360-degree assessment of residents in internal medicine in a Danish setting: A feasibility study. Medical Teacher, 29(2–3), 166–170.
5. Andrews, J. J. W., Violato, C., Al Ansari, A., Donnon, T., & Pugliese, G. (2013). Assessing psychologists in practice: Lessons from the health professions using multisource feedback. Professional Psychology: Research and Practice, 44(4), 193–207.
6. Archer, J. C., Norcini, J., & Davies, H. A. (2005). Use of SPRAT for peer review of paediatricians in training. British Medical Journal, 330(7502), 1251–1253.
7. Arnold, E. L., Blank, L. L., Race, K. E. H., & Cipparrone, N. (1998). Can professionalism be measured? The development of a scale for use in the medical environment. Academic Medicine, 73(10), 1119–1121.
8. Bolsin, S., Pal, R., Wilmshurst, P., & Pena, M. (2011). Whistleblowing and patient safety: The patient’s or the profession’s interests at stake? Journal of the Royal Society of Medicine, 104(7), 278–282.
9. Campbell, J. L., Roberts, M., Wright, C., Hill, J., Greco, M., Taylor, M., et al. (2011). Factors associated with variability in the assessment of UK doctors’ professionalism: Analysis of survey results. British Medical Journal, 343, d6212.
10. Chandler, N., Henderson, G., Park, B., Byerley, J., Brown, W. D., & Steiner, M. J. (2010). Use of a 360-degree evaluation in the outpatient setting: The usefulness of nurse, faculty, patient/family, and resident self-evaluation. Journal of Graduate Medical Education, 2(3), 430–434.
11. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464–504.
12. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255.
13. Cohen, J. J. (2006). Professionalism in medical education, an American perspective: From evidence to accountability. Medical Education, 40(7), 607–617.
14. Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. American Journal of Medicine, 119(2), 166.e7–166.e16.
15. Davies, H., Archer, J., Bateman, A., Dewar, S., Crossley, J., Grant, J., et al. (2008). Specialty-specific multi-source feedback: Assuring validity, informing training. Medical Education, 42(10), 1014–1020.
16. Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. Journal of the American Medical Association, 296(9), 1094–1102.
17. Deptula, P., & Chun, M. B. (2013). A literature review of professionalism in surgical education: Suggested components for development of a curriculum. Journal of Surgical Education, 70(3), 408–422.
18. Donnon, T., Al Ansari, A., Al Alawi, S., & Violato, C. (2014). The reliability, validity, and feasibility of multisource feedback physician assessment: A systematic review. Academic Medicine, 89(3), 511–516.
19. Donnon, T., McIlwrick, J., & Woloschuk, W. (2013). Investigating the reliability and validity of self and peer assessment to measure medical students’ professional competencies. Creative Education, 4(6A), 23–28.
20. Evans, A. W., McKenna, C., & Oliver, M. (2002). Self-assessment in medical practice. Journal of the Royal Society of Medicine, 96(10), 511–513.
21. Ginsburg, S., Regehr, G., Hatala, R., McNaughton, N., Frohna, A., Hodges, B., et al. (2000). Context, conflict, and resolution: A new conceptual framework for evaluating professionalism. Academic Medicine, 75(Suppl 10), S6–S11.
22. Ginsburg, S., Regehr, G., & Lingard, L. (2004). Basing the evaluation of professionalism on observable behaviors: A cautionary tale. Academic Medicine, 79(Suppl 10), S1–S4.
23. Gornall, J. (2009). The price of silence. British Medical Journal, 339, 1000–1004.
24. Ho, M. J., Lin, C. W., Chiu, Y. T., Lingard, L., & Ginsburg, S. (2012). A cross-cultural study of students’ approaches to professional dilemmas: Sticks or ripples. Medical Education, 46(3), 245–256.
25. Ho, M. J., Yu, K. H., Hirsh, D., Huang, T. S., & Yang, P. C. (2011). Does one size fit all? Building a framework for medical professionalism. Academic Medicine, 86(11), 1407–1414.
26. Hu, L. T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453.
27. Hu, L., & Bentler, P. M. (1999). Cut off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.
28. Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70, 631–639.
29. Jöreskog, K. G., & Sörbom, D. (2004). LISREL 8.7 for Windows. Lincolnwood, IL: Scientific Software International, Inc.
30. Legge, J. (translator). (1861). The Chinese classic, etc.: Vol I. Confucian analects, the great learning, and the doctrine of the mean. Hong Kong: Selbstverl.
31. Leung, K. K., Wang, W. D., & Chen, Y. Y. (2012). Multi-source evaluation of interpersonal and communication skills of family medicine residents. Advances in Health Sciences Education, 17, 717–726.
32. Lockyer, J. (2003). Multisource feedback in the assessment of physician competencies. Journal of Continuing Education in the Health Professions, 23(1), 4–12.
33. Lockyer, J. M., & Clyman, S. G. (2008). Multisource feedback (360 degree evaluation). In E. S. Holmboe & R. E. Hawkins (Eds.), Practical guide to the evaluation of clinical competence (1st ed., pp. 75–85). Philadelphia, PA: Mosby-Elsevier.
34. Lockyer, J. M., Violato, C., & Fidler, H. (2006a). The assessment of emergency physicians by a regulatory authority. Academic Emergency Medicine, 13(12), 1296–1303.
35. Lockyer, J. M., Violato, C., & Fidler, H. (2006b). A multi source feedback program for anesthesiologists. Canadian Journal of Anesthesia, 53(1), 33–39.
36. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543.
37. Meredith, W., & Teresi, J. A. (2006). An essay on measurement and factorial invariance. Medical Care, 44, S69–S77.
38. Miller, A., & Archer, J. (2010). Impact of workplace based assessment on doctors’ education and performance: A systematic review. British Medical Journal, 341, c5064.
39. Motl, R. W., Dishman, R. K., Birnbaum, A. S., & Lytle, L. A. (2005). Longitudinal invariance of the Center for Epidemiological Studies Depression Scale (CES-D) among girls and boys in middle school. Educational and Psychological Measurement, 65, 90–108.
40. Musick, D. W., McDowell, S. M., Clark, N., & Salcido, R. (2003). Pilot study of a 360-degree assessment instrument for physical medicine & rehabilitation residency programs. American Journal of Physical Medicine and Rehabilitation, 82(5), 394–402.
41. National Board of Medical Examiners. (2006). Assessment of professional behaviors. Retrieved 2010. https://www2.nbme.org/APB/Schools/APB/join.asp.
42. Overeem, K., Wollersheim, H. C., Arah, O. A., Cruijsberg, J. K., Grol, R. P. T. M., & Lombarts, K. M. J. M. H. (2012). Evaluation of physicians’ professional performance: An iterative development and validation study of multisource feedback instruments. BMC Health Services Research, 12, 80.
43. Qu, B., Zhao, Y. H., & Sun, B. Z. (2012). Assessment of resident physicians in professionalism, interpersonal and communication skills: A multisource feedback. International Journal of Medical Sciences, 9(3), 228–236.
44. Raee, H., Amini, M., Nasab, A. M., Pour, A. M., & Jafari, M. M. (2014). Team-based assessment of professional behavior in medical students. Journal of Advances in Medical Education and Professionalism, 2(3), 126–130.
45. Reise, S. P., Widaman, K. F., & Pugh, R. H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114, 552–566.
46. Rhodes, R., & Strain, J. J. (2004). Whistleblowing in academic medicine. Journal of Medical Ethics, 30, 35–39.
47. Richmond, M., Canavan, C., Holtman, M. C., & Katsufrakis, P. J. (2011). Feasibility of implementing a standardized multisource feedback program in the graduate medical education environment. Journal of Graduate Medical Education, 3(4), 511–516.
48. Roberts, M. J., Campbell, J. L., Richards, S. H., & Wright, C. (2013). Self-other agreement in multisource feedback: The influence of doctor and rater group characteristics. Journal of Continuing Education in the Health Professions, 33(1), 14–23.
49. Sargeant, J. M., Mann, K. V., Ferrier, S. N., Langille, D. B., Muirhead, P. D., Hayes, V. M., & Sinclair, D. E. (2003). Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: A pilot study. Academic Medicine, 78(Suppl 10), S42–S44.
50. Sargeant, J., Mann, K., Sinclair, D., van der Vleuten, C., & Metsemakers, J. (2007). Challenges in multisource feedback: Intended and unintended outcomes. Medical Education, 41(6), 583–591.
51. Sargeant, J., Mann, K., van der Vleuten, C., & Metsemakers, J. (2008). “Directed” self-assessment: Practice and feedback within a social context. The Journal of Continuing Education in the Health Professions, 28(1), 47–54.
52. Shrank, W. H., Reed, V. A., & Jernstedt, G. C. (2004). Fostering professionalism in medical education: A call for improved assessment and meaningful incentives. Journal of General Internal Medicine, 19(8), 887–892.
53. Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69–75.
54. Tromp, F., Vernooij-Dassen, M., Kramer, A., Grol, R., & Bottema, B. (2010). Behavioural elements of professionalism: Assessment of a fundamental concept in medical care. Medical Teacher, 32(4), e161–e169.
55. van Mook, W. N. K. A., Gorter, S. L., O’Sullivan, H., Wass, V., Schuwirth, L. W., & van der Vleuten, C. P. M. (2009). Approaches to professional behaviour assessment: Tools in the professionalism toolbox. European Journal of Internal Medicine, 20(8), e153–e157.
56. van Mook, W. N., Van Luijk, S. J., Fey-Schoenmakers, M. J., Tans, G., Rethans, J. J., Schuwirth, L. W., & van der Vleuten, C. P. (2010). Combined formative and summative professional behaviour assessment approach in the bachelor phase of medical school: A Dutch perspective. Medical Teacher, 32(12), e517–e531.
57. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–69.
58. Veloski, J. J., Fields, S. K., Boex, J. R., & Blank, L. L. (2005). Measuring professionalism: A review of studies with instruments reported in the literature between 1982 and 2002. Academic Medicine, 80(4), 366–370.
59. Violato, C., Lockyer, J., & Fidler, H. (2003). Multisource feedback: A method of assessing surgical practice. British Medical Journal, 326(7388), 546–548.
60. Violato, C., Lockyer, J. M., & Fidler, H. (2006). Assessment of pediatricians by a regulatory authority. Pediatrics, 117(3), 796–802.
61. Wilkinson, T. J., Wade, W. B., & Knock, L. D. (2009). A blueprint to assess professionalism: Results of a systematic review. Academic Medicine, 84(5), 551–558.
62. Wilmshurst, P. (2013). No doctor should be untouchable. British Medical Journal, 346, f2338.
63. Wood, J., Collins, J., Burnside, E., Albanese, M. A., Propeck, P. A., Kelcz, F., et al. (2004). Patient, faculty, and self-assessment of radiology resident performance: A 360-degree method of measuring professionalism and interpersonal/communication skills. Academic Radiology, 11(8), 931–939.