The reliability of workplace-based assessment in postgraduate medical education and training: A national evaluation in general practice in the United Kingdom

Advances in Health Sciences Education

Volumn 14, Issue 2, 2009, Pages 219-232

  Murphy, Douglas J ^a   Bruce, David A ^b   Mercer, Stewart W ^c   Eva, Kevin W ^d  

^a UNIVERSITY OF DUNDEE   (United Kingdom)

^b NHS EDUCATION FOR SCOTLAND   (United Kingdom)

^c UNIVERSITY OF GLASGOW   (United Kingdom)

^d MCMASTER UNIVERSITY   (Canada)

Author keywords

Medical education; Multi source feedback; Patient satisfaction; Physician assessment; Workplace based assessment

Indexed keywords

ADULT; ARTICLE; CLINICAL COMPETENCE; EDUCATION; EDUCATIONAL STATUS; FEASIBILITY STUDY; FEMALE; GENERAL PRACTICE; HUMAN; MALE; MEDICAL EDUCATION; PATIENT SATISFACTION; PILOT STUDY; QUESTIONNAIRE; REPRODUCIBILITY; STANDARD; STATISTICS; UNITED KINGDOM; VALIDATION STUDY; WORKPLACE; STANDARDS; STATISTICS AND NUMERICAL DATA;

ADULT; CLINICAL COMPETENCE; EDUCATIONAL MEASUREMENT; EDUCATIONAL STATUS; FAMILY PRACTICE; FEASIBILITY STUDIES; FEMALE; GREAT BRITAIN; HUMANS; INTERNSHIP AND RESIDENCY; MALE; PATIENT SATISFACTION; PILOT PROJECTS; QUESTIONNAIRES; REPRODUCIBILITY OF RESULTS; WORKPLACE;

EID: 63949083879     PISSN: 13824996     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10459-008-9104-8     Document Type: Article

References (48)

1. E. W. Ackerman G. K. Mitchell 2006 An audit of structured diabetes care in a rural general practice Medical Journal of Australia 185 2 69 72
2. J. C. Archer J. Norcini H. A. Davies 2005 Use of SPRAT for peer review of paediatricians in training British Medical Journal 330 1251 1253
3. P. Aveyard 1997 Monitoring the performance of general practices Journal of Evaluation in Clinical Practice 3 4 275 281
4. R. Baker R. Jones David P. Goldblatt 2003 Monitoring mortality rates in general practice after Shipman British Medical Journal 326 7383 274 276
5. L. M. Campbell J. G. R. Howie T. S. Murray 1993 Summative assessment: A pilot project in the west of Scotland British Journal of General Practice 43 430 434
6. L. M. Campbell J. G. Howie T. S. Murray 1995 Use of videotaped consultations in summative assessment of trainees in general practice British Journal of General Practice 45 392 137 141
7. J. G. M. Crossley A. Howe D. Newble B. Jolly H. A. Davies 2001 Sheffield Assessment Instrument for Letters (SAIL): Performance assessment using outpatient letters Medical Education 35 1115 1124
8. M. H. Davis M. Friedman R. M. Harden P. Howie J. Ker C. McGhee M. J. Pippard D. Snadden 2001 Portfolio assessment in medical students' final examinations Medical Teacher 23 357 366
9. K. W. Eva 2007 Putting the cart before the horse: Testing to improve learning British Medical Journal 334 535
10. R. G. Evans A. Edwards S. Evans B. Elwyn G. Elwyn 2007 Assessing the practicing physician using patient surveys: A systematic review of instruments and feedback methods Family Practice 24 128 137
11. R. Evans G. Elwyn A. Edwards 2004 Review of instruments for peer assessment of physicians British Medical Journal 328 1240 1243
12. Grant, A. J., Vermunt, J. D., Kinnersley, P., & Houston, H. (2007). Exploring students' perceptions on the use of significant event analysis, as part of a portfolio assessment process in general practice, as a tool for learning how to use reflection in learning. BMC Medical Education, 7, 5. doi: 10.1186/1472-6920-7-5.
13. J. G. R. Howie D. J. Heaney M. Maxwell J. J. Walker 1998 A comparison of a Patient Enablement Instrument (PEI) against two established satisfaction scales as an outcome measure of primary care consultations Family Practice 15 2 165 171
14. Joshi, H. et al. (2007). Developing and maintaining an assessment system - a PMETB guide to good practice. January 2007. http://www.pmetb.org.uk/ fileadmin/user/QA/Assessment/Assessment-system-guidance-0107.pdf (accessed 10.01.2008).
15. J. Lockyer 2003 Multi-source feedback in the assessment of physician competencies Journal of Continuing Education in the Health Professions 23 1 4 12
16. J. R. Lough T. S. Murray 2001 Audit and summative assessment: A completed audit cycle Medical Education 35 4 357 363
17. M. R. Lynn 1986 Determination and quantification of content validity Nursing Research 35 382 385
18. J. McKay P. Bowie M. Lough 2003 Evaluating significant event analyses: Implementing change is a measure of success Education for Primary Care 14 34 38
19. J. McKay D. J. Murphy P. Bowie M. Schmuck M. Lough K. W. Eva 2007 Development and testing of an instrument for the formative peer assessment of significant event analyses Quality and Safety in Health Care 16 150 153
20. S. W. Mercer J. G. R. Howie 2006 CQI-2, a new measure of holistic, interpersonal care in primary care consultations British Journal of General Practice 56 525 262 268
21. S. W. Mercer A. McConnachie M. Maxwell D. H. Heaney G. C. M. Watt 2005 Relevance and performance of the Consultation and Relational Empathy (CARE) Measure in general practice Family Practice 22 3 328 334
22. S. W. Mercer G. C. M. Watt M. Maxwell D. H. Heaney 2004 The development and preliminary validation of the Consultation and Relational Empathy (CARE) Measure: An empathy-based consultation process measure Family Practice 21 6 699 705
23. Modernising Medical Careers (MMC). http://www.mmc.nhs.uk/pages/home (accessed 10.05.2007).
24. Multi-Source Feedback: 360° Team Assessment of Behaviour (TAB) West Midlands Deanery, UK. http://www.wmdeanery.org/Downloads/360download.asp (accessed 10.05.2007).
25. D. J. Murphy D. A. Bruce K. W. Eva 2008 Workplace-based assessment for general practitioners: Using stakeholder perception to aid blueprinting of an assessment battery Medical Education 42 96 103
26. National Office for Summative Assessment. First level assessor's instructions and marking schedule. http://www.nosa.org.uk (accessed 17.02.2008)
27. National Office for Summative Assessment. http://www.nosa.org.uk/ downloads/html/audit/marking.htm (accessed 22.05.2007).
28. J. J. Norcini L. L. Blank G. K. Arnold H. R. Kimball 1995 The Mini-CEX (Clinical Evaluation Exercise): A preliminary investigation Annals of Internal Medicine 123 10 795 799
29. J. Pitts C. Coles P. Thomas 1999 Educational portfolios in the assessment of general practice trainers: Reliability of assessors Medical Education 33 515 520
30. P. Ram R. Grol J. J. Rethans B. Schouten C. van der Vleuten A. Kester 1999 Assessment of general practitioners by video observation of communicative and medical performance in daily practice: Issues of validity, reliability, and feasibility Medical Education 33 6 447 454
31. P. G. Ramsay M. D. Weinrich J. D. Carline T. S. Innui E. B. Larson J. P. LoGerfo 1993 Use of peer ratings to evaluate physician performance The Journal of the American Medical Association 269 1655 1660
32. P. Ramsey M. Wenrich 1999 Peer ratings: An assessment tool whose time has come Journal of General Internal medicine 14 581 582
33. RCGP: Video assessment of consulting skills in 2008; Workbook and instructions http://www.rcgp.org.uk/the-gp-journey/mrcgp/video-workbook.aspx (accessed 13.01.2008).
34. Referral Advice. (2001). A guide to appropriate referral from general to specialist services. London: National Institute for Clinical Evidence (NICE).
35. R. Reznick S. Smee A. Rothman A. Chalmers D. Swanson L. Dufresne G. Lacombe J. Baumber P. Poldre L. Lavasseur 1992 An objective structured clinical examination for the licentiate: Report of the pilot project of the Medical Council of Canada Academic Medicine 67 487 494
36. C. Roberts 2002 Portfolio-based assessments in medical education: Are they valid and reliable for summative purposes? Medical Education 36 899 900
37. J. Sargeant K. Mann S. Ferrier 2005 Making available a mentoring service to support physician feedback, reflection, learning and change, can increase acceptance and use of feedback Medical Education 39 497 504
38. J. Sargeant K. Mann D. Sinclair C. van der Vleuten J. Metsemakers 2007 Challenges in multi-source feedback: Intended and unintended outcomes Medical Education 41 583 591
39. L. W. T. Schuwirth C. P. M. van der Vleuten 2006 Challenges for educationalists British Medical Journal 333 544 546
40. Scottish Intercollegiate Guideline Network (SIGN). (1998). Report on a recommended referral document. Edinburgh: SIGN.
41. Scottish Revalidation Toolkit, RCGP Scotland. http://www.rcgp.org.uk/pdf/ Complete%20Revalidation%20Toolkit%20(Read%20Only)%20PDF.pdf (accessed 10.05.2007).
42. Streiner, D. L., & Norman, G. R. (2003). Health measurement scales (3rd ed.). Oxford: Oxford Medical Publications.
43. D. Swanson G. R. Norman R. I. Linn 1995 Performance based assessment: Lessons from the health professions Educational Researcher 24 5 12
44. P. Tate J. Foulkes R. Neighbour P. Campion S. Field 1999 Assessing physicians' interpersonal skills via videotaped encounters: A new approach for the Royal College of general practitioners membership examination Journal of Health Communication 4 143 152
45. C. P. M. van der Vleuten 1996 The assessment of professional competence: Developments, research and practical implications Advances in Health Sciences Education 1 41 67
46. S. J. Verhulst J. A. Colliver R. E. Paiva R. G. Williams 1986 A factor analysis study of first-year residents Journal of Medical Education 61 132 134
47. wbapilot@chs.dundee.ac.uk available http://www.dundee.ac.uk/gptraining
48. R. G. Williams S. Verhulst J. A. Colliver G. L. Dunnington 2005 Assessing the reliability of resident performance appraisals: More items or more observations? Surgery 137 141 147