Issues in developing valid assessments of speech pathology students' performance in the workplace

International Journal of Language and Communication Disorders

Volumn 45, Issue 1, 2010, Pages 1-14

  McAllister, Sue ^a   Lincoln, Michelle ^a   Ferguson, Alison ^b   McAllister, Lindy ^c  

^a UNIVERSITY OF SYDNEY   (Australia)

^b UNIVERSITY OF NEWCASTLE   (Australia)

^c UNIVERSITY OF QUEENSLAND   (Australia)

Author keywords

Clinical education; Competency based assessment; Workplace based learning

Indexed keywords

CLINICAL ASSESSMENT TOOL; COMPETENCE; CONTENT ANALYSIS; DECISION MAKING; JOB PERFORMANCE; LEARNING; LITERATURE; PERFORMANCE MEASUREMENT SYSTEM; QUALITY CONTROL; RASCH ANALYSIS; RELIABILITY; REVIEW; SPEECH ANALYSIS; STUDENT; VALIDITY; WORKPLACE;

EDUCATIONAL MEASUREMENT; HUMANS; SPEECH-LANGUAGE PATHOLOGY;

EID: 72749124243     PISSN: 13682822     EISSN: 14606984     Source Type: Journal    
DOI: 10.3109/13682820902745461     Document Type: Review

References (81)

1. ALEXANDER, H., 1996, Physiotherapy student clinical education: the influence of subjective judgements on observational assessment. Assessment and Evaluation in Higher Education, 21, 357-366.
2. AMERICAN EDUCATIONAL RESEARCH ASSOCIATION (AERA), 1999, Validity. Standards for Educational and Psychological Testing (Washington, DC: American Educational Research Association).
3. ANDERSON, J., 1988, The Supervisory Process in Speech-Language Pathology and Audiology (Boston, MA: College Hill).
4. BARNARD, J. J., 1999, Item analysis in test construction. In G. N. Masters and J. P. Keeves (eds), Advances in Measurement in Educational Research and Assessment (Oxford: Elsevier), pp. 195-206.
5. BARRIE, S. C., 2006, Understanding what we mean by the generic attributes of graduates. Higher Education, 51, 215-241.
6. BENNER, P., 1984, From Novice to Expert: Excellence and Power in Clinical Nursing Practice (Englewood Cliffs, NJ: Prentice- Hall).
7. BENNER, P. A., TANNER, C. A. and CHESLA, C. A., 1996, Expertise in Nursing Practice: Caring, Clinical Judgment, and Ethics (New York, NY: Springer).
8. BIGGS, J. B. and COLLIS, K. F., 1982, Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome) (New York, NY: Academic Press).
9. BITZER, E., 1999, Assessing Learning in the MPhil (Higher Education Studies): paving new ways with cornerstones. Paper presented at the Higher Education Research and Development Society of Australasia Annual International Conference, Melbourne, Australia.
10. BOND, T. G. and FOX, C. M., 2007, Applying the Rasch Model: Fundamental Measurement in the Human Sciences (Mahwah, NJ: Lawrence Erlbaum Associates).
11. BOUD, D., 2000, Sustainable assessment: rethinking assessment for the learning society. Studies in Continuing Education, 22, 151-167.
12. BOUD, D. and FALCHIKOV, N., 2006, Aligning assessment with long-term learning. Assessment and Evaluation in Higher Education, 31, 399-413.
13. BRASSEUR, J., 1989, The supervisory process: a continuum perspective. Language, Speech and Hearing Services in Schools, 20, 274-295.
14. BROOKHART, S. M., 2001, Successful students' formative and summative uses of assessment information. Assessment in Education, 8, 153-168.
15. BRUNER, J., 1983, Child's Talk: Learning to Use Language (New York, NY: W.W. Norton).
16. CARRACCIO, C., WOLFSTHAL, S. D., ENGLANDER, R., FERENTZ, K. and MARTIN, C., 2002, Shifting paradigms: drom Flexner to competencies. Academic Medicine, 77, 361-367.
17. CARTER, R., 1985, A taxonomy of objectives for professional education. Studies in Higher Education, 10, 135-149.
18. CHAPMAN, J., 1998, Agonising about assessment. In D. Fish and C. Coles (eds), Developing Professional Judgement in Health Care: Learning through the Critical Appreciation of Practice (Oxford: Butterworth-Heinemann), pp. 157-181.
19. CLARK, L. A. and WATSON, D., 1995, Constructing validity: basic issues in objective scale development. Psychological Assessment, 7, 309-319.
20. COX, K., 2000, Examining and recording clinical performance: a critique and some recommendations. Education for Health, 13, 45-52.
21. CROSS, V., HICKS, C. and BARWELL, F., 2001, Exploring the gap between evidence and judgement: using video vignettes for practice-based assessment of physiotherapy undergraduates. Assessment and Evaluation in Higher Education, 26, 189-212.
22. DAWSON, V., 1995, C-BOS for students: developing professional competencies in speech pathology students (part 1). Australian Communication Quarterly, Summer, 6-7.
23. DAWSON, V., 2003, The competence of the Australian new graduate speech pathologist. Paper presented at the Speech Pathology Australia National Conference, Hobart, Australia, 4-8 May 2003.
24. DREYFUS, H. L. and DREYFUS, S. E., 1996, The relationship of theory and practice in the acquisition of skill. In P. A. Benner, C. A. Tanner and C. A. Chesla (eds), Expertise in Nursing Practice: Caring, Clinical Judgment, and Ethics (New York, NY: Springer), pp. 29-47.
25. DUKE, M., 1996, Clinical evaluation - difficulties experienced by sessional clinical teachers of nursing: a qualitative study. Journal of Advanced Nursing, 23, 408-414.
26. EMBRETSON, S. E., 1999, Issues in the measurement of cognitive abilities. In S. E. Embretson and S. L. Hershberger (eds), The New Rules of Measurement: What Every Psychologist and Educator Should Know (Mahwah, NJ: Lawrence Erlbaum Associates), pp. 1-16.
27. EMBRETSON, S. E. and REISE, S. P., 2000, Item Response Theory for Psychologists (Mahwah, NJ: Lawrence Erlbaum Associates).
28. EPSTEIN, R.M. and HUNDERT, E. M., 2002, Defining and assessing professional competence. Journal of the American Medical Association, 287, 226-235.
29. FAY, C. H. and LATHAM, G. P., 1982, Effects of training and rating scales on rating errors. Personnel Psychology, 35, 105-117.
30. FERGUSON, A., 2006, Competency-based occupational standards: influences on Australian speech pathology education. Folia Phoniatrica et Logopaedica, 58, 23-31.
31. FERGUSON, A., LINCOLN, M.,MCALLISTER, L. andMCALLISTER, S., 2008, COMPASSw Directions: Leading the Integration of a Competency-based Assessment Tool in Speech Pathology Learning and Teaching. Report prepared for Australian Learning & Teaching Council Ltd, Sydney, an initiative of the Australian Government Department of Education, Employment and Workplace Relations (available at: http:// www.altc.edu.au/).
32. FRIEDMAN, M. and MENNIN, S., 1991, Rethinking critical issues in performance assessment. Academic Medicine, 66, 390-395.
33. GOMEZ-MEJIA, G., 1988, Evaluating employee performance: does the appraisal instrument make a difference? Journal of Organizational Behavior Management, 9, 155-172.
34. GOVAERTS, J. J. B., VAN DER VLEUTEN, C. and SCHUWIRTH, L. W. T., 2002, Optimising the reproducibility of a performancebased test in midwifery education. Advances in Health Sciences Education, 7, 133-145.
35. GOVAERTS, M. J. B., VAN DER VLEUTEN, C. P. M., SCHUWIRTH, L. W. T. and MUIJTJENS, A. M. M., 2007, Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12, 239-260.
36. HAGER, P., 2000, Know-how and workplace practical judgement. Journal of Philosophy of Education, 34, 281-296.
37. HAGER, P., ATHANASOU, J. and GONCZI, A., 1994, Assessment Technical Manual (Canberra, ACT: Australian Government Publishing Service).
38. HAGER, P. and GILLIS, S., 1995, Assessment at Higher Levels. In W. C. Hall (ed.), Key aspects of competency-based assessment (Adelaide, SA: NCVER), pp. 59-71.
39. HARRIS, R., GUTHRIE, H., HOBART, B. and LUNDBERG, D., 1995, Competency-based Education and Training: Between a Rock and a Whirlpool (Sydney: Macmillan Education Australia).
40. HAYS, R. B., DAVIES, H. A., BEARD, J. D., CALDON, L. F. M., FARMER, E. A., FINUCANE, P. M., MCCRORIE, P., NEWBLE, D., SCHUWIRTH, L. W. T. and SIBBALD, G. R., 2002, Selecting performance assessment methods for experienced physicians. Medical Education, 36, 910-917.
41. HIGGS, J. and BITHELL, C., 2001, Professional expertise. In J. Higgs and A. Titchen (eds), Practice Knowledge and Expertise in the Health Professions (Oxford: Butterworth-Heinemann), pp. 59-68.
42. HUNT, D. D., 1992, Functional and dysfunctional characteristics of the prevailing model of clinical evaluation systems in North American medical schools. Academic Medicine, 67, 254-259.
43. ILOTT, I. and MURPHY, R., 1997, Feelings and failing in professional training: the assessor's dilemma. Assessment and Evaluation in Higher Education, 22, 307-316.
44. JOHNSON, C. J. and SHEWAN, C. M., 1988, A new perspective in evaluating clinical effectiveness: the UWO clinical grading system. Journal of Speech and Hearing Disorders, 53, 328-340.
45. JONES, A., 2000, The place of judgment in competency-based assessment. Journal of Vocational Education and Training, 51, 145-160.
46. KEEN, A. J. A., KLEIN, S. and ALEXANDER, D. A., 2003, Assessing the communication skills of doctors in training: reliability and sources of error. Advances in Health Sciences Education, 8, 5-16.
47. KINGSTROM, P. O. and BASS, A. R., 1981, A critical analysis of studies comparing Behaviorally Anchored Rating Scales (BARS) and other rating formats. Personnel Psychology, 34, 263-289.
48. LANDON, B. E., NORMAND, S.-L. T., BLUMENTHAL, D. and DALEY, J., 2003, Physician Clinical performance assessment. Journal of the American Medical Association, 9, 1183-1189.
49. LANDY, F. J. and FARR, J. L., 1980, Performance rating. Psychological Bulletin, 87, 72-107.
50. LEACH, L., NEUTZE, G. and ZEPKE, N., 2001, Assessment and empowerment: Some critical questions. Assessment and Evaluation in Higher Education, 26, 293-305.
51. LINACRE, J. M., 1994, Sample size and item calibration stability. Rasch Measurement Transactions [e-journal], 7 (4), 328.
52. LINACRE, J. M., 2002, Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85-106.
53. LINCOLN, M., FERGUSON, A.,MCALLISTER, L. andMCALLISTER, S., 2008, Benchmarking Clinical Learning in Speech Pathology to Support Assessment, Discipline Standards, Teaching Innovation and Student Learning. Report prepared for Australian Learning & Teaching Council Ltd, Sydney, an initiative of the Australian Government Department of Education, Employment and Workplace Relations (available at: http:// www.altc.edu.au/).
54. LINN, R. L., BAKER, E. L. and DUNBAR, S. B., 1991, Complex, performance-based assessment: expectations and validation criteria. Educational Researcher, 20, 15-21.
55. MCALLISTER, S., 2005, Competency Based Assessment of Speech Pathology Students' Performance in the Workplace (Sydney: Faculty of Health Sciences, The University of Sydney) (available at: http://hdl.handle.net/2123/1130).
56. MCALLISTER, S., 2008, Introduction to the use of Rasch analysis to assess patient performance. International Journal of Therapy and Rehabilitation, 15, 482-490.
57. MCALLISTER, S., LINCOLN, M., FERGUSON, A. andMCALLISTER, L., 2006, COMPASSw: Competency Assessment in Speech Pathology (Melbourne: Speech Pathology Association of Australia).
58. MCALLISTER, S., LINCOLN, M., FERGUSON, A. andMCALLISTER, L., 2008, Engaging speech pathology students in design and validation of competency-based assessment in the workplace. Paper presented at the ATN Assessment Conference, Adelaide, Australia.
59. MCALLISTER, S. M., LINCOLN, M., FERGUSON, A. andMCALLISTER, L., 2004, User evaluation of the content and processes of an assessment of student performance in workplace settings. Paper presented at the IALP World Congress, 2004, Brisbane, Australia.
60. MCGAGHIE, W. C., 1993, Evaluating competence for professional practice. In L. Curry and J. F. Wergin (eds), Educating Professionals (San Francisco, CA: Jossey-Bass), pp. 229-261.
61. MICHELL, J., 1997, Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88, 355-383. (Pubitemid 127457836)
62. MOSS, P., 1992, Shifting conceptions of validity in educational measurement: implications for performance assessment. Review of Educational Research, 62, 229-258.
63. NEWBLE, D., NORMAN, G. and VAN DER VLEUTEN, C., 2000, Assessing clinical reasoning. In J. Higgs and M. Jones (eds), Clinical Reasoning in the Health Professions, 2nd edn (Oxford: Butterworth-Heinemann), pp. 156-165.
64. NORCINI, J. J., 2005, Current perspectives in assessment: the assessment of performance at work. Medical Education, 39, 880-889.
65. NORMAN, G., VAN DER VLEUTEN, C. and DE GRAAFF, E., 1991, Pitfalls in the pursuit of objectivity: issues of reliability. Medical Education, 25, 110-118.
66. PETERS, G., FRASER, J., COWIE, F., LOADER, J., RUTTER, S., SCOTT, R. and DAVIE, B., 2001, Competency-based assessment in a perioperative nursing graduate diploma. ACORN Journal, 15, 11-14.
67. RETHANS, J.-J., NORCINI, J., BARON-MALDONADO, M., BLACKMORE, D., JOLLY, B. C., LADUCA, T., LEW, S. R., PAGE, G. G. and SOUTHGATE, L. J., 2002, The relationship between competence and performance: Implications for assessing practice performance. Medical Education, 36, 901-909. (Pubitemid 35286311)
68. ROBERTSON, S., ROSENTHAL, J. and DAWSON, V., 1997, Using assessment to promote student learning. In L. McAllister, M. Lincoln and S. McLeod (eds), Facilitating Learning in Clinical Settings (Cheltenham: Stanley Thornes), pp. 154-184.
69. ROSE, M. and BEST, D. (eds), 2005, Transforming Practice through Clinical Education, Professional Supervision and Mentoring (Edinburgh: Elsevier Churchill Livingstone).
70. ROYAL COLLEGE OF SPEECH & LANGUAGE THERAPISTS (RCSLT), 2007, Newly Qualified Practitioners Competency Framework: Speech and Language Therapy Competency Framework to Guide Transition to certified RCSLT Membership (London: Royal College of Speech & Language Therapists) (available at: http://www.rcslt.org/resources/newlyqualifiedpractitioners) (accessed on 11 October 2008).
71. SCHUWIRTH, L. W. T. and VAN DER VLEUTEN, C. P. M., 2006, A plea for new psychometric models in educational assessment. Medical Education, 40, 296-300.
72. SHAVELSON, R. J., GAO, X. and BAXTER, G., 1993, Sampling Variability in Performance Assessments. CSE Technical Report Number 361 (Santa Barbara, CA: National Center for Research on Evaluation, Standards, and Student Testing, University of California).
73. SPEECH PATHOLOGY ASSOCIATION OF AUSTRALIA (SPAA), 2001, Competency-based Occupational Standards for Speech Pathologists (Entry Level) (Melbourne: Speech Pathology Association of Australia).
74. SPIKE, N., ALEXANDER, H., ELLIOTT, S., HAZLETT, C., KILMINSTER, S., PRIDEAUX, D. and ROBERTS, T., 2000, In-training assessment-its potential in enhancing clinical teaching. Medical Education, 34, 858-861.
75. WASS, V., VAN DER VLEUTEN, C., SHATZER, J. and JONES, J., 2001, Assessment of clinical competence. Lancet, 357, 945-949.
76. WILKINSON, T. J. and FRAMPTON, C. M., 2003, Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting? Medical Education, 37, 233-240.
77. WOEHR, D. J. and HUFFCUTT, A. I., 1994, Rater training for performance appraisal: a quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.
78. WOLF, A., 1995, Competence-based Assessment (Philadelphia, PA: Open University Press).
79. WOLFE, E. W. and GITOMER, D. H., 2001, The influence of changes in assessment design on the psychometric quality of scores. Applied Measurement in Education, 14, 91-107.
80. WRIGHT, B. D., 1999, Fundamental measurement for psychology. In S. E. Embretson and S. L. Hershberger (eds), The New Rules of Measurement: What Every Psychologist and Educator Should Know (Mahwah, NJ: Lawrence Erlbaum Associates), pp. 65-106.
81. ZHU, W., 1996, Should total scores from a rating scale be used directly? Research Quarterly for Exercise and Sports, 67, 363-372.