Exploring the impact of mental workload on rater-based assessments

Advances in Health Sciences Education

Volumn 18, Issue 2, 2013, Pages 291-303

  Tavares, Walter ^a,b   Eva, Kevin W ^c  

^a CENTENNIAL COLLEGE   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

^c UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Assessment; Clinical performance; Competence; Mental workload; Performance assessment; Rater cognition

Indexed keywords

EID: 84876083659     PISSN: 13824996     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10459-012-9370-3     Document Type: Review

References (71)

1. Alexander, A. L., Nygren, T. E., & Vidulich, M. A. (2000). Examining the relationship between mental workload and situation awareness in a simulated air combat task AFRL-HE-WPTR-2000-0094. Air Force Research Laboratory, Human Effectiveness Directorate, Wright-Patterson AFB, OH.
2. Baddeley, A. D. (2007). Working memory, thought and action. Oxford: Oxford University Press.
3. Bannert, M. (2002). Managing cognitive load-recent trends in cognitive load theory. Learning and Instruction, 12(1), 139-146.
4. Beckmann, J. F. (2010). Taming a beast of burden-On some issues with the conceptualisation and operationalisation of cognitive load. Learning and Instruction, 20(3), 250-264.
5. Bogo, M., Regehr, C., Power, R., Hughes, J., Woodford, M., & Regehr, G. (2004). Toward new approaches for evaluating student field performance: Tapping the implicit criteria used by experienced field instructors. Journal of Social Work Education, 40(3), 417-428.
6. Borman, W. C. (1978). Exploring upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63(2), 135-144.
7. Botvinick, M. M., & Rosen, Z. B. (2009). Anticipation of cognitive demand during decision-making. Psychological Research, 73(6), 835-842.
8. Byrne, A., Oliver, M., Bodger, O., Barnett, W., Williams, D., Jones, H., et al. (2010). Novel method of measuring the mental workload of anaesthetists during clinical practice. British Journal of Anaesthesia, 105(6), 767.
9. Cook, D. A., Beckman, T. J., Mandrekar, J. N., & Pankratz, V. S. (2010). Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Advances in Health Sciences Education, 15(5), 633-645.
10. Cooper, W. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218-244.
11. Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: construct alignment improves the performance of workplace based assessment scales. Medical education, 45(6), 560-569.
12. Crossley, J., & Jolly, B. (2012). Making sense of work-based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28-37.
13. Davis, D., Oliver, M., & Byrne, A. (2009). A novel method of measuring the mental workload of anaesthetists during simulated practice. British Journal of Anaesthesia, 103(5), 665.
14. DeNisi, A., Cafferty, T., & Meglino, B. (1984). A cognitive view of the performance appraisal process: A model and research propositions* 1. Organizational Behavior and Human Performance, 33(3), 360-396.
15. Dore, K. L., Hanson, M., Reiter, H. I., Blanchard, M., Deeth, K., & Eva, K. W. (2006). Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Academic Medicine, 81(10), S70.
16. Downing, S. (2005). Threats to the validity of clinical teaching assessments: What about rater error? Medical Education, 39(4), 353-355.
17. Eva, K. W. (2008). On the limits of systematicity. Medical Education, 42(9), 852-853.
18. Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004). An admissions OSCE: The multiple mini interview. Medical Education, 38(3), 314-326.
19. Feldman, J. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66(2), 127-148.
20. Gaugler, B., & Thornton, G. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74(4), 611.
21. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780.
22. Gopher, D., & Braune, R. (1984). On the psychophysics of workload: Why bother with subjective measures? Human Factors, 26, 519-532.
23. Govaerts, M. J. B., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151-165.
24. Govaerts, M., van der Vleuten, C., Schuwirth, L., & Muijtjens, A. (2007). Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Advances in Health Sciences Education, 12(2), 239-260.
25. Green, M. L., & Holmboe, E. (2010). Perspective: The ACGME toolbox: Half empty or half full? Academic Medicine, 85(5), 787.
26. Haber, R., & Avins, A. (1994). Do ratings on the American board of internal medicine resident evaluation form detect differences in clinical competence? Journal of General Internal Medicine, 9(3), 140-145.
27. Herbers, J., Noel, G., Cooper, G., Harvey, J., Pangaro, L., & Weaver, M. (1989). How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine, 4(3), 202-208.
28. Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121(1), 43.
29. Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., & Hanson, M. (1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine, 74(10), 1129.
30. Holmboe, E. (2004). Faculty and the observation of trainees' clinical skills: Problems and opportunities. Academic Medicine, 79(1), 16.
31. Huwendiek, S., Mennin, S., Dern, P., Friedman Ben-David, M., van Der Fleuten, C., Tonshoff, B., et al. (2010). Expertise, needs and challanges of medical educators: Results of an international web survey. Medical Teacher, 32(11), 912-918.
32. Ilgen, D. (1983). & Feldman, J. Performance appraisal: A process focus. Research in Organizational Behavior.
33. Ilgen, D., Barnes-Farrell, J., & McKellin, D. (1993). Performance appraisal process research in the 1980s: what has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321.
34. James, W. (1890). The Principles of Psychology. The Principles of Psychology. New York: Holt.
35. Kalet, A., Earp, J., & Kowlowitz, V. (1992). How well do faculty evaluate the interviewing skills of medical students? Journal of General Internal Medicine, 7(5), 499-505.
36. Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. Journals of American Medical Association, 302(12), 1316-1326.
37. Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010). Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology: General, 139(4), 665.
38. LaMantia, J., Rennie, W., Risucci, D., Cydulka, R., Spillane, L., Graff, L., et al. (1999). Interobserver variability among faculty in evaluations of residents clinical skills. Academic Emergency Medicine, 6(1), 38-44.
39. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72-107.
40. Lavie, N. (1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance, 21(3), 451.
41. Levitin, D. J. (2002). Foundations of cognitive psychology: core readings: The MIT Press, Cambridge.
42. Lord, R. G., & Maher, K. J. (1990). Alternative information-processing models and their implications for theory, research, and practice. The Academy of Management Review, 15(1), 9-28.
43. Lurie, S., Mooney, C., & Lyness, J. (2009). Measurement of the general competencies of the accreditation council for graduate medical education: A systematic review. Academic Medicine, 84(3), 301.
44. Mack, A. (2003). Inattentional blindness. Current Directions in Psychological Science, 12(5), 180.
45. Mack, A., & Rock, I. (1998). Inattentional Blindness. Cambridge, MA: MIT Press.
46. Margolis, M., Clauser, B., Cuddy, M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10), S56.
47. Marois, R., & Ivanoff, J. (2005). Capacity limits of information processing in the brain. Trends in Cognitive Sciences, 9(6), 296-305.
48. Mayer, R. E. (2010). Applying the science of learning to medical education. Medical Education, 44(6), 543-549.
49. Melchers, K. G., Kleinmann, M., & Prinz, M. A. (2010). Do assessors have too much on their plates? The effects of simultaneously rating multiple assessment center candidates on rating quality. International Journal of Selection and Assessment, 18(3), 329-341.
50. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81.
51. Morgeson, F., & Campion, M. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82(5), 627-654.
52. Noel, G., Herbers, J., Caplow, M., Cooper, G., Pangaro, L., & Harvey, J. (1992). How well do internal medicine faculty members evaluate the clinical skills of residents? Annals of Internal Medicine, 117(9), 757.
53. Norcini, J., Blank, L., Duffy, F., & Fortna, G. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476-481.
54. Paas, F., Tuovinen, J. E., Tabbers, H., & van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist, 38(1), 63-71.
55. Pashler, H. (Ed.). (1998). Attention. East Sussex, UK: Psychology Press, Ltd.
56. Payne, J. W. (1980). Information processing theory: Some concepts and methods applied to decision research. Cognitive processes in choice and decision behavior, 95, 115.
57. Pérez Moreno, E., Conchillo, Á., & Recarte, M. A. (2011). The Role of Mental Load in Inattentional Blindness. Psicológica: Revista de metodología y psicología experimental, 32(2), 255-278.
58. Regehr, G., Bogo, M., Regehr, C., & Power, R. (2007). Can we build a better mousetrap? Improving the measures of practice performance in the field practicum. Journal of Social Work Education, 43(2), 327-344.
59. Schulz, C. M., Skrzypczak, M., Schneider, E., Hapfelmeier, A., Martin, J., Kochs, E. F., et al. (2011). Assessment of subjective workload in an anaesthesia simulator environment: reliability and validity. European Journal of Anaesthesiology, 28(7), 502.
60. Shah, A. K., & Oppenheimer, D. M. (2008). Heuristics made easy: An effort-reduction framework. Psychological Bulletin, 134(2), 207.
61. Simons, D. (2000). Attentional capture and inattentional blindness. Trends in Cognitive Sciences, 4(4), 147-155.
62. Thompson, W. G., Lipkin, M., Jr, Gilbert, D. A., Guzzo, R. A., & Roberson, L. (1990). Evaluating evaluation: assessment of the American board of internal medicine resident evaluation form. Journal of General Internal Medicine, 5(3), 214-217.
63. Tsang, P. S., & Vidulich, M. A. (2003). Principles and practice of aviation psychology Mahwah, NJ: Lawrence Erlbaum Associates.
64. Tsang, P. S., & Vidulich, M. A. (2006). Mental workload and situation awareness. Hoboken, NJ: Wiley.
65. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124.
66. van Merriënboer, J. J. G., & Sweller, J. (2010). Cognitive load theory in health professional education: Design principles and strategies. Medical Education, 44(1), 85-93.
67. Wickens, C. D., & Carswell, C. (2006). Handbook of human factors and ergonomics (3rd ed.). Hoboken, NJ: Wiley.
68. Williams, R., Klamen, D., & McGaghie, W. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270-292.
69. Young, G., Zavelina, L., & Hooper, V. (2008). Assessment of workload using NASA task load index in perianesthesia nursing. Journal of PeriAnesthesia Nursing, 23(2), 102-110.
70. Yurko, Y. Y., Scerbo, M. W., Prabhu, A. S., Acker, C. E., & Stefanidis, D. (2010). Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simulation in Healthcare, 5(5), 267.
71. Zheng, B., Cassera, M. A., Martinec, D. V., Spaun, G. O., & Swanström, L. L. (2010). Measuring mental workload during the performance of advanced laparoscopic tasks. Surgical Endoscopy, 24(1), 45-50.