A systematic review of behavioural marker systems in healthcare: What do we know about their attributes, validity and application?

BMJ Quality and Safety

Volumn 23, Issue 12, 2014, Pages 1031-1039

  Dietz, Aaron S ^a   Pronovost, Peter J ^a   Benson, Kari N ^a   Mendez Tellez, Pedro Alejandro ^a   Dwyer, Cynthia ^b   Wyskiel, Rhonda ^a   Rosen, Michael A ^a  

^a JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b JOHNS HOPKINS HOSPITAL   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIOR; CHECKLIST; CONSENSUS; DECISION MAKING; FEEDBACK SYSTEM; HEALTH CARE; HEALTH CARE QUALITY; HUMAN; LIKERT SCALE; MEASUREMENT ERROR; MEDICAL RESEARCH; METHODOLOGY; NOMENCLATURE; PATIENT SAFETY; PSYCHOMETRY; RELIABILITY; REVIEW; SKILL; SYSTEMATIC REVIEW (TOPIC); TASK PERFORMANCE; TEAMWORK; VALIDITY; HEALTH SERVICES RESEARCH; PATIENT CARE; PROCEDURES; STANDARDS; TOTAL QUALITY MANAGEMENT; TREATMENT OUTCOME;

FEEDBACK; HEALTH SERVICES RESEARCH; HUMANS; OUTCOME AND PROCESS ASSESSMENT (HEALTH CARE); PATIENT CARE TEAM; PSYCHOMETRICS; QUALITY IMPROVEMENT; QUALITY INDICATORS, HEALTH CARE;

EID: 84910609619     PISSN: 20445415     EISSN: None     Source Type: Journal    
DOI: 10.1136/bmjqs-2013-002457     Document Type: Review

References (87)

1. Pronovost PJ, Vohr E. Safe patients, smart hospitals: how one doctor 's checklist can help us change healthcare from the inside out. New York, NY: Plume, 2010.
2. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. results of the harvard medical practice study II. N Engl J Med 1991;324:377-84.
3. Gawande AA, Zinner MJ, Studdert DM, et al. Analysis of errors reported by surgeons at three teaching hospitals. Surgery 2003;133:614-21.
4. Pronovost PJ, Thompson DA, Holzmueller CG, et al. Toward learning from patient safety reporting systems. J Crit Care 2006;21:305-15.
5. Burke CS, Salas E, Wilson-Donnelly K, et al. How to turn a team of experts into an expert medical team: guidance from the aviation and military communities. Qual Saf Health Care 2004;13(Suppl 1):i96-104.
6. Rosen MA, Salas E, Silvestri S, et al. A measurement tool for simulation-based training in emergency medicine: the simulation module for assessment of resident targeted event responses (SMARTER) approach. Simul Healthc 2008;3:170-9.
7. Schmutz J, Manser T. Do team processes really have an effect on clinical performance? A systematic literature review. Br J Anaesth 2013;110:529-44.
8. Buljac-Samardzic M, Dekker-van Doorn CM, van Wijngaarden JD, et al. Interventions to improve team effectiveness: a systematic review. Health Policy 2010;94:183-95.
9. Rosen MA, Salas E, Wilson KA, et al. Measuring team performance in simulation-based training: adopting best practices for healthcare. Simul Healthc 2008;3:33-41.
10. Rosen MA, Schiebel N, Salas E, et al. How can team performance be measured, assessed, and diagnosed? In: Salas E, Frush K, eds. Improving patient safety through teamwork and team training. Oxford, UK: Oxford University Press, 2012:59-79.
11. Flin R, Martin L. Behavioral markers for crew resource management: a review of current practice. Int J Aviat Psychol 2001;11:95-118.
12. Rosen MA, Weaver SJ, Lazzara EH, et al. Tools for evaluating team performance in simulation-based training. J Emerg Trauma Shock 2010;3:353-9.
13. Pedhazur EJ, Schmelkin LP. Measurement, design, and analysis: an integrated approach. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Inc., 1991.
14. Kozlowski SWJ, Ilgen DR. Enhancing the effectiveness of work groups and teams. Psychol Sci Public Interest 2006;7:77-124.
15. Salas E, Cooke NJ, Rosen MA. On teams, teamwork, and team performance: discoveries and developments. Hum Factors 2008;50:540-7.
16. Nestel D, Walker K, Simon R, et al. Nontechnical skills: an inaccurate and unhelpful descriptor? Simul Healthc 2011;6:2-3.
17. Salas E, Dickinson TL, Converse S, et al. Toward an understanding of team performance and training. In: Swezey RW, Salas E, eds. Teams: their training and performance. Norwood, NJ: Ablex, 1992:3-29.
18. Baker DP, Salas E. Principles for measuring teamwork skills. Hum Factors 1992;34:469-75.
19. Salas E, Stagl KC, Burke CS, et al. Fostering team effectiveness in organizations: Toward an integrative theoretical framework of team performance. In: Shuart JW, Spaulding W, Poland J, eds. Modeling complex systems: Motivation, cognition and social processes. nebraska symposium on motivation. Vol 51.Lincoln, NE: University of Nebraska Press, 2007:185-243.
20. Gordon M, Darbyshire D, Baker P. Non-technical skills training to enhance patient safety: a systematic review. Med Educ 2012;46:1042-54.
21. Reader TW, Flin R, Mearns K, et al. Developing a team performance framework for the intensive care unit. Crit Care Med 2009;37:1787-93.
22. Marks MA, Mathieu JE, Zaccaro J. A temporally based framework and taxonomy of team processes. Acad Manag Rev 2001;26:356-76.
23. LePine JA, Piccolo RF, Jackson CL, et al. A meta-analysis of teamwork processes: tests of a multidimensional model and relationships with team effectiveness criteria. Personnel Psychol 2008;61:273-307.
24. Endsley MR. Toward a theory of situation awareness in dynamic systems. Hum Factors 1995;37:32-64.
25. Reader TW, Flin R, Mearns K, et al. Team situation awareness and the anticipation of patient progress during ICU rounds. BMJ Qual Saf 2011;20:1035-42.
26. ANTS. ANTS: A behavioural marker system for rating anaesthetists' non-technical skills. http://www.abdn.ac.uk/iprc/ ants/. Updated 2012. (accessed 13 Feb 2013).
27. NOTSS. NOTSS: Non-technical skills for surgeons. http:// www.abdn.ac.uk/iprc/notss. Updated 2012. (accessed 13 Feb 2013).
28. Cannon-Bowers JA, Tannenbaum SI, Salas E, et al. Defining competencies and establishing team training requirements. In: Guzzo R, Salas E, eds. Team effectiveness and decision making in organizations. San Francisco, CA: Jossey-Bass, 1995:333-80.
29. Healey AN, Undre S, Vincent CA. Developing observational measures of performance in surgical teams. Qual Saf Health Care 2004;13(Suppl 1):i33-40.
30. Grand JA, Pearce M, Rench TA, et al. Going DEEP: Guidelines for building simulation-based team assessments. BMJ Qual Saf 2013;22:436-48.
31. Feldman M, Lazzara EH, Vanderbilt AA, et al. Rater training to support high-stakes simulation-based assessments. J Contin Educ Health Prof 2012;32:279-86.
32. Yule S, Flin R, Maran N, et al. Surgeons' non-technical skills in the operating room: Reliability testing of the NOTSS behavior rating system. World J Surg 2008;32:548-56.
33. nd edn. Boston, MA: Allyn & Bacon, 2010.
34. Mishra A, Catchpole K, McCulloch P. The oxford NOTECHS system: Reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care 2009;18:104-8.
35. OTAS. Observational teamwork assessment for surgery (OTAS). http://www1.imperial.ac.uk/resources/018F4A1D -5129-444E-96CF-04C524C2EA99/otas-manual.pdf. Updated 2011. (accessed 13 Feb2013).
36. Nishisaki A, Nguyen J, Colborn S, et al. Evaluation of multidisciplinary simulation training on clinical performance and team behavior during tracheal intubation procedures in a pediatric intensive care unit. Pediatr Crit Care Med 2011;12:406-14.
37. Fletcher G, Flin R, McGeorge P, et al. Anaesthetists' non-technical skills (ANTS): Evaluation of a behavioural marker system. Br J Anaesth 2003;90:580-8.
38. Frengley RW, Weller JM, Torrie J, et al. The effect of a simulation-based training intervention on the performance of established critical care unit teams. Crit Care Med 2011;39:2605-11.
39. Flowerdew L, Brown R, Vincent C, et al. Development and validation of a tool to assess emergency physicians' nontechnical skills. Ann Emerg Med 2012;59: 376-385.e4.
40. Andersen PO, Jensen MK, Lippert A, et al. Development of a formative assessment tool for measurement of performance in multi-professional resuscitation teams. Resuscitation 2010;81:703-11.
41. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006;119:166.e7-16.
42. Crossley J, Marriott J, Purdie H, et al. Prospective observational study to evaluate NOTSS (non-technical skills for surgeons) for assessing trainees' non-technical performance in the operating theatre. Br J Surg 2011;98:1010-20.
43. Mitchell L, Flin R, Yule S, et al. Development of a behavioural marker system for scrub practitioners' non-technical skills (SPLINTS system). J Eval Clin Pract 2013;19:317-23.
44. Mitchell L, Flin R, Yule S, et al. Evaluation of the scrub practitioners ' list of intraoperative non-technical skills system. Int J Nurs Stud 2012;49:201-11.
45. Yule S, Rowley D, Flin R, et al. Experience matters: Comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg 2009;79:154-60.
46. Russ S, Hull L, Rout S, et al. Observational teamwork assessment for surgery: Feasibility of clinical and nonclinical assessor calibration with short-term training. Ann Surg 2012;255:804-9.
47. Graham J, Hocking G, Giles E. Anaesthesia non-technical skills: Can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth 2010;104:440-5.
48. Lamb BW, Wong HW, Vincent C, et al. Teamwork and team performance in multidisciplinary cancer teams: Development and evaluation of an observational assessment tool. BMJ Qual Saf 2011;20:849-56.
49. Westli HK, Johnsen BH, Eid J, et al. Teamwork skills, shared mental models, and performance in simulated trauma teams: an independent group design. Scand J Trauma Resusc Emerg Med 2010;18:47.
50. Hull L, Arora S, Kassab E, et al. Assessment of stress and teamwork in the operating room: an exploratory study. Am J Surg 2011;201:24-30.
51. Mishra A, Catchpole K, Dale T, et al. The influence of non-technical performance on technical outcome in laparoscopic cholecystectomy. Surg Endosc 2008;22:68-73.
52. Wetzel CM, Black SA, Hanna GB, et al. The effects of stress and coping on surgical performance during simulations. Ann Surg 2010;251:171-6.
53. Capella J, Smith S, Philp A, et al. Teamwork training improves the clinical care of trauma patients. J Surg Educ 2010;67: 439-43.
54. Symons NR, Wong HW, Manser T, et al. An observational study of teamwork skills in shift handover. Int J Surg 2012;10:355-9.
55. Sevdalis N, Wong HW, Arora S, et al. Quantitative analysis of intraoperative communication in open and laparoscopic surgery. Surg Endosc 2012;26:2931-8.
56. Baker DP, Gustafson S, Beaubien J, et al. Medical teamwork and patient safety: the evidence-based relation. 2005; 05-0053.
57. Salas E, Rosen MA, Burke CS, et al. Markers for enhancing team cognition in complex environments: the power of team performance diagnosis. Aviat Space Environ Med 2007;78(5 Suppl):B77-85.
58. Fowlkes J, Lane N, Salas E, et al. Improving the measurement of team performance: The TARGETS methodology. Mil Psychol 1994;6:47-61.
59. Fowlkes JE, Burke CS. Event-based approach to training (EBAT). In: Stanton N, Hedge A, Brookhuis K, Salas E, Hendrick H, eds. Handbook of human factors and ergonomics methods. Boca Raton, FL: CRC Press, 2005:47.1-5.
60. Dwyer DJ, Fowlkes JE, Oser RL, et al. Team performance measurement in distributed environments: the TARGETS methodology. In: Brannick MT, Salas E, Prince C, eds. Team performance assessment and measurement: theory, methods, and applications. Mahwah, NJ: Lawrence Erlbaum Associates, 1997:137-53.
61. Brennan RL. Generalizability theory. New York, NY: Springer, 2001.
62. DeShon RP. Generalizability theory. In: Drasgow F, Schmitt N, eds. Measuring and analyzing behavior in organizations. San Francisco, CA: Jossey-Bass, 2002:189-220.
63. Shavelson RJ, Webb NM. Generalizability theory: a primer. Thousand Oaks, CA: SAGE Publications, 1991.
64. Jeffcott SA, Mackenzie CF. Measuring team performance in healthcare: review of research and implications for patient safety. J Crit Care 2008;23:188-96.
65. Baker DP, Salas E. Principles for measuring teamwork: a summary. In: Brannick MT, Salas E, Prince E, eds. Team performance assessment and measurement: theory, methods, and applications. Mahwah, NJ: Lawrence Erlbaum Associates, 1997:331-55.
66. Brannick MT, Prince C. An overview of team performance measurement. In: Brannick MT, Salas E, Prince C, eds. Team performance assessment and measurement: theory, methods, and applications. Mahwah, NJ: Lawrence Erlbaum Associates, 1997:3-16.
67. Lamb BW, Sevdalis N, Mostafid H, et al. Quality improvement in multidisciplinary cancer teams: an investigation of teamwork and clinical decision-making and cross-validation of assessments. Ann Surg Oncol 2011;18:3535-43.
68. Thomas EJ, Sexton JB, Helmreich RL. Translating teamwork behaviours from aviation to healthcare: development ofbehavioural markers for neonatal resuscitation. Qual Saf Health Care 2004;13(Suppl 1):i57-64.
69. Hamilton N, Freeman BD, Woodhouse J, et al. Team behavior during trauma resuscitation: a simulation-based performance assessment. J Grad Med Educ 2009;1:253-9.
70. Mullan PC, Wuestner E, Kerr TD, et al. Implementation of an in situ qualitative debriefing tool for resuscitations. Resuscitation 2013;84:946-51.
71. Cooper S, Cant R, Porter J, et al. Managing patient deterioration: assessing teamwork and individual performance. Emerg Med J 2013;30:377-81.
72. Walker S, Brett S, McKay A, et al. Observational skill-based clinical assessment tool for resuscitation (OSCAR): development and validation. Resuscitation 2011;82:835-44.
73. Reid J, Stone K, Brown J, et al. The simulation team assessment tool (STAT): development, reliability and validation. Resuscitation 2012;83:879-86.
74. Yule S, Flin R, Paterson-Brown S, et al. Development of a rating system for surgeons' non-technical skills. Med Educ 2006;40:1098-104.
75. O'Leary KJ, Boudreau YN, Creden AJ, et al. Assessment of teamwork during structured interdisciplinary rounds on medical units. J Hosp Med 2012;7:679-83.
76. Wetzel CM, George A, Hanna GB, et al. Stress management training for surgeons-a randomized, controlled, intervention study. Ann Surg 2011;253:488-94.
77. Hull L, Arora S, Kassab E, et al. Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg 2011;212:234-43.e1-5.
78. Undre S, Sevdalis N, Healey AN, et al. Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg 2007;31:1373-81.
79. Nagpal K, Abboudi M, Fischler L, et al. Evaluation of postoperative handover using a tool to assess information transfer and teamwork. Ann Surg 2011;253:831-7.
80. Moorthy K, Munz Y, Adams S, et al. A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg 2005;242:631-9.
81. Flowerdew L, Gaunt A, Spedding J, et al. A multicentre observational study to evaluate a new tool to assess emergency physicians' non-technical skills. Emerg Med J 2013;30:437-43.
82. Sutton G, Liao J, Jimmieson NL, et al. Measuring multidisciplinary team effectiveness in a ward-based healthcare setting: development of the team functioning assessment tool. J Healthc Qual 2011;33:10-23; quiz 23- 4.
83. Sudikoff SN, Overly FL, Shapiro MJ. High-fidelity medical simulation as a technique to improve pediatric residents' emergency airway management and teamwork: a pilot study. Pediatr Emerg Care 2009;25:651-6.
84. nd edn. http://www.socialresearchmethods.net/kb/. Updated 20 October 2006.
85. Lynn MR. Determination and quantification of content validity. Nurs Res 1986;35:382-5.
86. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 1959;56:81-105.
87. th edn. Boston, MA: Pearson Education, Inc., 2007.